ariadne
/
pkgconf
3
6
Fork 4
Code Issues 6 Pull Requests 2 Projects Releases 2 Wiki Activity

[WIP] Attempt to add /usr/local search paths #199

Closed
anordal wants to merge 1 commits from opensuse-1175039 into master
pull from: opensuse-1175039
merge into: ariadne:master
Conversation 19 Commits 1 Files Changed 1
anordal commented 2020-08-09 14:27:41 +00:00 (Migrated from github.com)

While this didn't actually work, please see:
https://bugzilla.opensuse.org/show_bug.cgi?id=1175039

While this didn't actually work, please see: https://bugzilla.opensuse.org/show_bug.cgi?id=1175039
ariadne commented 2020-08-09 14:43:19 +00:00

In Alpine, we use autoconf build system still because meson isn't in the bootstrap.

To set the paths we use --with-pkg-config-dir: https://git.alpinelinux.org/aports/tree/main/pkgconf/APKBUILD#n24

The meson build system should be adjusted to have the same configuration option.

In Alpine, we use autoconf build system still because meson isn't in the bootstrap. To set the paths we use `--with-pkg-config-dir`: https://git.alpinelinux.org/aports/tree/main/pkgconf/APKBUILD#n24 The meson build system should be adjusted to have the same configuration option.
ariadne commented 2020-08-09 15:41:51 +00:00

Anyway, opensuse uses autoconf and --with-pkg-config-dir correctly:

%global pkgconf_libdirs %{_libdir}/pkgconfig:%{_datadir}/pkgconfig
[...]
%build
autoreconf -fiv
%configure --disable-static \
           --with-pkg-config-dir=%{pkgconf_libdirs} \
           --with-system-includedir=%{_includedir} \
           --with-system-libdir=%{_libdir}

While it is true that the Meson build system does not support configuring the search path, they aren't using Meson, so it's not a problem.

The redhat customer complaining about pkgconf claims that PKG_CONFIG_PATH is not working for him.

Most likely it is actually a problem with his .pc file, or some RHEL-specific packaging issue, but we cannot be certain because he is using a portal that only RHEL customers may participate on.

Can you elaborate on what problem you are actually trying to solve, since it appears SuSE does not include /usr/local paths in the default search path on purpose?

Anyway, [opensuse uses autoconf and `--with-pkg-config-dir` correctly](https://build.opensuse.org/package/view_file/openSUSE:Factory/pkgconf/pkgconf.spec?expand=1): ``` %global pkgconf_libdirs %{_libdir}/pkgconfig:%{_datadir}/pkgconfig [...] %build autoreconf -fiv %configure --disable-static \ --with-pkg-config-dir=%{pkgconf_libdirs} \ --with-system-includedir=%{_includedir} \ --with-system-libdir=%{_libdir} ``` While it is true that the Meson build system does not support configuring the search path, they aren't using Meson, so it's not a problem. The redhat customer complaining about pkgconf claims that PKG_CONFIG_PATH is not working for him. Most likely it is actually a problem with his `.pc` file, or some RHEL-specific packaging issue, but we cannot be certain because he is using a portal that only RHEL customers may participate on. Can you elaborate on what problem you are actually trying to solve, since it appears SuSE does not include /usr/local paths in the default search path on purpose?
ariadne commented 2020-08-09 15:43:16 +00:00

I would have said the latter part on opensuse bugzilla but I cannot for the life of me figure out how to create an account there that actually works. Either way, this is not an issue in pkgconf, so closing this PR.

I would have said the latter part on opensuse bugzilla but I cannot for the life of me figure out how to create an account there that actually works. Either way, this is not an issue in pkgconf, so closing this PR.
Conan-Kudo commented 2020-08-09 15:54:33 +00:00 (Migrated from github.com)

The RH/Fedora and openSUSE packaging are functionally identical since I wrote both. The /usr/local paths are not in the automatic search paths by policy to prevent accidental, difficult-to-uncover issues.

As far as I'm aware $PKG_CONFIG_PATH should still work to append to the paths.

The RH/Fedora and openSUSE packaging are functionally identical since I wrote both. The `/usr/local` paths are not in the automatic search paths by policy to prevent accidental, difficult-to-uncover issues. As far as I'm aware `$PKG_CONFIG_PATH` should still work to append to the paths.
Conan-Kudo commented 2020-08-09 16:42:20 +00:00 (Migrated from github.com)

I've confirmed this does work as intended with a minimally valid test.pc file:

~> podman run --pull=always --rm -it registry.opensuse.org/opensuse/tumbleweed
Trying to pull registry.opensuse.org/opensuse/tumbleweed...
Getting image source signatures
Copying blob 18a29e57145b done
Copying config b500be0ac9 done
Writing manifest to image destination
Storing signatures
b2deda799032:/ # zypper install -C pkg-config
Retrieving repository 'openSUSE-Tumbleweed-Non-Oss' metadata .............[done]
Building repository 'openSUSE-Tumbleweed-Non-Oss' cache ..................[done]
Retrieving repository 'openSUSE-Tumbleweed-Oss' metadata .................[done]
Building repository 'openSUSE-Tumbleweed-Oss' cache ......................[done]
Retrieving repository 'openSUSE-Tumbleweed-Update' metadata ..............[done]
Building repository 'openSUSE-Tumbleweed-Update' cache ...................[done]
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following 4 NEW packages are going to be installed:
  libpkgconf3 pkgconf pkgconf-m4 pkgconf-pkg-config

4 new packages to install.
Overall download size: 109.7 KiB. Already cached: 0 B. After the operation,
additional 158.3 KiB will be used.
Continue? [y/n/v/...? shows all options] (y): y
Retrieving package libpkgconf3-1.7.3-1.2.x86_64
                                           (1/4),  35.6 KiB ( 68.8 KiB unpacked)
Retrieving: libpkgconf3-1.7.3-1.2.x86_64.rpm .................[done (3.8 KiB/s)]
Retrieving package pkgconf-m4-1.7.3-1.2.noarch
                                           (2/4),  18.7 KiB ( 13.9 KiB unpacked)
Retrieving: pkgconf-m4-1.7.3-1.2.noarch.rpm ..............................[done]
Retrieving package pkgconf-1.7.3-1.2.x86_64
                                           (3/4),  41.3 KiB ( 75.2 KiB unpacked)
Retrieving: pkgconf-1.7.3-1.2.x86_64.rpm .................................[done]
Retrieving package pkgconf-pkg-config-1.7.3-1.2.x86_64
                                           (4/4),  14.2 KiB (  457   B unpacked)
Retrieving: pkgconf-pkg-config-1.7.3-1.2.x86_64.rpm .........[done (10.7 KiB/s)]

Checking for file conflicts: .............................................[done]
(1/4) Installing: libpkgconf3-1.7.3-1.2.x86_64 ...........................[done]
(2/4) Installing: pkgconf-m4-1.7.3-1.2.noarch ............................[done]
(3/4) Installing: pkgconf-1.7.3-1.2.x86_64 ...............................[done]
(4/4) Installing: pkgconf-pkg-config-1.7.3-1.2.x86_64 ....................[done]
b2deda799032:/ # cd ~
b2deda799032:~ # cat >> test.pc <<EOF
> Name: test
> Description: test library
> Version: 0
> Libs: -ltest
> Cflags: -I/usr/include
> EOF

b2deda799032:~ # PKG_CONFIG_PATH=/root pkg-config --libs test
-ltest
I've confirmed this does work as intended with a minimally valid `test.pc` file: ``` ~> podman run --pull=always --rm -it registry.opensuse.org/opensuse/tumbleweed Trying to pull registry.opensuse.org/opensuse/tumbleweed... Getting image source signatures Copying blob 18a29e57145b done Copying config b500be0ac9 done Writing manifest to image destination Storing signatures b2deda799032:/ # zypper install -C pkg-config Retrieving repository 'openSUSE-Tumbleweed-Non-Oss' metadata .............[done] Building repository 'openSUSE-Tumbleweed-Non-Oss' cache ..................[done] Retrieving repository 'openSUSE-Tumbleweed-Oss' metadata .................[done] Building repository 'openSUSE-Tumbleweed-Oss' cache ......................[done] Retrieving repository 'openSUSE-Tumbleweed-Update' metadata ..............[done] Building repository 'openSUSE-Tumbleweed-Update' cache ...................[done] Loading repository data... Reading installed packages... Resolving package dependencies... The following 4 NEW packages are going to be installed: libpkgconf3 pkgconf pkgconf-m4 pkgconf-pkg-config 4 new packages to install. Overall download size: 109.7 KiB. Already cached: 0 B. After the operation, additional 158.3 KiB will be used. Continue? [y/n/v/...? shows all options] (y): y Retrieving package libpkgconf3-1.7.3-1.2.x86_64 (1/4), 35.6 KiB ( 68.8 KiB unpacked) Retrieving: libpkgconf3-1.7.3-1.2.x86_64.rpm .................[done (3.8 KiB/s)] Retrieving package pkgconf-m4-1.7.3-1.2.noarch (2/4), 18.7 KiB ( 13.9 KiB unpacked) Retrieving: pkgconf-m4-1.7.3-1.2.noarch.rpm ..............................[done] Retrieving package pkgconf-1.7.3-1.2.x86_64 (3/4), 41.3 KiB ( 75.2 KiB unpacked) Retrieving: pkgconf-1.7.3-1.2.x86_64.rpm .................................[done] Retrieving package pkgconf-pkg-config-1.7.3-1.2.x86_64 (4/4), 14.2 KiB ( 457 B unpacked) Retrieving: pkgconf-pkg-config-1.7.3-1.2.x86_64.rpm .........[done (10.7 KiB/s)] Checking for file conflicts: .............................................[done] (1/4) Installing: libpkgconf3-1.7.3-1.2.x86_64 ...........................[done] (2/4) Installing: pkgconf-m4-1.7.3-1.2.noarch ............................[done] (3/4) Installing: pkgconf-1.7.3-1.2.x86_64 ...............................[done] (4/4) Installing: pkgconf-pkg-config-1.7.3-1.2.x86_64 ....................[done] b2deda799032:/ # cd ~ b2deda799032:~ # cat >> test.pc <<EOF > Name: test > Description: test library > Version: 0 > Libs: -ltest > Cflags: -I/usr/include > EOF b2deda799032:~ # PKG_CONFIG_PATH=/root pkg-config --libs test -ltest ```
anordal commented 2020-08-09 17:36:58 +00:00 (Migrated from github.com)

The /usr/local paths are not in the automatic search paths by policy to prevent accidental, difficult-to-uncover issues.

That's extremely sad. I don't know what good reason you have, but you're breaking every build system (case in point: Meson) on purpose. Nobody expects to set PKG_CONFIG_PATH. This used to work.

$PKG_CONFIG_PATH should still work to append to the paths.

Indeed, PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig makes it work for me, but I seriously didn't have the fantasy to try that.

> The /usr/local paths are not in the automatic search paths by policy to prevent accidental, difficult-to-uncover issues. That's extremely sad. I don't know what good reason you have, but you're breaking every build system (case in point: Meson) on purpose. Nobody expects to set PKG_CONFIG_PATH. This used to work. > $PKG_CONFIG_PATH should still work to append to the paths. Indeed, `PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig` makes it work for me, but I seriously didn't have the fantasy to try that.
ariadne commented 2020-08-09 18:42:04 +00:00

For what its worth, you can install a site config file to change this behavior. Read the pkgconf-personality man page. You will want to name the personality file default.personality.

I suspect Fedora/RHEL/SUSE have excluded /usr/local to ensure that only their libraries are pulled in on their builds.

It may be helpful to streamline the site configuration somehow to make it easier for your specific scenario. I'm open to ideas.

For what its worth, you can install a site config file to change this behavior. Read the pkgconf-personality man page. You will want to name the personality file `default.personality`. I suspect Fedora/RHEL/SUSE have excluded /usr/local to ensure that only their libraries are pulled in on their builds. It may be helpful to streamline the site configuration somehow to make it easier for your specific scenario. I'm open to ideas.
👍 1
anordal commented 2020-08-09 22:07:29 +00:00 (Migrated from github.com)

The site configuration might be a solution for the workplace, but as a library author (https://github.com/anordal/narg), I'm still devastated – what's the point of shipping a pkg-config file anymore? Now that Meson supports CMake packages, it makes more sense to ship that.

The site configuration might be a solution for the workplace, but as a library author (https://github.com/anordal/narg), I'm still devastated – what's the point of shipping a pkg-config file anymore? Now that Meson supports CMake packages, it makes more sense to ship that.
Conan-Kudo commented 2020-08-09 22:14:54 +00:00 (Migrated from github.com)

I suspect Fedora/RHEL/SUSE have excluded /usr/local to ensure that only their libraries are pulled in on their builds.

This is pretty much the reason. Given that pkgconf is configurable and allows for overrides via variables and config files, it's not really a big deal that it's not searched by default.

For what it's worth, most distributions have never enabled search of /usr/local paths with legacy pkgconfig, so this behavior already existed across the board.

> I suspect Fedora/RHEL/SUSE have excluded /usr/local to ensure that only their libraries are pulled in on their builds. This is pretty much the reason. Given that pkgconf is configurable and allows for overrides via variables and config files, it's not really a big deal that it's not searched by default. For what it's worth, _most_ distributions have _never_ enabled search of `/usr/local` paths with legacy pkgconfig, so this behavior already existed across the board.
anordal commented 2020-08-09 23:15:45 +00:00 (Migrated from github.com)

Isn't that a bit backwards? Distro-packaging is normally the exception. E.g. all software is supposed to install to /usr/local by default, while distro packagers are entitled to set the prefix to /usr. If that's a principle to follow, it would make sense if distro packagers were also the entitled ones to override pkg-config.

I'm surprised to hear that most distros never searched in /usr/local, but in that case, consider this a feature request ;-)
Letting it "just work" for end users (those who are just trying to build the damn thing) is attractive, even to someone as indirectly responsible as a library author.

Isn't that a bit backwards? Distro-packaging is normally the exception. E.g. all software is supposed to install to `/usr/local` by default, while distro packagers are entitled to set the prefix to `/usr`. If that's a principle to follow, it would make sense if distro packagers were also the entitled ones to override pkg-config. I'm surprised to hear that most distros never searched in /usr/local, but in that case, consider this a feature request ;-) Letting it "just work" for end users (those who are just trying to build the damn thing) is attractive, even to someone as indirectly responsible as a library author.
ariadne commented 2020-08-10 00:06:23 +00:00

Personally I have to agree with @anordal. The way that we handle this in alpine is to just override the search path in the build environment. But either way, this is probably a conversation to have downstream -- the pkgconf project can't change how downstream distributions are shipping the software.

(Oh, and if someone wants to fix the meson --with-pkg-config-path feature, that would be welcome here.)

Personally I have to agree with @anordal. The way that we handle this in alpine is to just override the search path in the build environment. But either way, this is probably a conversation to have downstream -- the pkgconf project can't change how downstream distributions are shipping the software. (Oh, and if someone wants to fix the meson --with-pkg-config-path feature, that would be welcome here.)
orbea commented 2020-08-10 02:21:33 +00:00 (Migrated from github.com)

For what it's worth, most distributions have never enabled search of /usr/local paths with legacy pkgconfig, so this behavior already existed across the board.

I just want to point out this is not true in Slackware where both pkg-config and pkgconf search /usr/local by default.

> For what it's worth, most distributions have never enabled search of /usr/local paths with legacy pkgconfig, so this behavior already existed across the board. I just want to point out this is not true in Slackware where both pkg-config and pkgconf search /usr/local by default.
ariadne commented 2020-08-10 02:59:27 +00:00

I feel like perhaps writing a document outlining best practices for distributions shipping pkg-config implementations might be useful.

Alpine and Debian ship /usr/local paths in default PKG_CONFIG_LIBDIR, then in package build environment explicitly set PKG_CONFIG_LIBDIR to /usr paths only. I suspect this is what most distributions should be doing. That allows for /usr/local to work as expected by end users, while ensuring the build environment is clean of /usr/local influence.

I feel like perhaps writing a document outlining best practices for distributions shipping pkg-config implementations might be useful. Alpine and Debian ship `/usr/local` paths in default `PKG_CONFIG_LIBDIR`, then in package build environment explicitly set `PKG_CONFIG_LIBDIR` to `/usr` paths only. I suspect this is what most distributions should be doing. That allows for `/usr/local` to work as expected by end users, while ensuring the build environment is clean of `/usr/local` influence.
👍 1
Conan-Kudo commented 2020-08-10 03:22:05 +00:00 (Migrated from github.com)

On the flip side, the RPM ecosystem tends to lean towards synchronizing build and user environment settings as much as possible, to eliminate "surprise" inconsistencies.

It's also still unfortunately common for people to build RPMs on their regular environments by just running rpmbuild. Not having /usr/local paths by default makes it so those simply don't leak in for those circumstances. And folks in the RPM ecosystem platforms do tend to make RPMs that mix prefixes (largely due to the larger third-party ISV ecosystems around Red Hat and SUSE distributions). In both commercial platforms, there is a very strong encouragement to ship software through the RPM build process, though this has been de-emphasized in recent years.

However, I am tempted to consider the idea of making the pkg-config wrapper behave differently depending on the environment. My concern is reliably switching behavior.

On the flip side, the RPM ecosystem tends to lean towards synchronizing build and user environment settings as much as possible, to eliminate "surprise" inconsistencies. It's also still unfortunately common for people to build RPMs on their *regular* environments by just running `rpmbuild`. Not having `/usr/local` paths by default makes it so those simply don't leak in for those circumstances. And folks in the RPM ecosystem platforms _do_ tend to make RPMs that mix prefixes (largely due to the larger third-party ISV ecosystems around Red Hat and SUSE distributions). In both commercial platforms, there is a _very_ strong encouragement to ship software through the RPM build process, though this has been de-emphasized in recent years. However, I *am* tempted to consider the idea of making the `pkg-config` wrapper behave differently depending on the environment. My concern is reliably switching behavior.
ariadne commented 2020-08-10 03:52:29 +00:00

Well, in Alpine, we can change the environment variables that are exposed when you run abuild by using /etc/abuild.conf. It's similar with Debian's debuild.

If rpmbuild has similar functionality, then it's just a matter of doing export PKG_CONFIG_LIBDIR=/usr/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig in some config file somewhere.

Well, in Alpine, we can change the environment variables that are exposed when you run `abuild` by using `/etc/abuild.conf`. It's similar with Debian's `debuild`. If `rpmbuild` has similar functionality, then it's just a matter of doing `export PKG_CONFIG_LIBDIR=/usr/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig` in some config file somewhere.
orbea commented 2020-08-10 04:47:13 +00:00 (Migrated from github.com)

In alpine or in any of the RPM distros do other tools like ld(1) protect against using /usr/local? For example in Slackware I see this as well.

$ ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012
SEARCH_DIR("/usr/x86_64-slackware-linux/lib64")
SEARCH_DIR("/usr/lib64")
SEARCH_DIR("/usr/local/lib64")
SEARCH_DIR("/lib64")
SEARCH_DIR("/usr/x86_64-slackware-linux/lib")
SEARCH_DIR("/usr/local/lib")
SEARCH_DIR("/lib")
SEARCH_DIR("/usr/lib"

I'm not sure how useful it would be to configure these protections for just pkgconf and not the rest of the system?

In alpine or in any of the RPM distros do other tools like `ld(1)` protect against using `/usr/local`? For example in Slackware I see this as well. ``` $ ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012 SEARCH_DIR("/usr/x86_64-slackware-linux/lib64") SEARCH_DIR("/usr/lib64") SEARCH_DIR("/usr/local/lib64") SEARCH_DIR("/lib64") SEARCH_DIR("/usr/x86_64-slackware-linux/lib") SEARCH_DIR("/usr/local/lib") SEARCH_DIR("/lib") SEARCH_DIR("/usr/lib" ``` I'm not sure how useful it would be to configure these protections for just pkgconf and not the rest of the system?
Conan-Kudo commented 2020-08-10 04:51:40 +00:00 (Migrated from github.com)

We used to have them, but they've been chipped away over the years as people have wanted relocatable RPMs and RPMs built on alternative prefixes to work (e.g. Flatpak runtimes and apps composed from RPM builds...).

We used to have them, but they've been chipped away over the years as people have wanted relocatable RPMs and RPMs built on alternative prefixes to work (e.g. Flatpak runtimes and apps composed from RPM builds...).
Conan-Kudo commented 2020-08-10 05:10:08 +00:00 (Migrated from github.com)

I've adjusted the behavior for Fedora 33+: 05c8560a2e

As for openSUSE Tumbleweed, I've made an SR to change this too: https://build.opensuse.org/request/show/825209

I've adjusted the behavior for Fedora 33+: https://src.fedoraproject.org/rpms/pkgconf/c/05c8560a2eadac70cdc0e66554b5a4a6ac298f01 As for openSUSE Tumbleweed, I've made an SR to change this too: https://build.opensuse.org/request/show/825209
🎉 1
ariadne commented 2020-08-11 04:05:46 +00:00

Alpine's LD does search /usr/local paths by default. Perhaps we should fix that. Thanks for the observation @orbea.

Alpine's LD does search /usr/local paths by default. Perhaps we should fix that. Thanks for the observation @orbea.

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
   
duplicate
   
enhancement
   
invalid
   
question
   
wontfix
No Label
bug
duplicate
enhancement
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: ariadne/pkgconf#199
There is no content yet.
Delete Branch "opensuse-1175039"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?