'portmaster -a' ordering port builds based on insufficient dependency tracking? #70
Comments
|
Hi John, I have tried to reproduce the issue, but did not succeed. I had already upgraded the libxml2 port, but for a test have performed the following steps: Then I have re-installed the previous versions of these packages: I have then re-built the gnome-user-docs port: It seems that the dependency tracking worked in this case, and I have no idea how to better reproduce the issue you observed with reasonable effort. There is a difference between package building in poudriere (which builds all build and run dependencies in advance and installs them all before proceeding to build the port that has these dependencies) - i.e. it treats build and run dependencies identically. (The only difference is that run depends are registered as package dependencies by poudriere.) Portmaster does depend on the correct specification of BUILD_DEPENDS and it seems that this has not been done in all ports. But a short check of the itstool, libxml2, py-libxml2, and gnome-user-docs ports does not show any issues: The dependencies marked with (*) should suffice to guarantee the correct build order. I do not doubt that you got this build issue, but I cannot spot or reproduce the issue and thus cannot suggest a fix. |
|
I will try to come up with a simple test case that can reproduce the observed problem. |
|
Instead of deleting gnome-user-docs (as you did above), I left the old version (41.1) in place. I also left itstool-2.0.7 installed. 'portmaster -a' gave: I don't know yet if this detail matters (I don't think it does), but in this test, I started with a ports tree at git hash 13977a9f03290d0580ad09f62d8903571cf7e4c2 (Mar 15). Then I updated gnome-user-docs, libxml2, and py-libxml2 to b6ac38cf27573d6d2037fcfffcdf7cffdd35f9a2 (Mar 28). This is a little different than what you did (build the latest, then revert to old packages, then try to build again). <later...> I removed gnome-user-docs and ran Notice that portmaster did not even detect that py-libxml2 needs to be updated (not listed in 'The following actions') in this case, unlike Then I tried deleting itstool. gnome-user-docs is still not installed. {py-,}libxml2 are still both at 2.9.12. That succeeded - because the itstool install triggered py-libxml2 to be built earlier (an update from itstool 2.0.6 to 2.0.7 would do the same). So the problem seems to be that if itstool is already up to date, it does not need to be built, and its run time dependency is not checked (because it is already installed and happy). So portmaster does not think it needs to order the py-libxml2 build before gnome-user-docs. Bumping the itstool PORTREVISION after the py-libxml2 update would also avoid this problem, of course. But should that be necessary? 'portmaster' COULD see that there is a dependency relationship from gnome-user-docs -> itstool -> py-libxml2 and decide to order py-libxml2 before gnome-user-docs even though we are not updating itstool. |
|
Yes, your observations match my knowledge about the inner workings of portmaster (which has not been developed by me, but I have taken over the maintainer role after the original author left the project and FLAVOR support needed to be added to portmaster). There is a -t (thorough) option that should help with the issue you report. It checks run time dependencies of dependencies that need not be rebuilt, too, and I think that is exactly what is required in this case. This is a time-consuming operation and therefore not performed in general. You may want to test whether is solves the issue (it should, IMHO). Adding a direct build dependency to a port affected by such a "deep" version mismatch would solve the issue, but is up to the maintainers of the affected ports. (And many do not bother too much if it successfully builds in poudriere ...) The portmaster script has evolved over 2 decades and had to be adapted to significant changes in the ports infrastructure. It has severe limitations (e.g. uses child processes for per port state, instead of complex data structures). I have been using a mostly compatible rewrite of portmaster (in LUA) for more than 2 years (available from the lua branch of stesser/portmaster on Github, but I do not push local commits often enough). It is not ready for general release (I have to re-implement the conflicts handling functions, for example), by I work on it as my spare time permits. An example run as speed test (with --no-confirm -n) on my development system is: That run included a "thorough" check of dependencies and was performed after updating all ports that had no issues (i.e. with a "warm" buffer cache, not requiring many actual disk accesses). The "fetching" message is due to the scan for required updates being faster then the (parallel) make checksum of all distfiles - without the distfiles checksum verification the time used to check my > 2500 ports for required updates would be lower. But the missing conflicts handling leads to e.g. dependencies on tshark not being redirected to the installed (and conflicting) wireshark. This can be worked around with the "--jailed" option of this new portmaster tool, which builds the port in a clean jail. But as long as the conflicts checks (which worked with a few flaws and needed to be re-implemented from scratch) is missing, this version is not suitable as a complete replacement of the existing portmaster ... (You may want to check it ou anyway, it has a number of other functions that are much faster than in the official portmaster version, e.g. pruning the distfiles or packages directories from stale files.) |
|
Thanks for the reply. And I understand about the history on portmaster. Thanks for helping keep it functional. I tried -t. It did not find that it needed to update py-libxml2, unfortunately. Same error. This was in the situation with {py38-,}libxml2-2.9.12 and itstool-2.0.7 installed, no gnome-user-docs - and invoking 'portmaster -t misc/gnome-user-docs'. It looks like -t does not actually call 'make all-depends-list' (as the documentation for -t states). It just adds 'run depends' (as you stated). Since py-libxml2 is not directly in the run depends for gnome-user-docs, it doesn't add it to the build list, so -t is not a workaround (currently) for the stated problem. 'make all-depends-list' is recursive, so it does include py-libxml2. It also includes a LOT more packages because it also includes all TEST_DEPENDS. I have a patch for make_dep_list to fix -t to be more like the actual all-depends-list, minus test depends which could be a "super thorough" mode - maybe -tt or --test-depends or something. Most people don't typically turn on test depends unless they have a specific need or are doing wider ports tree testing. I'll verify my patch and submit. I'll also take a look at the lua version. Sounds interesting. |
I had not run
portmaster -ain nearly a month on FreeBSD 12/amd64. A bunch of ports were queued up including textproc/libxml2, misc/gnome-user-docs and textproc/py-libxml2, in that order.Those three ports and itstool were already installed:
libxml2-2.9.12
gnome-user-docs-41.1
py38-libxml2-2.9.12
itstool-2.0.7
textproc/libxml2 was successfully updated from 2.9.12 to 2.9.13
Then it came time for misc/gnome-user-docs to be built and the following error occurred:
gnome-user-docs correctly has a build dependency on textproc/itstool. itstool, has a run time dependency on textproc/py-libxml2 (which has a run time dependency on textproc/libxml2).
The itstool python script imports the 'libxml2' python module. But the shared object, libxml2mod.cypthon-38.so, installed by textproc/py-libxml2 had not been rebuilt yet (to catch up to the libxml2-2.9.13 update) when the gnome-user-docs build tried to use itstool. py-libxml2 should be rebuilt because of the libxml2.so.2 change (which in this case with the update from libxml2 2.9.12 to 2.9.13, had a breaking change between the shared lib, libxml.so.2, and the shared object module, libxml2mod.cypthon-38.so). The ldd output showing the dependency on libxml2.so.2 is below.
The question is: Is this a ports bug (where itstool's PORTREVISION should have been bumped due to the py-libxml2 update)? Or is this a portmaster bug where it should have been able to follow the dependency chain and get py-libxml2 (a run time dependency of itstool) built before building gnome-user-docs (which has the build time dependency on itstool)?
The workaround is to build textproc/py-libxml2 alone before going back to build all ports with portmaster -a.
The text was updated successfully, but these errors were encountered: