Problem:
I am a Satellite/katello user who identified that pulp-3 dependency solving does not work in a deterministic way. As I am aware of some limitations of libsolv, I am asking: is that nondeterminism expected behaviour or not?
More technically: having a system with a given repo synced and copying a given set of packages to another repo, should dependency solving add the same set of packages, regardless of what system I execute it?
Particular example (as minimal as I was able to prepare, so bit artificial):
- have a RHEL8 x86_64 BaseOS repository
- create a new repo and copy there - with dependency solving enabled:
- RPMs without an errata
- errata (and their content) issued no newer than 2020-01-01
- that use case corresponds to katello’s Content View with filters “add RPMs without errata” and “add errata issued no newer than 2020-01-01” and with dep.solving enabled
- the copy itself copies 3394 packages
- then, dependency solver adds some more packages
And here the problem comes: on two identical pulp servers with identical synced repo content, I get different outcome. E.g. on one pulp server, I always get 3571 packages, repeatedly - but on another server, I get (repeatedly) 3577 packages.
First and main question: is that nondeterminism expected or allowed?
Second, I see some subtle weirdness in libsolv reasoning when inspecting /var/tmp/pulp/*/depsolving_summary.txt - aren’t these bugs alone?
In particular, one of extra package added by the dep.solving on just one pulp server was python3-hawkey-0:0.55.0-7.el8.x86_64 explained via:
python3-hawkey-0:0.55.0-7.el8.x86_64
Pulp Content unit 'e5ee4d30-aa2b-4458-87e8-a889fae9e72a' from repo 'Red_Hat_Enterprise_Linux_8_for_x86_64_-_BaseOS_RPMs_8-1144: version=1'
Reason: SOLVER_REASON_RESOLVE - The package was installed to fulfill package dependencies.
Rules:
SOLVER_RULE_PKG_REQUIRES - Similar to SOLVER_RULE_PKG_NOTHING_PROVIDES_DEP, but in this case some packages provided the dependency but none of them could be installed due to other dependency issues.
Because package 'kpatch-dnf-0:0.4-3.el8.noarch' requires 'python3-hawkey'
kpatch-dnf was not in the copied packages, it was also added by the libsolv. BUT this package was added on both pulp servers - so why one server triggered "oh, I additionally need python3-hawkey" step but not the other pulp server?
Further, when checking why kpatch-dnf was added by libsolv, I got confusing reasoning from one pulp server:
kpatch-dnf-0:0.4-3.el8.noarch
Pulp Content unit '80b88ce4-79c9-41f4-abbd-2365dfdbddcf' from repo 'Red_Hat_Enterprise_Linux_8_for_x86_64_-_BaseOS_RPMs_8-15544: version=1'
Reason: SOLVER_REASON_UNIT_RULE - The package was installed/erased/kept because of a unit rule, i.e. a rule where all literals but one were false.
Rules:
SOLVER_RULE_JOB - Job rules implement the job given to the solver.
Because package 'ledmon-0:0.96-2.el8.x86_64' requires 'iucode_rev(fname:intel/06-55-04;cpuid:00050654;pf_mask:0xb7;segment:"Server";codename:"Skylake";stepping:"U0";pf_model:0xb7)'
But ledmon-0.96-2.el8.x86_64 does not require that iucode_rev capability, and kpatch-dnf package does not provide that capability - so why the reasoning…? (I see no reason of a dependency between those two packages at all)
Expected outcome:
- Should be dependency solving deterministic (not only on one pulpserver, but also “server-wide”)?
- Why
python3-hawkeywas added on one pulp server but not on the other? - Why the incorrect reasoning of the
kpatch-dnf<->ledmondependency?
Pulpcore version:
python39-pulpcore-3.18.11-1.el8pc.noarch
python39-pulp-rpm-3.18.9-1.el8pc.noarch
libsolv-0.7.22-4.el8pc.x86_64
Pulp plugins installed and their versions:
a typical Satellite 6.12 deployment, but the problem is generic (observed on older versions also)
Operating system - distribution and version:
RHEL8.7
Other relevant data:
I can provide anything upon request, like:
- list of packages added without dep.solving
- list of packages added by depsolving
depsolving_summary.txt- or even access to either reproducer system
Just any data is too big to copy&paste here.