Plugins and "transitive" dependencies

Development
#1

There has been some discussion on matrix to what extent plugins should or should not depend on pulpcore to pull in their dependencies. As I understood it the central tension was between the following two undesirable situations:

  • Concern 1: Not declaring your dependencies explicitly is inherently risky and opaque and by extension bad practice.
  • Concern 2: Plugins and pulpcore all declaring dependencies on the same things (with specific version ranges) risks unsatisfiable dependencies whenever the version ranges from different plugins or pulpcore accidentally diverge.
    • This risk implies a related discussion about how to define sufficiently broad version ranges for dependencies without risking accidental incompatibilities with some new version of some dependency.

I have a few thoughts regarding this discussion, that I want to explain using some concrete examples:

  • Example 1: As a plugin author I don’t want to manage any django or asyncio dependencies. I consider these to be an inherent part of pulpcore, and pulpcore alone is responsible for pulling in the right versions, warning plugin authors about breaking changes when pulpcore decides to pull in newer versions, etc. As far as I can tell this is also the status quo of pulp development and has worked well so far, so no action needs to be taken on this front.
  • Example 2: pulp_deb currently does not declare its dependency on python-gnupg. This has not caused any problems to date, since pulpcore has always pulled in this dependency. However, while I think it unlikely that pulpcore will drop this dependency any time soon, in contrast to the django dependencies, it does not strike me as a “inherent necessary part of pulpcore” to have a dependency on python-gnupg. Changes to this dependency may be unlikely, but not impossible. I therefor think pulp_deb should make it’s dependency explicit. How then, should we handle “Concern 2” in this case? Personally I feel like there is no one general way how to handle version level dependencies, so I would like to choose a practical approach to each individual case: If I know the dependency in question makes a real effort to use semantic versioning, I can choose a version range on that basis. However, I don’t just want to define such a range based on the vague assumption that upstream projects should be using semantic versioning, when in fact I have no evidence that they do. I also quite frankly don’t want to handle dependabot PRs every other day, for something that was zero work in the past, and has never caused a problem. All of this gets me here: I would like to add this particular dependency to pulp_deb without a version range. This is an improvement over the status quo by making the dependency visible, without causing more work, or any risk of “Concenr 2”. Of course, it does not guarantee that we don’t run into some version level incompatibility one day. But the only way to eliminate that risk, would be to allow only versions the plugin was tested with, and that would come at the cost of significant maintenance effort and “Concern 2 risk”. One can always add a version range when one hits some actual reason to do so later. Thoughts?
Katello/Pulp3 Integration meeting
#2

I think we can add some kind of #noqa to CI, so it can ignore the specified package

python-gnupg #noqa

#3

Allowing to specify dependencies without an upper bound on the version comes with its own drawbacks. One recurring example that has bitten us several times in multiple repositories was the release of click version 8. Yes, click made incompatible changes there. And they had all permission to do so (even with semver). But still we had dependencies declining to specify they support only click<8. We needed to add that versioned dependency for a transitive dependency to our repos for a looong time.
But worse than that is, once you need to add an upper bound on the version, and release with that, pip will start to dig up older version of e.g. Pulpcore that didn’t have that upper bound, because they are obviously compatible with the new version of the dependency, while the latest version of Pulpcore isn’t. Naturally this will then fail horribly.
To me stating the unversioned dependency in the plugin, relying on pulpcore to have a versioned one, defeats the purpose of restating the dependency in the first place.

#4

Ok, I did not know this. This invalidates my idea of adding the dependency without a version range.

Meanwhile I searched through pulp_deb to find some more dependencies that are not made explicit in the requirements file. I came up with:

python-gnupg  # Currently pulled in by pulpcore
jsonschema  # Currently pulled in by pulp_rpm but not pulpcore!
asgiref  # No idea how this is being pulled in but all the plugins seem to use it.
aiohttp  # Probably fine to let pulpcore handle this one (similar to django)
asyncio  # Is this being pulled in by the pulpcore asyncio-throttle requirement? Otherwise I don't know where it is being pulled in...

For now, I have created a draft PR, so I don’t forget about this: https://github.com/pulp/pulp_deb/pull/585