What to do to reduce unecessary plugin "compatibility" releases

Development
#1

At pulpcon 2021 the problem came up that the N+1 forward compatibility with pulpcore causes a lot of unnecessary plugin releases. These are unnecessary in the sense that they are only required to bump the upper bound of compatibility.

As a problem example, here’s a pulp_file commit for a pulp_file release that sets the upper bound at release time.

There was a session at pulpcon discussing ideas for this and I’ll try to summarize them.

Two ideas we should do no matter what

  1. Merge breaking changes in the plugin API very early in the pulpcore release cycle. So if we are removing this in pulpcore 3.20, we should be removing all of those things say 5-7 calendar days after we release pulpcore 3.19. This gives plugin writers a long time to update their code for compatibility.

  2. Update the CI to show that there are deprecations emitted as each plugin runs its test suite. The goal here is to make it as clear as possible to plugin writers their plugin is not compatible with the upcoming pulpcore release.

Additional ideas:

Only make breaking changes in specific releases

This approach would have say every 3 pulpcore releases or perhaps everyone divisible evenly by 5, 3.20, 3.25, etc allow breaking changes. This would allow plugins to declare compatibility up until the next “3rd” or “5th” release which would reduce the number of plugin releases significantly.

I believe this idea had the most consensus ^.

Make the pulpcore version number apply to the plugin API

In this case, when we wanted to introduce breaking plugin API changes we would be releasing Pulp 4. At least one concern resonated especially with me. @dralley pointed out that if we left the api at /pulp/api/v3/ it would be confusing so we’d have /pulp/api/v4/. Even if upgrading is easy, it’s hard to ask all users to switch from using /pulp/api/v3/ to /pulp/api/v4/ unless /pulp/api/v4/ is meaningfully a new, backwards incompatible API.

3 Likes
Pulpcore Meeting Minutes
#2

This sounds great to me. No complaints here.

#3

+1 to the no-matter-what ideas, I would just emphasize that the deprecations need to be loud. They are present and visible in CI now but the job is green and no one notices. I do not think it’s the best approach to fail the main job for that but ideally this would be a loud but non-blocking check in CI.

Note to myself, the additional ideas do not necessarily exclude each other.
I do not have strong opinion one way or the other but I’ll add some perspective to both:

Additional idea #1 Only make breaking changes in specific releases
Pros:

  • every N releases to allow breaking changes in plugin API gives predictability for plugin writers
  • gives a lot of time to address incompatibilities if deprecations are happening gradually in pulpcore and addressed asap by plugin maintainers

Cons:

  • potentially all breaking changes are getting “collected” closer to The Release, or plugin writers postpone to address deprecations till last moment (making deprecations loud helps with that to an extent but we are still humans :slight_smile: )
  • what to do if we do not have breaking changes for Nth release ready but it would be convenient to do it in the N+1 one, do we wait another N releases because we missed the breaking-change release?

Addiitonal idea #2 Make the pulpcore version number apply to the plugin API
Pros:

  • same +'s as in the idea #1, with additional +. We still can wait but no less than N releases before introducing breaking changes and we do not need to stick to every N releases in case we “missed” the breaking release.
  • it’s easier to use the next X release as an upper limit for pulpcore requirement than remember and follow the convention that every N release is a breaking one (especially, if you are not living in pulp world only and maintain/develop other software)

Cons:

  • the issue with potentially not addressing deprecations early enough due to pulpcore or maintainers being late stays

  • inconsistency between pulpcore major verison and plugins major versions and REST API version can be confusing.

    Maybe I miss something but I’m not sure why REST API version (which is what we refer to as /pulp/api/v3/) should be necessarily changed when the plugin API has breaking changes. The idea #2 is a clear no-go for me if REST API version changes with every major release but I’m not sure it’s necessary.

#4

If we make backwards incompatible changes to the plugin API every 5th release, these changes would happen about every 7 months. I like this plan, but maybe we should go with every 4th release?

#5

Maybe you can specify the version the plugin should be compatible with and all deprecations that are found for that release will be breaking CI. But then we should still show the ones you can already know of.

#6

I propose we make the next pulpcore release to contain breaking plugin API changes to be 3.20. We don’t need a “forever” change in policy before we’ve tried this idea. If we can agree on 3.20 it will give these benefits:

  • Plugins can declare their next releases as compatible with <3.20
  • We can try before committing to something forever

What sorts of CI changes do we need to have the release automation declare 3.20 instead of 3.next?

#7

+1, 3.20 is basically 3 releases ahead which is not too short, not too long period to try it out.

To my knowledge the automation does not set requirements, it’s done manually by a plugin maintainer.

#8

With all positive feedback and no negative feedback heard I’ve written this blog post to make it an official thing: https://github.com/pulp/pulpproject.org/pull/399

Additionally, the Roles work has plugin API deprecations and eventual removals as part of it and that is declaring in those deprecations they will be removed in 3.20.

1 Like
#9

+1 to this. I’d still discuss though every which release there will be breaking changes in plugin API.

#10

At the pulpcore meeting we decided to try this. This has been announced publicly here: Increasing Pulpcore Compatibility with Plugin Versions | software repository management

When we are preparing the pulpcore==3.20 release, we’ll need to decide what the next “plugin API breaking change will be”, e.g. pulpcore 3.24 or 3.25.

#11

I think it’s safe to say the “bundling of all breaking plugin API changes into one release” has successfully reduced unnecessary plugin compat releases. It’s time for us to pick the next “plugin API breaking change release” after 3.20.

I propose we select pulpcore==3.25 because that would be roughly 6-7 months from now. I see a lot of benefit of plugins not having to all release to have declare forward comparability with pulpcore, and I see little benefit to introducing plugin API breaking changes sooner than 6-7 months.

What do you think we should do?

2 Likes
#12

Plus one for the 3.25 idea and more importantly on the “every minor version divisible by 5” idea, because that is rather easy to remember. Also roughly half a year sounds like a good balance between managing the churn with breaking changes and the resentment for missing the window with any particular change.

Another question that came to my mind: When do we start declaring deprecations for the new breaking release? e.g. I think, we should not remove anything with 3.20 that has not been deprecated with 3.19 at least. Should we extend that to 3.18 (as in 2 releases back from the breaking one)?

#13

Ok, since pulpcore 3.19 is now out, and the dreaded breaking 3.20 is coming up next, I would like to ask the following:

When doing a plugin Y-release now, (assuming the plugin is not experiencing any deprecation warnings), should that plugin declare compatibility up to pulpcore<3.25 or up to pulpcore<3.20?

If it is the latter (i.e. there can be breaking changes in pulpcore 3.20 that have not already been advertised as deprecations), I suppose this means, that once pulpcore 3.20 releases, all plugins will need to check for breakage, fix them, and do a compatibility Y-release. This would mean there would be a period (however short) where pulpcore 3.20 is released, but has no compatible plugins. More importantly there would be a probably not so short period, where not all plugins are compatible with 3.20. Thoughts, corrections, additions?

EDIT: I guess a somewhat bumpy 3.20 release is tolerable since the breaking releases are only expected every 6 months or so going forward.

#14

That basically represents the other end of my question

To be fair, that is kind of only asserting the original policy, that we cannot remove things without deprecating them in an earlier release.

#15

I propose we select pulpcore==3.25 because that would be roughly 6-7 months from now. I see a lot of benefit of plugins not having to all release to have declare forward comparability with pulpcore, and I see little benefit to introducing plugin API breaking changes sooner than 6-7 months.

I don’t think that matches our actual cadence.

3.18 → 3.19 = 7 weeks
3.17 → 3.18 = 9 weeks (that includes winter shutdown)
3.16 → 3.17 = 9 weeks
3.15 → 3.16 = 5 weeks
3.14 → 3.15 = 9 weeks

We’re currently at nearly 8 months since 3.15, and I don’t see 3.20 releasing any sooner than 6-8 weeks from now, so that will be 9-10 months between breaking changes. So if we change nothing else about our process that would be roughly what it would be going forwards, too - not 6-7 months between breaking changes.

If we like our current release cadence (and I mostly do), then it ought to be every 3 releases, because that will put us at 6-7 months between breaking changes. Or maybe every 4, which feels like a nicer number - but 5 is definitely too many IMO.

Pulp/Galaxy integration meeting
#16

I tend to agree. Very often we postpone release date due to various reasons and it is rare when we release on time.

#17

Well, i think the “half a year” was just a convenient coincidence. Having the deprecations living for a little longer is probably not a big deal and should not hold us back in terms of innovation. We should stick to a rebuild-side-by-side model for big refactoring changes anyway.
Having said that, i still vote for 5 releases.

#18

@dralley thanks for the analysis; it’s more accurate than my guessing for sure.

Even with it being maybe 9-10 months, I’m still in favor of 3.25 as the next one because I think we benefit from a longer compatibility cycle. I’m also in favor of “every release divisible by 5” so 3.25, 3.30, 3.35, … , etc because it’s simple.

1 Like
Need to reduce the number of release branches (aka we need an LTS strategy)
#19

I feel like we ought to build in an assumption, then, that the releases with breaking changes (e.g. 3.20) will take significantly longer - ~10-16 weeks instead of ~8. Because otherwise we’re just not going to be able to pay down the technical debt at a reasonable pace - especially if we have to maintain parallel implementations of massive bits of core functionality (which I’m a bit skeptical of us doing, because it feels like a great way to unintentionally introduce new bugs accidentally, or result in failing to fix bugs properly in both places, or spend an inordinate amount of time writing different patches for each side).

The extra time would also reduce the need for said parallel implementations.

#20

Just to followup, through the pulpcore meetings it’s been established that 3.25 is the “next” breaking change release.

2 Likes