Need to reduce the number of release branches (aka we need an LTS strategy)

Development
#1

We have a lot of “release branches”, e.g. with pulpcore there is 3.14, 3.15, 3.16, 3.17, 3.18, 3.19, and main. For each of these branches we have the responsibility of:

  • backporting a fix to every branch in-between
  • ensuring the plugin template is always up to date (which changes daily now)
  • releasing all versions at once so the bugfix is always available when upgrading to any newer version

Across the project, we likely have 30ish release branches, and with the CI taking 30-50 minutes for each backport on each branch, backport of the plugin template, or release, a huge amount of our time is tied up in release engineering.

We need to find a simpler way as I believe we will soon have a CI that is so clogged up with these types of jobs we can’t make normal dev changes at a reasonable rate. I suspect we are very close to that point or we’ve already reached it.

So I propose we apply the textbook solution to this problem and use an Long Term Support (LTS) branch so that for each project we only have two branches to maintain, main and the release branch. Each plugin would figure out for itself what the LTS version should be.

To transition into this type of arrangement we would:

  1. Designate one LTS branch per component, e.g. pulpcore, pulp_rpm, etc
  2. Have the plugin template auto-apply machinery apply to that LTS branch and main only (instead of everything in between)
  3. We only continue backporting and releasing to the LTS

Like most projects, e.g. Django that declare an LTS, we would declare a new one in the future and expect the new LTS to replace the old one. Those users who want to stay with the LTS can upgrade from one LTS directly to the next, e.g. how Pulp itself upgraded from Django 2.2.z (the old LTS) to Django 3.2.z (the current Django LTS).

I know there are many reasons why we probably “can’t do this” however I believe we cannot continue the direction and policies we currently have. This will be a controversial and challenging change, so feedback on it is very welcome.

2 Likes
#2

I think only supporting one release isn’t tenable, but I completely agree that supporting every single release branch is not going to work well for us long-term, for the reasons you point out. We will have to continue supporting 3.14 for downstream consumers for another 6 months minimum, that means our pipeline of supported releases will be 9-10 releases deep if we continue down this path, and it’s just not realistic.

Designating one LTS release every 6-8 months seems doable. Let’s say 3.14, 3.18, 3.22… etc.

#3

From Katello’s perspective, if one LTS needs to be chosen quickly, I’d choose 3.18. Katello 4.5 is about to be released on it (we’ve yet to upgrade but it’s in the planning). It seems that Pulpcore 3.18 has all of the features that Katello needs to run happily.

That being said though, I do really like @dralley’s idea of having multiple LTSs that are spread out. I’m sure Katello will want to integrate with new Pulp features in the future. I’m not sure how having only one LTS would work with new features in fact…

Here are the list of 3.18 plugin versions that Katello has built for the curious:

pulpcore 3.18.4
pulp-rpm 3.17.4
pulp-container 2.10.3
pulp-deb 2.18.0
pulp-file 1.10.2
pulp-ostree 2.0.0-0.5.a5
pulp-python 3.6.0
pulp-ansible 0.13.0
pulp-certguard 1.5.2

1 Like
#4

An alternative is tweaking our release policy to release slower, say every 3 months instead of every 6-10 weeks. Even numbered releases could be LTS, odd-numbered releases could be allowed to do cleanups (see other discussion).

If I can lay out what I think would look like:

  • 3.20 → plugin API changes allowed
    • (2 months - roughly 1 month from now if we keep the status-quo cadence)
  • 3.21 → plugin API changes allowed
    • (2 months)
  • 3.22 → stable, LTS for 12-18 months
    • (3 months)
  • 3.23 → plugin API changes allowed
    • (3 months)
  • 3.24 → stable, LTS for 12-18 months
    • (3 months)
  • 3.25 → plugin API changes allowed
    • (3 months)

This has the benefits that:

  1. We would be roughly sync’d up with Django LTS release cycles.
  2. Plugins that can’t develop at the same cadence as the rest would have ample time to integrate with API changes - there would be less pressure to do it because there’s always a supported LTS release.

If it doesn’t work for all stakeholders, we could even make every release LTS and it wouldn’t be too bad. ~4 supported releases wouldn’t be terrible.

#5

I think it would be workable for pulpcore to declare 3.14 and 3.18 as the current LTS branches and then add 3.22 and drop LTS support for 3.14 (to only carry two LTS branches at a time).

I think these parts of the plugin_template would need to be adjusted to only updated a comma separated set of branches. Also having the way in the github UI to trigger the workflow is great too. Having them auto-apply weekly as a time-based trigger (like it does now) would also be good.

#6

Here’s our version matrix for Galaxy NG: Galaxy NG Pulp Requirments · ansible/galaxy_ng Wiki · GitHub. All of these are currently supported, but some may be EoL soonish. I need to double check when. Unfortunately we can’t change the pulpcore version on any of these releases.

#7

After some feedback on Matrix and here, my proposal is to have pulpcore==3.18 be the current LTS, with the understanding that if another version needs a fix we’ll try to handle them on a case by case basis. This has the benefit of bundling all the necessary plugin_template changes into one PR, but has the downside that anytime a backport needs to occur to a non-LTS branch we need to bring that branch up to date with the plugin_template.

We still need to hear from RHUI to incorporate their feedback on this proposal also.

I’m trying to focus on the immediate goal of reducing the maintained branches, and then later revisit the future LTS versions convo.

1 Like
Katello/Pulp3 Integration meeting
#8

My understanding is that if we declare LTS-Releases early enough, we will simply abandon all the intermediate ones and never cherry pick anything there. Instead, we will focus on a smooth upgrade experience from one LTS to the next. And by strongly suggesting to use only the latest or LTS releases, we would be able to make these few branches even more stable.
I think, allowing the occasional bugfix on non-LTS branches will create a lot of additional workload for a worse experience on the consumer side than we have today.

#9

If this all puts too much pressure on “catching the merge window for LTS”, we can just as well release less often, which results in less release branches quite naturally.
The one thing that we would benefit from in all cases, is making the CI faster, more reliable and easier to understand/maintain.
BTW, I wonder no one has brought up the rolling release idea yet.

#10

We are using 3.17 version of Pulp and generally we update Pulp version only when there is a security issue or new functionality that we need and that usually happens when Y-stream version is released every 6 months. So going forward we are fine with using the latest LTS available at the time as long as it has all the functionality we need.