Pulpcore Meeting Minutes

Minutes
#65

December 13, 2022

  • [decko] reviewed/approved 3427 (HStore)
  • Zero-downtime next steps
  • 3190 (domains)
    • discussion ensues
    • AI: [gerrod] run all/many plugins’ CI tests against the PR
    • who are the SMEs?
      • gerrod
      • RTFM group?
      • gerrod’s presentation makes it possible for everyone to understand
    • until we have confident LG2M from confident SMEs, we should maybe release core/3.22 sans domains, then 3.23 w/ this
  • core/3.22 discussion around probs caused by merged content-app pr
    • affects python and container
    • do we revert? wait for a fix? hold off 3.22?
    • need to resolve prior to a core/3.22 release
    • needs to be fixed/addressed in pulpcore - issue?
    • “fixing it” in container/python
      • fails at core/3.22 release if older plugins are installed
    • what happens if we “fix” this in plugins and backport?
      • container tinks this is supportable
      • how many active z-streams? (2 for container)
    • CI is currently broken because of this problem
      • about 1/3 tests in container
      • “some” tests broken in python
    • reverting the two commits fixes CI - but removes the major drivers for core/3.22 being date-driven
      • revert-commit-msg needs to explain WHY we’re reverting
    • need to have something in place for core/3.22 and pulp_rpm
    • we need to expand on pulpcore’s interface in this area, in a way that plugins can reliably take advantage of it - for core/3.25
    • AI: this needs A LOT of discussion at OpenFloor
#66

December 20, 2022

  • “stream” test is really unreliable
    • we need to fix this problem, or disable the unreliable tests - they teach us to “ignore red”
  • Upstream high-value tickets, should we prio-list them?
  • ggainey out 27-DEC - cancel the mtg next week? or appoint a facilitator?
    • AI: ggainey to cancel next week
  • gerrod becomes facilitator for Jan-Feb
    • prob should not be “optional” for those months :slight_smile:
    • AI: ggainey to voluntold gerrod
#67

Jan 3, 2023

  • “stream” test is really unreliable
    • we need to fix this problem, or disable the unreliable tests - they teach us to “ignore red”
    • [team] decided to keep the stream job on and leave this work for Michal, as planned. The bug is sporadic so should not have high impact
  • Upstream high-value tickets, should we prio-list them?
  • [ttereshc] added Mike and Humberto as optional attendees when and if they have some questions or need clarifications on pulp architecture, or pulpcore related topics, which are easier to bring up over a call.
#68

January 10

  • review Cheat sheet for the pulpcore release shepherd - HackMD
    • [AI] Dennis to bring this up in the deployers meeting
  • Galaxy[Bruno] - Add a ContentLabel model
    • Most likely implement in pulp_ansible for now, hard to be generic enough for all plugins
    • Could eventually replace AnsibleCollectionDeprecated
    • Should probably be similar in implementation to pulp_container Tags
    • ContentMark is favorite name
  • issues with pulp-in-one-container image https://github.com/pulp/pulpcore/issues/3492
    • Dennis to continue following up with poster on the issue
#69

January 17

#70

January 24

  • [dkliban] to update pulpcore release docs to include OCI images release information
  • Pulp replica discussion
    • fun with repo-attached Distributions and auto-publish
#71

January 31

#72

February 14

  • Faster release schedule (and no more waiting on blockers) is needed if we want our cloud deployment downstreams to not deploy from the main branch.

    • Are migrations guaranteed to be stable between adding to main and releasing?
    • We could defer merging PRs with migrations until later in the cycle?
      • Might not be a later phase if we are releasing faster. Need to review faster
    • Goal is to release often so we don’t run into a scenario where we want to release a plugin that is waiting on a pulpcore release
      • Cloud services might end up using pulpcore from main anyway
        • But they may also consume pulp as a library from only ga versions.
    • Release every week?
      • Y release if features present, else will be a z-stream release
      • What about bugfixes? How does this impact backporting?
      • Could be a backport nightmare, need to have unofficial LTS versions that we get products to share
      • Need to choose a new larger future compatible pulpcore version for plugins to declare compatibility with, maybe 10 versions: 3.35?
        • Potentially change pulpcore versioning to date base format?
      • [AI] dkliban to write a discourse post discussing how many new versions we want to declare future compatibility against

  • [lmjachky] Review of https://github.com/pulp/pulpcore/issues/3429

    • Again, my take on this is that we can mark almost all of the features in tech preview as production ready NOW.
    • Features in tech preview
      • Based on the comments, we have direct feedback from users who request features. Why do we need analytics first in order to mark the features as production ready?

  • Let’s use merge queues for Pulpcore

  • Stream runner (analysis)

  • Replication PR

#73

February 21, 2023

#74

February 28, 2023

  • Upgrading Django, 4.2 (LTS) is coming in April
    • 3.2 upstream support is till Dec 1, 2023.
    • Any known changes which affect us?
    • AAP is asking for 4.1 even now, pushing back to wait for 4.2
    • With django 4.* and django-channels, we may be able to consolidate the codebase (sync and async parts) on django.
    • Do migrations break with this change?
    • SUMMARY: We want to switch to 4.2 as soon as possible. Pulpcore 3.25 is the best opportunity. If we find FTE, someone should experiment with 4.1 asap.
    • [AI DKliban] Look for an issue; add it to 3.25 blockers.
    • [AI DKliban] advertise the Django change on Discourse.
  • crazy idea time: Can we turn Upload into a content type?
    • Chunks are really “just” artifacts.
    • We’d get orphan cleanup for free.
    • One less type of object to store in the storage bucket.
    • [AI MDellweg] Write a story
  • Open Telemetry update
    • Do we want this to be feature flagged?
      • YES
    • How do you want it turned on/off?
      • Question of deployment
    • Should it be on-by-default?
    • Can we make it an optional dependency?
    • [AI BBouters] Present <= 10’ about telemetry
  • What is the process to add metrics to analytics?
1 Like
#75

March 7, 2023

  • Can we get a Pulp/Katello Integration Mtg representative to cover dkliban’s leave?
    • ggainey is It
    • [AI] ggainey to suggest every-other-week for this mtg
  • core/3.23 - release when?
    • domains PR almost merged - today
    • pulp-cli/glue will have a release
    • then pulpcore
    • then pulp_file
  • calVer recap
    • current thinking: let’s just release faster and not switch to calver right now
    • not possible anyway before 3.25
  • Next breaking change Release after 3.25?
  • Analytics proposal
1 Like
#76

March 14, 2023

  • release core/3.23 today?

    • yes please!
    • AI: ggainey to turn the release crank today

  • moved some AIs from dkliban/bmbouter to decko (lucky man)

  • opentelemetry scope-of-work discussion

    • upstream user-interest/conversations
      • talk to decko if you want to be involved
    • developers
    • Scope of Work
      • Develop and merge OTEL instrumentation in pulpcore
        • code-changes only in pulpcore-workers
        • autoinstrumenting handling Other Parts (wsgi, aiohttp, postgres)
      • Write user-facing pulpcore documentation on how to configure metrics and tracing
      • Ensure oci-env has a profile that fully sets up prometheus, jaeger, grafana, and collector
      • Record 5-minute demo showing metrics collection with Grafana and Prometheus (pulp-administrator-facing)
      • Record 5-minute demo showing tracing with Jaeger (developer-facing)
      • demos need to be sent to youtube-admin and then publicized

  • open telemetry performance update

  • open telemetry dependency for pulp

    • would include the various autoinstrumentation pkgs (postgres, redis, wsgi, etc) and utilities
    • prob ~10ish new pkgs
    • decision needs to be made on whether pulp requires these “always” or “optionally”
      • needs discussion and decision
    • even if deps are present, maybe env-var to turn on/off?
    • is operator interested?
      • yes - primarily worker-metrics, for scaling decisions
      • kubernetes envs will handle scaling-due-to-hardware-issues (ie, CPU-utilization)

  • Can this be reviewed/merged before 3.23 release?

  • What about this one:

  • observation/evaluation of https://analytics.pulpproject.org/ status

    • discussion around understanding why so many “old” core-versions
    • should/can we think about which “downstream” is reporting analytics?
    • lots of discussion around how to interpret results
2 Likes
#77

March 21, 2023

  • Can we delete all the nightly published bindings from pypi and rubygem?
    • I do not see any way they are useful.
    • There are a lot.
    • At best they are compatible with a specific commit per day.
    • AI: [ggainey] bring up at katello integration mtg
    • AI: [somebody] PyPi/rubygem.org needs to be cleaned up
  • AI: [somebody] get quba42 access to pulp_deb & pulp-cli-deb & pulp-glue-deb on pypi
  • AI: [bmbouter]
  • We need to bump the pulpcore version in pulp_file (and pulp-certguard) and perform a backport/release
  • please review + merge + release this PR for pulp_file
    • an ask from @bmbouter to allow upstream testing of the 3.23 replication feature
  • file/1.13 and lower-bounds
    • create a PR by hand to update release branch
    • yank 1.13.0 from pypi/rubygem
  • discussion around PyPI permissions
    • Fun!
  • why is decko not marked as a “Developer” on discourse? Let’s fix!
  • CI Update is stuck on certguard tests not being domain compatible
    • domains-and-certguard-tests doesn’t work currently
    • certguard isn’t pytest-style tests yet
    • let’s separate concerns - do 2 CI changes
  • discussion: should we be shipping tests in the pypi installs?
    • prob not “expected” behavior
    • currently we do
    • originally done to support django-unit-tests (which expect things to be part of the app)
    • want tests to be runnable in pulp-users’ integration CIs
  • discussion around Django/4 investigation
  • discussion around pulling together an Official List of who has access to what/clean house
1 Like
#78

March 28, 2023

  1. Do we want to run the test suite in the release pipeline?
    • we run it before merging every commit
    • thoughts: we want faster releases, and we can always put this back if it starts causing us issues
    • AI: [mdellweg] remove it
  2. Do we want to run the lowerbounds CI check with == or ~= pins?
  3. Can we raise the lower bound (Z version) of a requirement in a plugin when releasing a bug fix?
    • pulpcore Z version bump when releasing a plugin
    • regression testing assessment (problems with accepting bug fixes)
  4. More friendly docs needed for certguard?
  5. Are we ready to stop publishing anything nightly?
    • we don’t publish clients any more, do publish docs
    • docs updated at release-time
    • changelog updated when that file changes
    • w/out nightly-docs, need to point into github directly (which is ok)
    • what is generating /latest/ ?
      • need to make sure that works first
    • AI: [mdellweg] investigate and stop doing nightly-publish if we can
1 Like
#79

April 4, 2023

  1. Django 4.2 released
    • Possibility to upgrade to Psycopg3
    • ASGI vs WSGI
    • decko is working on OTEL :frowning:
    • Must to line up with 3.25, and “before August” to meet a stakeholder ask
    • Make Postgres/12+ required as well
    • do we have a prio-list issue?
    • x9c4 to pick up the work?
  2. When to release z-streams
    • If there’s anything unreleased on a Tuesday, Push The Button
  3. Release pulpcore?
  • only z-releases
  1. Release pulp_file?
  • pytest plugin PR needs a release
  • release 1.14 to get PulpReplicator out
1 Like
#80

April 11

  • Circular CI testing dependencies
  • List endpoint optimizations
  • [ggainey] pulpcore z-releases today
    • 3.23.2, 3.22.4, 3.21.7, 3.18.17, 3.16.17
    • let ggainey know if he should wait, before noon-GMT-5 today
  • [decko] pulp-maven permissions
    • need github perms to at least re-run actions
    • same as merge-pull-request perms
    • who has access? - ttereshc to fix
    • AI: [ggainey] update our permissions-matrix so we have some idea of who can do what, where
1 Like
#81

April 18

  • Weekly release check-in
    • pulpcore
      • [ ] 3.24
      • [X] 3.23
      • [ ] 3.22
      • [ ] 3.21
      • [ ] 3.18
      • [ ] 3.16
    • pulp_file
      • [X] 1.14
  • Django 4.2 updates
  • move from Twitter to mastodon? (fosstodon.org maybe?)
    • AI: [ggainey] bring up at Team mtg next week
  • https://github.com/pulp/pulpcore/issues/3710 needs discussion
    • Labels? - not available at repo-version level
      • maybe start w/ Labels, and see how far we can get?
    • AH/repo-mgt - already talked about repo-group concept?
      • Pulp2 had something like
    • What about repo-version being a piece of content that could be held by a repository? (RepoCeption!)
      • need a content-object that is-a repo-version?
    • plugin implications?
  • Docs issue in plugin CI’s
    • plugin docs-build (pulp_file for sure)
    • we install pulp “outside” the container
    • conflicts w/ cli/glue, pip-installing, plugin-vs-core docs-requirements conflicts
    • have a bandaid in-place(ish) for core
    • need some thought on how to fix the problem “appropriately”
    • root-cause(ish): plugin-doc-build requires pulpcore-doc-requirements-install, and then plugin-doc-requirements-install, and core can therefore break the plugins w/out knowing
    • AI: [mdellweg] to investigate better workaround in template
    • AI: [ggainey] report back on any breakage in today’s pulp_file release (attempt)
1 Like
#82

April 25

2 Likes
#83

[missed posting some minutes, catching up here]

May 2nd

  • Weekly release check-in
    • pulpcore
      • [X] 3.24
      • [X] 3.23
      • [ ] 3.22
      • [ ] 3.21
      • [ ] 3.18
      • [ ] 3.16
    • pulp_file
      • [ ] 1.14
  • core/3.24
  • Let’s nail down what cadence we will use as “number of Y-releases without breaking changes”
    • see previous notes Pulpcore team meeting - HackMD
    • suggestion: core/3.40 (15 versions post-3.25)
      • “probably” six-months-ish
      • see how well this works/revisit at that point
  • With the new release policy, is it still useful to have tests for new pulpcore features living in pulp_file only?
    • In the long run, it does not matter if the tests are in pulpcore or pulp_file.
    • “Test should come with the feature” suggests adding them to pulpcore if it’s a pulpcore only change.
    • What was the original reason to move tests using pulp_file to the plugin?
      • mdellweg’s memory: We had compatibility issues with all the branches moving around. But we fixed that and can now depend on stable pulp_file releases.
    • if there is any code-change in pulp_file for a Thing, then its test needs to live in pulp_file
    • if there is no pulp_file code-change (e.g., core is adding a Thing that pulp_file gets “for free” due to inheritance), then the test for it “can” live in pulpcore
    • this own’t solve the problem - pulp_file changes that require core-changes still collide oddly/badly
    • what about fixtures? can you use pulp_file fixtures from inside pulpcore?
      • yes if pulp_file is a pytest plugin
    • discussion: should core and file be the same project?

May 9

  • pulpcore
    • [X] 3.25 (NEW)
    • [ ] 3.24
    • [X] 3.23
    • [ ] 3.22
    • [ ] 3.21
    • [ ] 3.18
    • [ ] 3.16
  • pulp_file
    • [ ] 1.15 (NEW)
    • [ ] 1.14
    • [ ] 1.13
    • [ ] 1.12
    • [ ] 1.11
    • [ ] 1.10
  • Issue discussion
    • https://github.com/pulp/pulp_rpm/issues/3134
    • add a repo-versions__in
    • how can we make this performant?
    • filters could use perf-analysis/fixes
    • needs a POC to understand implications
      • materialized views? (django? or postgres?)
  • A proposal from telemetry workgroup
    • current state
      • Dependencies declared in the pulp-ci-centos build
      • Code missing in the pulpcore’s main branch. We can’t merge the code if we don’t add the opentelemetry basic stack as a direct dependency on pulpcore's requirements.
      • A new PR for OpenTelemetry aiohttp-server instrumentation project.
        • needed for pulp-content instrumentation
      • pulp-api is in a great spot
      • aiohttp is pretty good, but we really want it released upstream to consume
      • nothing for tasking yet - phase-2 perhaps?
      • Got a oci-env profile for development purposes.
        • Still not merged given some instabilities with CI’s docker tests.
          • need some help/eyes
    • The new proposal
      • Given the new pulpcore release process
        • “We can undo it later if needed”
      • Phase 1
        • We want to add a trimmed opentelemetry stack list of dependencies directly into pulpcore.
          • opentelemetry-distro[otlp]>=0.38b0,<=0.38b0
          • opentelemetry-exporter-otlp-proto-http>=1.17.0,<=1.17.0
          • opentelemetry-instrumentation-django>=0.38b0,<=0.38b0
          • opentelemetry-instrumentation-wsgi>=0.38b0,<=0.38b0
        • We have a PR open for instrumenting pulp-api
        • We are about to open a PR for opentelemetry-instrumentation-aiohttp-server
        • Will have a PR for instrumenting pulp-content after getting the otel PR merged
          • We can release a package with the instrumentator if the PR got stalled for some reason(ex: got no reviewers)
          • Or we can add it directly into pulpcore code base(vendor it?)
      • Phase 2
        • instrument the workers?
        • Any specific pulp related code?
    • discussion
    • decision:
      • merge reqs/PR to pulpcore
      • merge otel-work in oci_env
      • release as soon as it’s merged (3.25? .26?)
  • difficulty in getting core-PRs reviewed
    • think about this for next week’s mtg
  • 3.25 Today?
  • Can we make a bot or something to tell us when there are un-released commits on our branches
    • this would be great!
    • maybe a cmd to be run on a local copy of the repo?
    • a tues-workflow to check branches and auto-release?
    • let’s gather some info and anyone who’s Very Excited about this can start pulling something together
  • discussion around drop-trailing-slash PR
    • gerrod’s refactoring would make this easier to implement
    • do we hold up 3.25 for this, or not
    • if we’re going to change this to opt-in, it can happen “whenever”
    • decision: remove from blocker-list, release core/3.25
1 Like
#84

May 16

  • pulpcore

    • [ ] 3.26 (NEW)
    • [ ] 3.25 (current release)
    • [ ] 3.23 (galaxyNG/4.7)
    • [ ] 3.22 (katello/4.9)
    • [ ] 3.21 (katello/4.7, galaxyNG/4.6)
    • [ ] 3.18 (katello/4.5)
    • [ ] 3.16 (katello/4.3)

  • pulp_file

    • [ ] 1.15 (NEW)
    • [ ] 1.14 (current)
    • [ ] 1.12 (katello/4.9)
    • [ ] 1.11 (katello/4.7)
    • [ ] 1.10 (katello/4.3. 4.5)
  • difficulty in getting core-PRs reviewed
    • think about this for next week’s mtg
    • “should we continue requiring 2 reviewers”?
      • still ok, except when everyone is on PTO?
      • problem w/ finding 2 SMEs at all times, moreso than github-access-issue
      • even just “extra eyes” (ie, not SME eyes) is useful
  • “keeping branch warm” discussion
    • remove 1.13 from pulp_file update_ci_branches? - YES
    • remove 3.24 from pulpcore update_ci_branches? - Yes(ish)
    • “whatever is used in downstream” and “currently-released branch”
    • what will be the implications for our upstream?
      • will they just stay on “downstream” branches (making them even more like an LTS?)
    • should core/3.24 be an exception?
      • if so - we can add it when we decide we need to
    • We should delete the backport labels for cold branches.
    • AI: [ggainey] make sure release-process is updated to be explicit about this
  • Create a working-group to improve CI automation to support weekly releases
    • Still too many manual steps in release process, quite a burden
    • only dkliban and x9c4 on the “CI Team” currently
    • still many manual-steps at Y-release time
    • lots of improvements discussed, not enough ppl to pick them up
    • let’s get a doc together and add ideas to it
    • who would like to champion for this?
      • gerrod to helm this
      • volunteers: ggainey, decko, x9c4
  • How to handle this migration:
  • This PR lacks a changelog and issue
  • OpenTelemetry is beta - discussion
    • there was a LONG matrix discussion on this
    • lots of other projects use this already
    • in both doc and in code explain why we’re relying on a “beta” dependency [decko]
    • [decko] will add missing changelog in his current PR
    • let’s make sure we continue to be good about having discussions exactly like this one
    • we should be asking the Otel Group directly, “when are you going to be not-beta”
1 Like