Pulpcore Meeting Minutes

Minutes
#61

November 29, 2022

  • Are we going to actually pay attention to the CodeQL output?
  • [davidd] Pulp’s policy around settings changes https://github.com/pulp/pulpcore/issues/3417#issuecomment-1329659503
    • We set a number of settings and are concerned about settings changes
    • If settings are not covered by semver, could there be a documented policy/deprecation cycle?
    • proposal: settings should follow same semver policy as REST API
      • i.e.: can deprecate, but not remove
      • each rename/removal is Going To Be A Pain - “please don’t” is the generic dev policy
    • proposal: doc settings.local more in docs?
  • pulp_labels
    • https://github.com/pulp/pulpcore/pull/3427
    • currently exposed for distributions, repositories, remotes (so far)
    • jsonb is overkill
    • would need to remove generic-relation
    • exposes some of the functionality in the API
    • breaking change?
      • plugin API affected
      • need to research if anyone is actively “aware of” the current implementation
      • marked tech-preview - so (technically) “can do anything”
      • OTOH - we’ guarantee that plugin-breaking-changes won’t happen until breaking-change releases (e.g. 3.25)
    • Base model introduction would require Very Complicated Migration
    • REST API unaffected
      • poss bug in current REST
  • pulpcore 3.22 GA is postponed
    • all thumbs-up
  • tech-preview discussion
    • we have a tech-preview problem
    • just because we say “we can break this”, doesn’t make it even remotely a good idea
    • how long can/should something say “tech-preview”
    • can we move to “how much is it used”?
      • analytics to the rescue!
#62

December 6, 2022

  • Proposal: have pulpcore adopt zero-downtime migration policy
  • Proposal: give decko write-perms to pulpcore
    • previously, generally waited a year/major feature addition as SME to add
    • feels like we need to give it decko now to allow reviews for pytest work to “count”
    • AI: [dkliban] add decko to pulpcore team
  • remove ppicka invitation/commit bit since January
    • AI: [bmbouter] to remove from mtg invite
    • AI: [team] move ppicka to Contributor
  • Should settings be semver controlled?
    • dkliban: leaning yes
    • bmbouter: users are asking for it, we should give it to them
      • still can do “emergency” changes (e.g. security issues) if we need to under semver
    • x9c4: can we get away with not-defining and deal w/ it case-by-case
      • since semver allows for emergency fixes, prob ok
    • ggainey: also leaning yes
    • mdellweg: we can decide for pulpcore, but not for Everyone
      • maybe we should put up in discourse for a week for discussion
    • bmbouter: transparency is good
    • AI: [x9c4] open thread in discourse
    • AI: [all] revisit and confirm at next mtg
  • Labels PR is ready and needs review/merge: https://github.com/pulp/pulpcore/pull/3427
    • do we want this in 3.22?
      • it’s a bugfix
      • merge it - “waiting in main doesn’t actually help”
    • 3.22 is 12-DEC - so review ASAP
    • AI: [team] get 3427 reviewed so we can merge no later than Fri
  • Q: how easy is it to create a new release?
    • Proposal: we should do releases more often
    • plugins that depend on core/main can’t release until that main becomes a ‘real’ release
    • Proposal: let’s start by releasing quickly (two weeks? three?)
      • also - no go/nogo
      • strictly time-based
      • Y-release ONLY if there are features
      • required-PR has Implications
      • what about backport policy?
        • prob OK, for multiple reasons
  • https://github.com/pulp/pulpcore/issues/3368
    • can we make this a prio-list? - yes

Pending AIs

  • [bmbouter] Analytics process needs to be in our developer docs
  • [dkliban] CodeQL issues opened
  • zero-downtime actions
    • AI: [bmbouter] open a doc-issue
    • AI: [bmbouter] write up plugin docs for this
  • ppicka is leaving us :’(
    • AI: [bmbouter] to remove from mtg invite
    • AI: [team] move ppicka to Contributor
  • settings controlled by semver
    • AI: [x9c4] open thread in discourse
    • AI: [all] revisit and confirm at next mtg
  • AI: [team] get 3427 reviewed so we can merge no later than Fri
1 Like
#63

FYI I gave write perms to Lubos as well, he is part of pytest group.

1 Like
#64

Opened the ticket for CodeQL issues

1 Like
#65

December 13, 2022

  • [decko] reviewed/approved 3427 (HStore)
  • Zero-downtime next steps
  • 3190 (domains)
    • discussion ensues
    • AI: [gerrod] run all/many plugins’ CI tests against the PR
    • who are the SMEs?
      • gerrod
      • RTFM group?
      • gerrod’s presentation makes it possible for everyone to understand
    • until we have confident LG2M from confident SMEs, we should maybe release core/3.22 sans domains, then 3.23 w/ this
  • core/3.22 discussion around probs caused by merged content-app pr
    • affects python and container
    • do we revert? wait for a fix? hold off 3.22?
    • need to resolve prior to a core/3.22 release
    • needs to be fixed/addressed in pulpcore - issue?
    • “fixing it” in container/python
      • fails at core/3.22 release if older plugins are installed
    • what happens if we “fix” this in plugins and backport?
      • container tinks this is supportable
      • how many active z-streams? (2 for container)
    • CI is currently broken because of this problem
      • about 1/3 tests in container
      • “some” tests broken in python
    • reverting the two commits fixes CI - but removes the major drivers for core/3.22 being date-driven
      • revert-commit-msg needs to explain WHY we’re reverting
    • need to have something in place for core/3.22 and pulp_rpm
    • we need to expand on pulpcore’s interface in this area, in a way that plugins can reliably take advantage of it - for core/3.25
    • AI: this needs A LOT of discussion at OpenFloor
#66

December 20, 2022

  • “stream” test is really unreliable
    • we need to fix this problem, or disable the unreliable tests - they teach us to “ignore red”
  • Upstream high-value tickets, should we prio-list them?
  • ggainey out 27-DEC - cancel the mtg next week? or appoint a facilitator?
    • AI: ggainey to cancel next week
  • gerrod becomes facilitator for Jan-Feb
    • prob should not be “optional” for those months :slight_smile:
    • AI: ggainey to voluntold gerrod
#67

Jan 3, 2023

  • “stream” test is really unreliable
    • we need to fix this problem, or disable the unreliable tests - they teach us to “ignore red”
    • [team] decided to keep the stream job on and leave this work for Michal, as planned. The bug is sporadic so should not have high impact
  • Upstream high-value tickets, should we prio-list them?
  • [ttereshc] added Mike and Humberto as optional attendees when and if they have some questions or need clarifications on pulp architecture, or pulpcore related topics, which are easier to bring up over a call.
#68

January 10

  • review Cheat sheet for the pulpcore release shepherd - HackMD
    • [AI] Dennis to bring this up in the deployers meeting
  • Galaxy[Bruno] - Add a ContentLabel model
    • Most likely implement in pulp_ansible for now, hard to be generic enough for all plugins
    • Could eventually replace AnsibleCollectionDeprecated
    • Should probably be similar in implementation to pulp_container Tags
    • ContentMark is favorite name
  • issues with pulp-in-one-container image https://github.com/pulp/pulpcore/issues/3492
    • Dennis to continue following up with poster on the issue
#69

January 17

#70

January 24

  • [dkliban] to update pulpcore release docs to include OCI images release information
  • Pulp replica discussion
    • fun with repo-attached Distributions and auto-publish
#71

January 31

#72

February 14

  • Faster release schedule (and no more waiting on blockers) is needed if we want our cloud deployment downstreams to not deploy from the main branch.

    • Are migrations guaranteed to be stable between adding to main and releasing?
    • We could defer merging PRs with migrations until later in the cycle?
      • Might not be a later phase if we are releasing faster. Need to review faster
    • Goal is to release often so we don’t run into a scenario where we want to release a plugin that is waiting on a pulpcore release
      • Cloud services might end up using pulpcore from main anyway
        • But they may also consume pulp as a library from only ga versions.
    • Release every week?
      • Y release if features present, else will be a z-stream release
      • What about bugfixes? How does this impact backporting?
      • Could be a backport nightmare, need to have unofficial LTS versions that we get products to share
      • Need to choose a new larger future compatible pulpcore version for plugins to declare compatibility with, maybe 10 versions: 3.35?
        • Potentially change pulpcore versioning to date base format?
      • [AI] dkliban to write a discourse post discussing how many new versions we want to declare future compatibility against

  • [lmjachky] Review of https://github.com/pulp/pulpcore/issues/3429

    • Again, my take on this is that we can mark almost all of the features in tech preview as production ready NOW.
    • Features in tech preview
      • Based on the comments, we have direct feedback from users who request features. Why do we need analytics first in order to mark the features as production ready?

  • Let’s use merge queues for Pulpcore

  • Stream runner (analysis)

  • Replication PR

#73

February 21, 2023

#74

February 28, 2023

  • Upgrading Django, 4.2 (LTS) is coming in April
    • 3.2 upstream support is till Dec 1, 2023.
    • Any known changes which affect us?
    • AAP is asking for 4.1 even now, pushing back to wait for 4.2
    • With django 4.* and django-channels, we may be able to consolidate the codebase (sync and async parts) on django.
    • Do migrations break with this change?
    • SUMMARY: We want to switch to 4.2 as soon as possible. Pulpcore 3.25 is the best opportunity. If we find FTE, someone should experiment with 4.1 asap.
    • [AI DKliban] Look for an issue; add it to 3.25 blockers.
    • [AI DKliban] advertise the Django change on Discourse.
  • crazy idea time: Can we turn Upload into a content type?
    • Chunks are really “just” artifacts.
    • We’d get orphan cleanup for free.
    • One less type of object to store in the storage bucket.
    • [AI MDellweg] Write a story
  • Open Telemetry update
    • Do we want this to be feature flagged?
      • YES
    • How do you want it turned on/off?
      • Question of deployment
    • Should it be on-by-default?
    • Can we make it an optional dependency?
    • [AI BBouters] Present <= 10’ about telemetry
  • What is the process to add metrics to analytics?
1 Like
#75

March 7, 2023

  • Can we get a Pulp/Katello Integration Mtg representative to cover dkliban’s leave?
    • ggainey is It
    • [AI] ggainey to suggest every-other-week for this mtg
  • core/3.23 - release when?
    • domains PR almost merged - today
    • pulp-cli/glue will have a release
    • then pulpcore
    • then pulp_file
  • calVer recap
    • current thinking: let’s just release faster and not switch to calver right now
    • not possible anyway before 3.25
  • Next breaking change Release after 3.25?
  • Analytics proposal
1 Like
#76

March 14, 2023

  • release core/3.23 today?

    • yes please!
    • AI: ggainey to turn the release crank today

  • moved some AIs from dkliban/bmbouter to decko (lucky man)

  • opentelemetry scope-of-work discussion

    • upstream user-interest/conversations
      • talk to decko if you want to be involved
    • developers
    • Scope of Work
      • Develop and merge OTEL instrumentation in pulpcore
        • code-changes only in pulpcore-workers
        • autoinstrumenting handling Other Parts (wsgi, aiohttp, postgres)
      • Write user-facing pulpcore documentation on how to configure metrics and tracing
      • Ensure oci-env has a profile that fully sets up prometheus, jaeger, grafana, and collector
      • Record 5-minute demo showing metrics collection with Grafana and Prometheus (pulp-administrator-facing)
      • Record 5-minute demo showing tracing with Jaeger (developer-facing)
      • demos need to be sent to youtube-admin and then publicized

  • open telemetry performance update

  • open telemetry dependency for pulp

    • would include the various autoinstrumentation pkgs (postgres, redis, wsgi, etc) and utilities
    • prob ~10ish new pkgs
    • decision needs to be made on whether pulp requires these “always” or “optionally”
      • needs discussion and decision
    • even if deps are present, maybe env-var to turn on/off?
    • is operator interested?
      • yes - primarily worker-metrics, for scaling decisions
      • kubernetes envs will handle scaling-due-to-hardware-issues (ie, CPU-utilization)

  • Can this be reviewed/merged before 3.23 release?

  • What about this one:

  • observation/evaluation of https://analytics.pulpproject.org/ status

    • discussion around understanding why so many “old” core-versions
    • should/can we think about which “downstream” is reporting analytics?
    • lots of discussion around how to interpret results
2 Likes
#77

March 21, 2023

  • Can we delete all the nightly published bindings from pypi and rubygem?
    • I do not see any way they are useful.
    • There are a lot.
    • At best they are compatible with a specific commit per day.
    • AI: [ggainey] bring up at katello integration mtg
    • AI: [somebody] PyPi/rubygem.org needs to be cleaned up
  • AI: [somebody] get quba42 access to pulp_deb & pulp-cli-deb & pulp-glue-deb on pypi
  • AI: [bmbouter]
  • We need to bump the pulpcore version in pulp_file (and pulp-certguard) and perform a backport/release
  • please review + merge + release this PR for pulp_file
    • an ask from @bmbouter to allow upstream testing of the 3.23 replication feature
  • file/1.13 and lower-bounds
    • create a PR by hand to update release branch
    • yank 1.13.0 from pypi/rubygem
  • discussion around PyPI permissions
    • Fun!
  • why is decko not marked as a “Developer” on discourse? Let’s fix!
  • CI Update is stuck on certguard tests not being domain compatible
    • domains-and-certguard-tests doesn’t work currently
    • certguard isn’t pytest-style tests yet
    • let’s separate concerns - do 2 CI changes
  • discussion: should we be shipping tests in the pypi installs?
    • prob not “expected” behavior
    • currently we do
    • originally done to support django-unit-tests (which expect things to be part of the app)
    • want tests to be runnable in pulp-users’ integration CIs
  • discussion around Django/4 investigation
  • discussion around pulling together an Official List of who has access to what/clean house
1 Like
#78

March 28, 2023

  1. Do we want to run the test suite in the release pipeline?
    • we run it before merging every commit
    • thoughts: we want faster releases, and we can always put this back if it starts causing us issues
    • AI: [mdellweg] remove it
  2. Do we want to run the lowerbounds CI check with == or ~= pins?
  3. Can we raise the lower bound (Z version) of a requirement in a plugin when releasing a bug fix?
    • pulpcore Z version bump when releasing a plugin
    • regression testing assessment (problems with accepting bug fixes)
  4. More friendly docs needed for certguard?
  5. Are we ready to stop publishing anything nightly?
    • we don’t publish clients any more, do publish docs
    • docs updated at release-time
    • changelog updated when that file changes
    • w/out nightly-docs, need to point into github directly (which is ok)
    • what is generating /latest/ ?
      • need to make sure that works first
    • AI: [mdellweg] investigate and stop doing nightly-publish if we can
1 Like
#79

April 4, 2023

  1. Django 4.2 released
    • Possibility to upgrade to Psycopg3
    • ASGI vs WSGI
    • decko is working on OTEL :frowning:
    • Must to line up with 3.25, and “before August” to meet a stakeholder ask
    • Make Postgres/12+ required as well
    • do we have a prio-list issue?
    • x9c4 to pick up the work?
  2. When to release z-streams
    • If there’s anything unreleased on a Tuesday, Push The Button
  3. Release pulpcore?
  • only z-releases
  1. Release pulp_file?
  • pytest plugin PR needs a release
  • release 1.14 to get PulpReplicator out
1 Like
#80

April 11

  • Circular CI testing dependencies
  • List endpoint optimizations
  • [ggainey] pulpcore z-releases today
    • 3.23.2, 3.22.4, 3.21.7, 3.18.17, 3.16.17
    • let ggainey know if he should wait, before noon-GMT-5 today
  • [decko] pulp-maven permissions
    • need github perms to at least re-run actions
    • same as merge-pull-request perms
    • who has access? - ttereshc to fix
    • AI: [ggainey] update our permissions-matrix so we have some idea of who can do what, where
1 Like