Pulpcore Meeting Minutes

Minutes
#57

September 27, 2022

#58

October 11, 2022

1 Like
#59

October 18, 2022

  • CI failures on 3.16
    • failure on the local filesystem runner
    • proposal: skip the single test because it’s preventing us from running all tests
    • observation: this pulp_rpm has a similar failure https://github.com/pulp/pulp_rpm/pull/2838
      • actually it’s different and it needs an actual code change in pulp_rpm test to fix so it’s unrelated
  • deadlock PR desperately needs another review(er) please
  • pulp_file needs some TLC
  • design discussion for: https://github.com/pulp/pulpcore/issues/2851
  • bmbouter scheduling the zero downtime working group
#60

(NOTE: apologies for these postings being behind - new meeting-facilitator skipped a groove and didna post. Catching up now.)

November 15, 2022

  • 3.22 release schedule
    • ipanova is the release shepherd
    • We’re at 2 months since 3.21
    • tentative GA date is Nov 30
    • 16 and 23 of Nov go/no-go meetings
    • what should go in?
      • domains
        • needs issue, add to 3.22 blockers [AI: gerrod]
      • blockers
  • django4/core-3.25 discussion
    • django-4 happens in April
    • 3.25 needs more time due to likelihood of breaking-changes
    • don’t want to get downstream products “stuck” on older versions that will EOL
    • “when should 3.23 and 3.24 happen”
      • IF 3.25 is going to be “django-4” - prob want to have significant burn-in
      • 3.23 - prob end-of Jan, to get in anything we want to have available for “conference season” in Feb
      • 3.24 - prob end-of March, for 2-month-cadence/pre-django-4
    • 3.25/django-4 linkage is not required - let’s “see what happens” w/ django and re-evaluate in April
  • AI: ggainey to post to discourse
  • AI: ggainey to engage in “creative rearranging” of team doc

November 8, 2022 - skipped due to PulpCON

November 1, 2022

  • Failing test related to repo version delete
    • dkliban to investigate

October 25, 2022

  • Filter-process appears to need some more refactoring
    • x9c4 is willing to do, but needs some buy-in for quick review turnarounds
    • will fix a couple of openapi-schema bugs
  • discussion ensues RE opportunity costs
    • what about new content-types?
    • how can we best prioritize competing priorities from groups that are not in agreement?
      • let’s get folk from “emerging” opportunities come talk to us directly
#61

November 29, 2022

  • Are we going to actually pay attention to the CodeQL output?
  • [davidd] Pulp’s policy around settings changes https://github.com/pulp/pulpcore/issues/3417#issuecomment-1329659503
    • We set a number of settings and are concerned about settings changes
    • If settings are not covered by semver, could there be a documented policy/deprecation cycle?
    • proposal: settings should follow same semver policy as REST API
      • i.e.: can deprecate, but not remove
      • each rename/removal is Going To Be A Pain - “please don’t” is the generic dev policy
    • proposal: doc settings.local more in docs?
  • pulp_labels
    • https://github.com/pulp/pulpcore/pull/3427
    • currently exposed for distributions, repositories, remotes (so far)
    • jsonb is overkill
    • would need to remove generic-relation
    • exposes some of the functionality in the API
    • breaking change?
      • plugin API affected
      • need to research if anyone is actively “aware of” the current implementation
      • marked tech-preview - so (technically) “can do anything”
      • OTOH - we’ guarantee that plugin-breaking-changes won’t happen until breaking-change releases (e.g. 3.25)
    • Base model introduction would require Very Complicated Migration
    • REST API unaffected
      • poss bug in current REST
  • pulpcore 3.22 GA is postponed
    • all thumbs-up
  • tech-preview discussion
    • we have a tech-preview problem
    • just because we say “we can break this”, doesn’t make it even remotely a good idea
    • how long can/should something say “tech-preview”
    • can we move to “how much is it used”?
      • analytics to the rescue!
#62

December 6, 2022

  • Proposal: have pulpcore adopt zero-downtime migration policy
  • Proposal: give decko write-perms to pulpcore
    • previously, generally waited a year/major feature addition as SME to add
    • feels like we need to give it decko now to allow reviews for pytest work to “count”
    • AI: [dkliban] add decko to pulpcore team
  • remove ppicka invitation/commit bit since January
    • AI: [bmbouter] to remove from mtg invite
    • AI: [team] move ppicka to Contributor
  • Should settings be semver controlled?
    • dkliban: leaning yes
    • bmbouter: users are asking for it, we should give it to them
      • still can do “emergency” changes (e.g. security issues) if we need to under semver
    • x9c4: can we get away with not-defining and deal w/ it case-by-case
      • since semver allows for emergency fixes, prob ok
    • ggainey: also leaning yes
    • mdellweg: we can decide for pulpcore, but not for Everyone
      • maybe we should put up in discourse for a week for discussion
    • bmbouter: transparency is good
    • AI: [x9c4] open thread in discourse
    • AI: [all] revisit and confirm at next mtg
  • Labels PR is ready and needs review/merge: https://github.com/pulp/pulpcore/pull/3427
    • do we want this in 3.22?
      • it’s a bugfix
      • merge it - “waiting in main doesn’t actually help”
    • 3.22 is 12-DEC - so review ASAP
    • AI: [team] get 3427 reviewed so we can merge no later than Fri
  • Q: how easy is it to create a new release?
    • Proposal: we should do releases more often
    • plugins that depend on core/main can’t release until that main becomes a ‘real’ release
    • Proposal: let’s start by releasing quickly (two weeks? three?)
      • also - no go/nogo
      • strictly time-based
      • Y-release ONLY if there are features
      • required-PR has Implications
      • what about backport policy?
        • prob OK, for multiple reasons
  • https://github.com/pulp/pulpcore/issues/3368
    • can we make this a prio-list? - yes

Pending AIs

  • [bmbouter] Analytics process needs to be in our developer docs
  • [dkliban] CodeQL issues opened
  • zero-downtime actions
    • AI: [bmbouter] open a doc-issue
    • AI: [bmbouter] write up plugin docs for this
  • ppicka is leaving us :’(
    • AI: [bmbouter] to remove from mtg invite
    • AI: [team] move ppicka to Contributor
  • settings controlled by semver
    • AI: [x9c4] open thread in discourse
    • AI: [all] revisit and confirm at next mtg
  • AI: [team] get 3427 reviewed so we can merge no later than Fri
1 Like
#63

FYI I gave write perms to Lubos as well, he is part of pytest group.

1 Like
#64

Opened the ticket for CodeQL issues

1 Like
#65

December 13, 2022

  • [decko] reviewed/approved 3427 (HStore)
  • Zero-downtime next steps
  • 3190 (domains)
    • discussion ensues
    • AI: [gerrod] run all/many plugins’ CI tests against the PR
    • who are the SMEs?
      • gerrod
      • RTFM group?
      • gerrod’s presentation makes it possible for everyone to understand
    • until we have confident LG2M from confident SMEs, we should maybe release core/3.22 sans domains, then 3.23 w/ this
  • core/3.22 discussion around probs caused by merged content-app pr
    • affects python and container
    • do we revert? wait for a fix? hold off 3.22?
    • need to resolve prior to a core/3.22 release
    • needs to be fixed/addressed in pulpcore - issue?
    • “fixing it” in container/python
      • fails at core/3.22 release if older plugins are installed
    • what happens if we “fix” this in plugins and backport?
      • container tinks this is supportable
      • how many active z-streams? (2 for container)
    • CI is currently broken because of this problem
      • about 1/3 tests in container
      • “some” tests broken in python
    • reverting the two commits fixes CI - but removes the major drivers for core/3.22 being date-driven
      • revert-commit-msg needs to explain WHY we’re reverting
    • need to have something in place for core/3.22 and pulp_rpm
    • we need to expand on pulpcore’s interface in this area, in a way that plugins can reliably take advantage of it - for core/3.25
    • AI: this needs A LOT of discussion at OpenFloor
#66

December 20, 2022

  • “stream” test is really unreliable
    • we need to fix this problem, or disable the unreliable tests - they teach us to “ignore red”
  • Upstream high-value tickets, should we prio-list them?
  • ggainey out 27-DEC - cancel the mtg next week? or appoint a facilitator?
    • AI: ggainey to cancel next week
  • gerrod becomes facilitator for Jan-Feb
    • prob should not be “optional” for those months :slight_smile:
    • AI: ggainey to voluntold gerrod
#67

Jan 3, 2023

  • “stream” test is really unreliable
    • we need to fix this problem, or disable the unreliable tests - they teach us to “ignore red”
    • [team] decided to keep the stream job on and leave this work for Michal, as planned. The bug is sporadic so should not have high impact
  • Upstream high-value tickets, should we prio-list them?
  • [ttereshc] added Mike and Humberto as optional attendees when and if they have some questions or need clarifications on pulp architecture, or pulpcore related topics, which are easier to bring up over a call.
#68

January 10

  • review Cheat sheet for the pulpcore release shepherd - HackMD
    • [AI] Dennis to bring this up in the deployers meeting
  • Galaxy[Bruno] - Add a ContentLabel model
    • Most likely implement in pulp_ansible for now, hard to be generic enough for all plugins
    • Could eventually replace AnsibleCollectionDeprecated
    • Should probably be similar in implementation to pulp_container Tags
    • ContentMark is favorite name
  • issues with pulp-in-one-container image https://github.com/pulp/pulpcore/issues/3492
    • Dennis to continue following up with poster on the issue
#69

January 17

#70

January 24

  • [dkliban] to update pulpcore release docs to include OCI images release information
  • Pulp replica discussion
    • fun with repo-attached Distributions and auto-publish
#71

January 31

#72

February 14

  • Faster release schedule (and no more waiting on blockers) is needed if we want our cloud deployment downstreams to not deploy from the main branch.

    • Are migrations guaranteed to be stable between adding to main and releasing?
    • We could defer merging PRs with migrations until later in the cycle?
      • Might not be a later phase if we are releasing faster. Need to review faster
    • Goal is to release often so we don’t run into a scenario where we want to release a plugin that is waiting on a pulpcore release
      • Cloud services might end up using pulpcore from main anyway
        • But they may also consume pulp as a library from only ga versions.
    • Release every week?
      • Y release if features present, else will be a z-stream release
      • What about bugfixes? How does this impact backporting?
      • Could be a backport nightmare, need to have unofficial LTS versions that we get products to share
      • Need to choose a new larger future compatible pulpcore version for plugins to declare compatibility with, maybe 10 versions: 3.35?
        • Potentially change pulpcore versioning to date base format?
      • [AI] dkliban to write a discourse post discussing how many new versions we want to declare future compatibility against

  • [lmjachky] Review of https://github.com/pulp/pulpcore/issues/3429

    • Again, my take on this is that we can mark almost all of the features in tech preview as production ready NOW.
    • Features in tech preview
      • Based on the comments, we have direct feedback from users who request features. Why do we need analytics first in order to mark the features as production ready?

  • Let’s use merge queues for Pulpcore

  • Stream runner (analysis)

  • Replication PR

#73

February 21, 2023

#74

February 28, 2023

  • Upgrading Django, 4.2 (LTS) is coming in April
    • 3.2 upstream support is till Dec 1, 2023.
    • Any known changes which affect us?
    • AAP is asking for 4.1 even now, pushing back to wait for 4.2
    • With django 4.* and django-channels, we may be able to consolidate the codebase (sync and async parts) on django.
    • Do migrations break with this change?
    • SUMMARY: We want to switch to 4.2 as soon as possible. Pulpcore 3.25 is the best opportunity. If we find FTE, someone should experiment with 4.1 asap.
    • [AI DKliban] Look for an issue; add it to 3.25 blockers.
    • [AI DKliban] advertise the Django change on Discourse.
  • crazy idea time: Can we turn Upload into a content type?
    • Chunks are really “just” artifacts.
    • We’d get orphan cleanup for free.
    • One less type of object to store in the storage bucket.
    • [AI MDellweg] Write a story
  • Open Telemetry update
    • Do we want this to be feature flagged?
      • YES
    • How do you want it turned on/off?
      • Question of deployment
    • Should it be on-by-default?
    • Can we make it an optional dependency?
    • [AI BBouters] Present <= 10’ about telemetry
  • What is the process to add metrics to analytics?
1 Like
#75

March 7, 2023

  • Can we get a Pulp/Katello Integration Mtg representative to cover dkliban’s leave?
    • ggainey is It
    • [AI] ggainey to suggest every-other-week for this mtg
  • core/3.23 - release when?
    • domains PR almost merged - today
    • pulp-cli/glue will have a release
    • then pulpcore
    • then pulp_file
  • calVer recap
    • current thinking: let’s just release faster and not switch to calver right now
    • not possible anyway before 3.25
  • Next breaking change Release after 3.25?
  • Analytics proposal
1 Like
#76

March 14, 2023

  • release core/3.23 today?

    • yes please!
    • AI: ggainey to turn the release crank today

  • moved some AIs from dkliban/bmbouter to decko (lucky man)

  • opentelemetry scope-of-work discussion

    • upstream user-interest/conversations
      • talk to decko if you want to be involved
    • developers
    • Scope of Work
      • Develop and merge OTEL instrumentation in pulpcore
        • code-changes only in pulpcore-workers
        • autoinstrumenting handling Other Parts (wsgi, aiohttp, postgres)
      • Write user-facing pulpcore documentation on how to configure metrics and tracing
      • Ensure oci-env has a profile that fully sets up prometheus, jaeger, grafana, and collector
      • Record 5-minute demo showing metrics collection with Grafana and Prometheus (pulp-administrator-facing)
      • Record 5-minute demo showing tracing with Jaeger (developer-facing)
      • demos need to be sent to youtube-admin and then publicized

  • open telemetry performance update

  • open telemetry dependency for pulp

    • would include the various autoinstrumentation pkgs (postgres, redis, wsgi, etc) and utilities
    • prob ~10ish new pkgs
    • decision needs to be made on whether pulp requires these “always” or “optionally”
      • needs discussion and decision
    • even if deps are present, maybe env-var to turn on/off?
    • is operator interested?
      • yes - primarily worker-metrics, for scaling decisions
      • kubernetes envs will handle scaling-due-to-hardware-issues (ie, CPU-utilization)

  • Can this be reviewed/merged before 3.23 release?

  • What about this one:

  • observation/evaluation of https://analytics.pulpproject.org/ status

    • discussion around understanding why so many “old” core-versions
    • should/can we think about which “downstream” is reporting analytics?
    • lots of discussion around how to interpret results
2 Likes