Katello/Pulp-Community Integration meeting minutes

Minutes
#21

2022-12-13 1000-1030 GMT-5

Attendees: ggainey, sjha

Regrets:

Agenda:

  • meeting called
  • will schedule first 2023 meeting for 10-JAN!
    • meeting value/format to be evaluated at this mtg

Action Items:

#22

2023-01-10 1000-1030 GMT-5

Attendees: ggainey, iballou, sjha, quba42, hstct

Regrets:

Agenda:

Action Items:

  • AI: [ggainey] schedule 6 months starting first tues in March
  • AI: [ggainey] clean up mtg-doc
1 Like
#23

2023-03-07 1000-1030 GMT-5

Attendees: ggainey, iballou, sjha, quba42, hstct

Regrets:

Agenda:

Action Items:

#24

2023-04-04 1000-1030 GMT-5

Attendees: quba42, hstct, sjha, ggainey

Regrets:

Agenda:

  • Previous AIs:

  • Possibility of releasing pulp_deb orphan cleanup fix for released versions of Katello?

    • See: https://community.theforeman.org/t/katello-4-8-0-rc1-feedback/33039/4
    • fix includes a migration
    • it’s the only “new” migration since release
    • should be “safe” in all respects (idempotent, no other possibly-skipped migrations “in the way”)
    • which version backporting to?
      • pulp_deb/2.20.1, for katello/4.7 users

  • Any plans to drop Puppet 6 support? If so, with what Foreman/Katello version?

  • Adding a mix of synced and uploaded content in a single repository

    • ATIX opinion:
      • It is a best practice to use separate repos for the two use cases. (Especially in the case of deb/APT content)
      • Katello should be opinionated about enforcing best practice here.
      • Pulp probably does not need to be as opinionated about this as Katello should be.
        • Pulp opinion: Pulp should not/cannot be opinionated in this area
      • See also: RFC: Separating Sync and Upload workflows in pulp_deb (and others?) - #9 by Stephen_Herr
      • can we get a discussion opened on theforeman discourse to gather user-feedback?
      • At a minimum, can we get the katello UI to give some guidance to keep users from accidentally shoot themselves in the foot

Action Items:

1 Like
#25

2023-05-02 1000-1030 GMT-5

Attendees: hstct, sjha, quba42, ggainey, iballou

Regrets:

Agenda:

  • Previous AIs:

  • core/3.24 releasing TODAY

  • core/3.25 #soon - mid/end of May?

    • 3.25.0 blockers Milestone ¡ GitHub
    • django-4.2
    • psycopg3
    • any deprecation warnings? fix em now!
    • next breaking-release: core/3.40
    • discussion
      • packaging?
      • where do plugins look to find dperecation-warnings?
        • quba42 to ask in OpenFloor
    • AI: [ggainey] is there a list of core-releases we backport stuff to?
      • write this down somewhere
      • eg, “what releases does katello depend on”

  • pulp_deb release: Pulp_deb 2.20.2 has been released!

    • bindings change implied?
      • makes katello-packaging harder
      • serializer-change - almost certainly yes
    • can this be ready for katello/4.9 release?
      • should be possible
      • quba42 to investigate/do/poke katello if more info needed

  • SLES advisory export/import problem

  • Working with the Katello upload implementation:

    • https://github.com/Katello/katello/blob/master/app/lib/actions/pulp3/orchestration/repository/upload_content.rb
    • Katello is open to using the repository parameter on content endpoints instead of separately calling the modify API after creating the content.
    • If the upload workflow needs to change for plugins that implement “create or retireve” logic on the Pulp side, there are two options:
      • A flag in the existing workflow (if the two workflows are not very different)
      • A separate workflow that is called for some plugins
      • The goal should be to ultimately move all plugins to the new workflow and move “create or retrieve” logic from Katello to Pulp.

Action Items:

#26

2023-06-06 1000-1030 GMT-5

Attendees: hstct, sjha, quba42, iballou

Regrets: ggainey

Agenda:

Action Items:

#27

2023-07-11 1000-1030 GMT-5

Attendees: quba42, hstct, bernhards, ggainey, mbucher

Regrets: all of katello

Agenda:

Action Items:

2 Likes
#28

2023-08-01 1000-1030 GMT-5

Attendees: ggainey, quba42, hstct, iballou, sjha

Regrets:

Agenda:

  • Previous AIs:
  • We had a strange edge case where a Pulp publication failed because of full disk. (The Pulp sync was successfull) This means the repo was synced and all the packages showed up in Katello, but the content wasn’t actually being served in the CV. Re-syncing did nothing, because the Pulp re-sync reported no new packages, so Katello did not bother to re-publish. Ultimately we solved it by doing a “republish repository metadata” using hammer.
    • Are there plans for “republish repository metadata” to come back to the UI, or will it remain hammer only?
    • The above showed that if sync succeeds, but publish fails, regular Katello usage will never repair this situation, can this be improved?
    • there are lots of ways metadata might get corrupted
      • katello is restoring “republish” to the UI for non-mirror-complete repositories
      • there is a push to make it globally available in spite of ways that might Go Worng - discussion in progress
  • review of structured-apt writeup
    • will be in pulp_deb 3.0
    • 2 katello PRs, 1 subscription-manager PR
    • how to make forklist use unreleased versions of “things”?
      • iballou: usually just do a pip-install of source
    • orcharino test-setup - see quba42 for details
    • still edge-cases that need to be solved
      • prob need to get together w/ someone on candlepin to iron out some details
    • who should quba42 work with/demo to, to work thru the pulp_deb/katello/candlepin pieces?
      • paji/jlentz/croberts?
    • AI: [quba42] get forklift-env set up w/ POC uip and running
    • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
    • AI: [iballou] nag more folk to review the structured-content-doc
    • is this something katello/4.10 has-to have?
      • is there a pulp_deb compatible with core/3.28?
        • yes - 2.22 (we think)
        • is that the right version for katello/4.10? If not, tell iballou “now”
      • there exists a long data-migration to get to pulp_deb/3
    • notes from chat

Action Items:

  • RE structured-deb-content
    • AI: [quba42] get forklift-env set up w/ POC uip and running
    • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
    • AI: [iballou] nag more folk to review the structured-content-doc
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like
#29

2023-09-05 1000-1030 GMT-4

Attendees: quba42, sjha, iballou, ggainey

Regrets:

Agenda:

  • Previous AIs:
    • RE structured-deb-content
      • AI: [quba42] get forklift-env set up w/ POC uip and running
        • still in-progress
        • pulp_deb/3.0 gated us - but it has been released!!
      • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
      • AI: [iballou] nag more folk to review the structured-content-doc
        • lots of responses already in thread
  • pulp_deb 3.0 is released.
    • What Katello version should this be added to? (long running DB migration!)
    • requires matching client-gem packaging, implies a potentially long-running migration
    • prob too late for katello/4.10
    • let’s shoot for katello/4.11?
    • how does pulp_deb make sure this ends up with the right katello/foreman packaging dir?
      • foreman-installer will handle updating if/when pulp_deb/3.0 “magically shows up” in the katello/4.10-pulpcore/3.28 repo-structure
    • how to handle N-1 smartproxy setup?
      • can a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
        • this needs to be flattened first
        • quba42 to work on investigation here
  • pulp_deb bugfix releases, e.g.: https://github.com/theforeman/pulpcore-packaging/pull/713
    • should “just work”
    • add required PRs to approproiaite release-branches
  • What exactly is the state of pulp_rpm dependency solving?
  • would be Really Nifty to have pulp_deb participation in foreman-community-demos
    • every 3 weeks on Thurs?
    • this is a fine, fine idea

Action Items:

  • RE structured-deb-content
    • AI: [quba42] get forklift-env set up w/ POC uip and running
      • still in-progress
      • pulp_deb/3.0 gated us - but it has been released!!
    • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
  • quba42: investigate whether a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like
#30

[previous month’s minutes]

2023-10-03 1000-1030 GMT-4

Attendees: sjha, quba42

Regrets: iballou, ggainey, hstct

Agenda:

  • Previous AIs:
    • RE structured-deb-content
      • AI: [quba42] get forklift-env set up w/ POC uip and running
        • still in-progress
        • pulp_deb/3.0 gated us - but it has been released!!
      • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
    • quba42: investigate whether a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
  • Remaining pulp_deb 3.0 packaging PRs:
  • Import/Export nil vs empty string issues. Has anyone else encountered this as a problem?
  • sjha: Question regarding smart proxy syncs for deb content:
    • Working on a new feature to display content counts for smart proxy syncs.
    • Noticed that Debian appeared to have two contents on the smart proxy for every one content on the server CV/Lcenv.
    • It appears every .deb package is duplicated on the smart proxy with two alternate relative paths.
    • Is this a bug?
      • quba42: This is an unintended design consequence of having both structured and simple publication mode on the server.
      • This will no longer be the case if we eventually transition to using only structued APT.
      • For now, we will just have to live with the fact that the content counts for .deb content will be double for the proxy.
      • Side note: For synced content the content count on the proxy should be exactly twice what it is on the server, however this may not be true for uploaded packages.

Action Items:

#31

2023-11-14

Attendees: quba42, iballou, mbucher, hstct, vsedmik, ggainey

Regrets:

Agenda:

  • Previous AIs:
    • quba42: Not yet any progress on structured APT (I got consumed in firefighting…)
  • Heal failed publications PR: https://github.com/Katello/katello/pull/10776
    • can we get some katello eyes to understand the failing test (and understand/approve)
  • Regarding: Slow content proxy sync - #38 by dralley - Support - TheForeman
  • Does Katello provide any features to trigger pulpcore’s artifact repair? Should it? (I am thinking a rake task or similar, or perhaps even a UI button?)
  • pulp_rpm publishes or syncs wrong modules.yaml:
  • UI performance:
    • We are getting customer feedback that it “feels like” things are growing slower and slower with each new version.
    • Note that these reports are mostly about Foreman 3.3, Katello 4.5
    • Question: Have you heard similar complaints? Did anything change (like system requirements, optimization confg)?
    • we have had specific performance issues, mostly on Really Large Instances
      • (list of closed perf-related issues here)
    • All hosts page: According to Bernhard Suttner a lot of data (including a lot of Katello related stuff) is loaded that is not even necessarily displayed, can this be optimized. (Examples: content_facets / concerns / host_managed_extensions)
    • from katello:
      • applicability has had some perf updates recently
      • host-queries can be really slow
      • RE katello/4.5: can’t think of any current complaints?
      • can we get specifics? please?
    • welcome to vsedmik (katello QE)

Action Items:

1 Like
#32

2023-12-05 1000-1030 GMT-5

Attendees: ggainey, mbucher, hstct, sjham iballou, dalley

Regrets: quba42

Agenda:

  • Previous AIs:
  • Meeting Review/Resched for 2024!
    • Still a useful mtg?
    • Still a useful cadence?
    • Still a useful time?
    • Yes to all
    • AI: ggainey to sched for 9-JAN, and then 5 1st-Tues into 2024
  • Question for Katello QE (if present): How do you test for N-1 proxy compatibility?
    • a lot of hand-testing
    • once basic smoke tests pass, move to N-1
    • sync upstream all content-type repos, then SmartProxy sync the results
    • QE uses robotello tests, will be getting enhanced
    • there’s a response “somewhere” in the foreman-community on this
  • Katello PR feedback:
  • hstct: publish-time for deb-repos got really long
    • due to on-demand/RemoteArtifact lack of indexing
    • https://github.com/pulp/pulpcore/pull/4836 adds an index
    • publish-time grew from 8 hours to 20+ at large installation
    • AI: hstct to report back on customer experience once applied
    • may cause specific issues in pulp-deb due to its code-path
    • specifically, affects installations w/ large amounts of on_demand content and publishing
  • Support for zstd metadata ¡ Issue #3254 ¡ pulp/pulp_rpm ¡ GitHub
    • need createrepo update as well?
    • yes - need 1.something
    • pulp_rpm patch updates createrepo_c dep as well
    • backported to rpm/3.18, .19, and .22

Action Items:

  • ggainey: schedule mtgs for 2024
  • iballou: summarize discussion RE global-repair into PR
  • iballou: Poke People to get other katello PRs reviewed
  • hstct to report back on customer experience once RemoteArtifact indices applied
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like
#33

2024-01-09 1000-1030 GMT-5

Attendees: bsuttner, quba42, iballou, ggainey, dalley, pbrochado, sweta

Regrets:

Agenda:

Action Items:

#35

2024-02-13 1000-1030 GMT-5

Attendees: ggainey, mbucher, sja, iballou, quba42, hstct

Regrets:

Agenda:

  • Previous AIs:
  • meeting in person at conference was Really Useful!
  • fun with Forklift - lots of fixing has happened!
    • discussion between ATIX/katello
  • discussion around katello/UI bug in 4.11
    • needs an upstream issue opened?
  • discussing next-pulp-version for katello/4.13
    • prob whatever the latest-Y is at branch-point
    • would be good if a pulp_deb-Y right before katello-branch so katello gets latest
  • pulpcore discussed “supported branches” this week
    • discussion ensues
  • how can we get katello reviews happening more quickly on pulp-deb/ATIX Stuff?
    • can we get ATIX approve-access to the affected repos?
    • sure - there is A Process, let’s do it!
    • iballou to bring up w/ katello team
    • ATIX needs a candidate to start doing regular katello-reviews?
      • work w/ iballou/sja to iron out details
  • ATIX customer is “losing publications”?!?
    • heavy CVV user
    • old CVV “loses” pub?
    • ATIX investigating - rake cleanup task?
    • how can a repo-version exist, but associated publications are gone?!?
    • check if pulp was told to delete?
    • katello/4.7 (we think) is what’s in use
    • might be a related fix, in a newer katello?
  • sja: there exists a PR to “freeze” repo-access while a publication is happening
#36

2024-03-05 1000-1030 GMT-5

Attendees: ggainey, iballou, sjha, sweta, quba42, hstct

Regrets:

Agenda: