Katello/Pulp-Community Integration meeting minutes

Minutes
#13

2022-04-05 1000-1030 GMT-5

Attendees: iballou, quba42, mbucher, sajha, paji, ggainey

Regrets:

Agenda:

  • State of aiohttp(s) progress? E.g.: https://github.com/theforeman/pulpcore-packaging/pull/379
    • “can’t use https proxies w/ Pulp w/out aiohttp/3.8+, puthon3.8+”
    • [AI] iballou to inquire and let quba42 know, doc here as well
  • katello-host-tools service restarts:
  • When is the next breaking changes pulcore release? How is Katello approaching the changed pulpcore deprecation policy?
    • core/3.20 is allowed to have breaking changes. 3.19 to be released “really soon now” (today?)
    • katello: doesn’t want to get stuck on old versions, will pay attention to the proboem if/when/as it happens
  • Smart-proxy download policy streamed: https://github.com/Katello/katello/pull/10005
    • Merged!
  • 2to3 migration issue
    • large migration, Pulp3-orphan-cleanup ran in the middle of it (?!?!?)
    • is there any way this could happen in a “vanilla katello/3.18” install?
    • poss a cfg-issue (katello-installer shouldn’t allow this to happen)
  • ATIX has targetted customers in mind to test 2to3
  • Pulp moving to “releases on backport-branches” to once a week
    • ATIX mostly interested in RPM results

Action Items:

#14

2022-05-03 1000-1030 GMT-4

Attendees: ggainey, quba42, mbucher, sjha, iballou, paji

Regrets:

Agenda:

  • Upgrading Katello nightly to pulp_deb 2.18.0 (including clients):
  • DB field size migration and backporting woes:
  • Debian Errata strategy regarding pulp3 & Katello 4.x
    • current draft Katello-only implementation reusing Katello::Erratum model
    • question1: should we add them to pulp_deb to more resemble yum-errata
    • question2: should we distinguish them more in Katello creating DebErratum and YumErratum models
    • feels possible to keep this in katello, make the inheritance/implementation hierarchy make more sense
    • maybe ATIX discuss internally, bring something more concrete to this meeting #eventually?
  • Discussing pulpcore-packaging minor version bump workflows.
    • and by packaged, we mean “get RPMs made”
    • how can ATIX make this request less ad-hoc?
    • current katello approach is similar to current ATIX - open pulp-packaging PR and then ping folk if ignored
    • new pulpcore-Y-release takes more work (since requires new directory)
1 Like
#15

2022-07-06 1000-1030 GMT-4

Attendees: ggainey, quba42, mbucher, hstct, iballou, sjha, paji

Regrets:

Agenda:

  • welcome to Tobias (hstct) from ATIX
  • orcharhino 6.0 is out https://orcharhino.com/orcharhino-6-0-0/
    • Uses Katello 4.1 with pulpcore 3.14
    • Reporting issues back to the community? (We have talked about this before)
      • Yes please.
      • discussion about when/whether to backport will be ongoing.
  • How will the breaking pulpcore 3.20 work?
    • release aimed for 14-JUN
  • Outlook: Improving pulp_deb sync performance
    • What has your experience been: Is performance a pure Pulp topic, or is it as much to do with how Katello uses Pulp?
      • almost all in the plugin itself
  • Tobias is rewriting tests to use pytest
    • ggainey cheers wildly
    • talk to dkliban/bmbouters
  • debian pkg-upload broken in katello master
    • ATIX nag incoming
    • one fix made, more probs discovered
  • katello removed reviewing deb-filters-PR from sprint
  • [paji] needs-restart PR stalled

Action Items:

1 Like
#16

2022-07-05 06-28 1000-1030 GMT-4 (rescheduled due to conflict)

Attendees: ggainey, iballou, quba42, hstct, msinghal

Regrets:

Agenda:

Action Items:

#17

2022-08-09 1000-1030 GMT-5 (rescheduled due to conflict)

Attendees: quba42, sjha, iballou, hstct, ggainey

Regrets:

Agenda:

Action Items:

  • RE backport/release discussion - iballou to make the public parts, public, in Discourse thread
  • RE HTTPS Proxy work - ggainey to look for any other discussion around this and include here [DONE]
  • RE pulp_deb and urlencoding content-URLS - quba42 to release pulp_deb w/ the fix
  • RE pulp_deb CV filtering - quba42 to talk to msinghal about presenting at katello-community-demo
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like
#18

2022-09-06 1000-1030 GMT-5

Attendees: msinghal, quba42, iballou, sjha, ggainey, hstct, mbucher

Regrets:

Agenda:

  • Previous AIs
    • RE pulp_deb and urlencoding content-URLS - quba42 to release pulp_deb w/ the fix
  • How does Katello install Pulp? I believe the pulp_installer is just for pip based installs. Could I just install the RPMs and start the services? Is a from RPM Pulp installation (without installing all of Katello) somehow supported? Is it documented somewhere?
  • pulp_deb optimize sync will depend on pulpcore >= 3.21.0. Given the latest pulpcore-packaging is 3.18, when can we reasonably expect this to become available for Katello?
  • HTTPS proxy (again), will this be a “downstream only fix” (for older python versions): Pulpcore 3.21 release schedule - #4 by bmbouter
    • in a word - “yes” :slight_smile:
  • I would like to set FORCE_IGNORE_MISSING_PACKAGE_INDICES = True in /etc/pulp/settings.py for Katello installations. I imagine this would be some foreman-installer change, but how can I find the right place?
  • Import/Export feature doesn’t work in Katello but works fine with pulp_deb.
    • work directly w/ Partha Aji (paji@redhat.com, partha in matrix in #pulp-dev)
  • “regenerate repository metadata” was removed from UI (for all content types) because of problems with pulp_rpm “complete mirror” mode? Katello 4.3 Cannot Republish Repository Metadata or View Packages of a Repository - #4 by jeremylenz - Support - TheForeman
    • pulp_deb would like to see it made available for deb-repos
    • “Why does RPM ‘Win’?”
      • Partially a one-off emergency fix
      • katello would like to have it back as well
      • what katello PR removed this?
      • great discussion around what “republish” means for other content types
      • maybe remove UI IFF “complete mirroring” in use
  • Forklift issues.
    • what’s up w/ katello-devel?
    • centos7/8-katello-stable/unstable aren’t working (for a variety of reasons)
    • see foreman-dev matrix
      • hstct: anually reinstall webpack?
      • sjha: “bundle exec rails webpack:compile in foreman dir…”
      • sjha: immediately upgrade nodejs to 12 on dev-box
      • sjha: delete package-lock/node files in foreman/katello and reinstall?
      • iballou: stable boxes should “just work”?
#19

2022-10-04 1000-1030 GMT-5

Attendees: sjha, iballou, ggainey

Regrets:

Agenda:

#20

2022-11-02 0900-0930 GMT-5

Attendees: ggainey, iballou, quba42, mbucher, sjha, hstct

Regrets:

Agenda:

Action Items:

  • AI: [iballou] to look at SCA-thread, add debugging commentary if/as useful
  • AI: [ATIX] to start a thread on import/export with “katello” tag, maybe ping iballou/sjha/paji in #foreman-dev
  • AI: [ATIX] someone at ATIX will poke at issue #35362
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
#21

2022-12-13 1000-1030 GMT-5

Attendees: ggainey, sjha

Regrets:

Agenda:

  • meeting called
  • will schedule first 2023 meeting for 10-JAN!
    • meeting value/format to be evaluated at this mtg

Action Items:

#22

2023-01-10 1000-1030 GMT-5

Attendees: ggainey, iballou, sjha, quba42, hstct

Regrets:

Agenda:

Action Items:

  • AI: [ggainey] schedule 6 months starting first tues in March
  • AI: [ggainey] clean up mtg-doc
1 Like
#23

2023-03-07 1000-1030 GMT-5

Attendees: ggainey, iballou, sjha, quba42, hstct

Regrets:

Agenda:

Action Items:

#24

2023-04-04 1000-1030 GMT-5

Attendees: quba42, hstct, sjha, ggainey

Regrets:

Agenda:

  • Previous AIs:

  • Possibility of releasing pulp_deb orphan cleanup fix for released versions of Katello?

  • Any plans to drop Puppet 6 support? If so, with what Foreman/Katello version?

  • Adding a mix of synced and uploaded content in a single repository

    • ATIX opinion:
      • It is a best practice to use separate repos for the two use cases. (Especially in the case of deb/APT content)
      • Katello should be opinionated about enforcing best practice here.
      • Pulp probably does not need to be as opinionated about this as Katello should be.
        • Pulp opinion: Pulp should not/cannot be opinionated in this area
      • See also: RFC: Separating Sync and Upload workflows in pulp_deb (and others?) - #9 by Stephen_Herr
      • can we get a discussion opened on theforeman discourse to gather user-feedback?
      • At a minimum, can we get the katello UI to give some guidance to keep users from accidentally shoot themselves in the foot

Action Items:

1 Like
#25

2023-05-02 1000-1030 GMT-5

Attendees: hstct, sjha, quba42, ggainey, iballou

Regrets:

Agenda:

  • Previous AIs:

  • core/3.24 releasing TODAY

  • core/3.25 #soon - mid/end of May?

    • 3.25.0 blockers Milestone · GitHub
    • django-4.2
    • psycopg3
    • any deprecation warnings? fix em now!
    • next breaking-release: core/3.40
    • discussion
      • packaging?
      • where do plugins look to find dperecation-warnings?
        • quba42 to ask in OpenFloor
    • AI: [ggainey] is there a list of core-releases we backport stuff to?
      • write this down somewhere
      • eg, “what releases does katello depend on”

  • pulp_deb release: Pulp_deb 2.20.2 has been released!

    • bindings change implied?
      • makes katello-packaging harder
      • serializer-change - almost certainly yes
    • can this be ready for katello/4.9 release?
      • should be possible
      • quba42 to investigate/do/poke katello if more info needed

  • SLES advisory export/import problem

  • Working with the Katello upload implementation:

    • https://github.com/Katello/katello/blob/master/app/lib/actions/pulp3/orchestration/repository/upload_content.rb
    • Katello is open to using the repository parameter on content endpoints instead of separately calling the modify API after creating the content.
    • If the upload workflow needs to change for plugins that implement “create or retireve” logic on the Pulp side, there are two options:
      • A flag in the existing workflow (if the two workflows are not very different)
      • A separate workflow that is called for some plugins
      • The goal should be to ultimately move all plugins to the new workflow and move “create or retrieve” logic from Katello to Pulp.

Action Items:

#26

2023-06-06 1000-1030 GMT-5

Attendees: hstct, sjha, quba42, iballou

Regrets: ggainey

Agenda:

Action Items:

#27

2023-07-11 1000-1030 GMT-5

Attendees: quba42, hstct, bernhards, ggainey, mbucher

Regrets: all of katello

Agenda:

Action Items:

2 Likes
#28

2023-08-01 1000-1030 GMT-5

Attendees: ggainey, quba42, hstct, iballou, sjha

Regrets:

Agenda:

  • Previous AIs:
  • We had a strange edge case where a Pulp publication failed because of full disk. (The Pulp sync was successfull) This means the repo was synced and all the packages showed up in Katello, but the content wasn’t actually being served in the CV. Re-syncing did nothing, because the Pulp re-sync reported no new packages, so Katello did not bother to re-publish. Ultimately we solved it by doing a “republish repository metadata” using hammer.
    • Are there plans for “republish repository metadata” to come back to the UI, or will it remain hammer only?
    • The above showed that if sync succeeds, but publish fails, regular Katello usage will never repair this situation, can this be improved?
    • there are lots of ways metadata might get corrupted
      • katello is restoring “republish” to the UI for non-mirror-complete repositories
      • there is a push to make it globally available in spite of ways that might Go Worng - discussion in progress
  • review of structured-apt writeup
    • will be in pulp_deb 3.0
    • 2 katello PRs, 1 subscription-manager PR
    • how to make forklist use unreleased versions of “things”?
      • iballou: usually just do a pip-install of source
    • orcharino test-setup - see quba42 for details
    • still edge-cases that need to be solved
      • prob need to get together w/ someone on candlepin to iron out some details
    • who should quba42 work with/demo to, to work thru the pulp_deb/katello/candlepin pieces?
      • paji/jlentz/croberts?
    • AI: [quba42] get forklift-env set up w/ POC uip and running
    • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
    • AI: [iballou] nag more folk to review the structured-content-doc
    • is this something katello/4.10 has-to have?
      • is there a pulp_deb compatible with core/3.28?
        • yes - 2.22 (we think)
        • is that the right version for katello/4.10? If not, tell iballou “now”
      • there exists a long data-migration to get to pulp_deb/3
    • notes from chat

Action Items:

  • RE structured-deb-content
    • AI: [quba42] get forklift-env set up w/ POC uip and running
    • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
    • AI: [iballou] nag more folk to review the structured-content-doc
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like
#29

2023-09-05 1000-1030 GMT-4

Attendees: quba42, sjha, iballou, ggainey

Regrets:

Agenda:

  • Previous AIs:
    • RE structured-deb-content
      • AI: [quba42] get forklift-env set up w/ POC uip and running
        • still in-progress
        • pulp_deb/3.0 gated us - but it has been released!!
      • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
      • AI: [iballou] nag more folk to review the structured-content-doc
        • lots of responses already in thread
  • pulp_deb 3.0 is released.
    • What Katello version should this be added to? (long running DB migration!)
    • requires matching client-gem packaging, implies a potentially long-running migration
    • prob too late for katello/4.10
    • let’s shoot for katello/4.11?
    • how does pulp_deb make sure this ends up with the right katello/foreman packaging dir?
      • foreman-installer will handle updating if/when pulp_deb/3.0 “magically shows up” in the katello/4.10-pulpcore/3.28 repo-structure
    • how to handle N-1 smartproxy setup?
      • can a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
        • this needs to be flattened first
        • quba42 to work on investigation here
  • pulp_deb bugfix releases, e.g.: https://github.com/theforeman/pulpcore-packaging/pull/713
    • should “just work”
    • add required PRs to approproiaite release-branches
  • What exactly is the state of pulp_rpm dependency solving?
  • would be Really Nifty to have pulp_deb participation in foreman-community-demos
    • every 3 weeks on Thurs?
    • this is a fine, fine idea

Action Items:

  • RE structured-deb-content
    • AI: [quba42] get forklift-env set up w/ POC uip and running
      • still in-progress
      • pulp_deb/3.0 gated us - but it has been released!!
    • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
  • quba42: investigate whether a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like
#30

[previous month’s minutes]

2023-10-03 1000-1030 GMT-4

Attendees: sjha, quba42

Regrets: iballou, ggainey, hstct

Agenda:

  • Previous AIs:
    • RE structured-deb-content
      • AI: [quba42] get forklift-env set up w/ POC uip and running
        • still in-progress
        • pulp_deb/3.0 gated us - but it has been released!!
      • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
    • quba42: investigate whether a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
  • Remaining pulp_deb 3.0 packaging PRs:
  • Import/Export nil vs empty string issues. Has anyone else encountered this as a problem?
  • sjha: Question regarding smart proxy syncs for deb content:
    • Working on a new feature to display content counts for smart proxy syncs.
    • Noticed that Debian appeared to have two contents on the smart proxy for every one content on the server CV/Lcenv.
    • It appears every .deb package is duplicated on the smart proxy with two alternate relative paths.
    • Is this a bug?
      • quba42: This is an unintended design consequence of having both structured and simple publication mode on the server.
      • This will no longer be the case if we eventually transition to using only structued APT.
      • For now, we will just have to live with the fact that the content counts for .deb content will be double for the proxy.
      • Side note: For synced content the content count on the proxy should be exactly twice what it is on the server, however this may not be true for uploaded packages.

Action Items:

#31

2023-11-14

Attendees: quba42, iballou, mbucher, hstct, vsedmik, ggainey

Regrets:

Agenda:

  • Previous AIs:
    • quba42: Not yet any progress on structured APT (I got consumed in firefighting…)
  • Heal failed publications PR: https://github.com/Katello/katello/pull/10776
    • can we get some katello eyes to understand the failing test (and understand/approve)
  • Regarding: Slow content proxy sync - #38 by dralley - Support - TheForeman
  • Does Katello provide any features to trigger pulpcore’s artifact repair? Should it? (I am thinking a rake task or similar, or perhaps even a UI button?)
  • pulp_rpm publishes or syncs wrong modules.yaml:
  • UI performance:
    • We are getting customer feedback that it “feels like” things are growing slower and slower with each new version.
    • Note that these reports are mostly about Foreman 3.3, Katello 4.5
    • Question: Have you heard similar complaints? Did anything change (like system requirements, optimization confg)?
    • we have had specific performance issues, mostly on Really Large Instances
      • (list of closed perf-related issues here)
    • All hosts page: According to Bernhard Suttner a lot of data (including a lot of Katello related stuff) is loaded that is not even necessarily displayed, can this be optimized. (Examples: content_facets / concerns / host_managed_extensions)
    • from katello:
      • applicability has had some perf updates recently
      • host-queries can be really slow
      • RE katello/4.5: can’t think of any current complaints?
      • can we get specifics? please?
    • welcome to vsedmik (katello QE)

Action Items:

1 Like
#32

2023-12-05 1000-1030 GMT-5

Attendees: ggainey, mbucher, hstct, sjham iballou, dalley

Regrets: quba42

Agenda:

  • Previous AIs:
  • Meeting Review/Resched for 2024!
    • Still a useful mtg?
    • Still a useful cadence?
    • Still a useful time?
    • Yes to all
    • AI: ggainey to sched for 9-JAN, and then 5 1st-Tues into 2024
  • Question for Katello QE (if present): How do you test for N-1 proxy compatibility?
    • a lot of hand-testing
    • once basic smoke tests pass, move to N-1
    • sync upstream all content-type repos, then SmartProxy sync the results
    • QE uses robotello tests, will be getting enhanced
    • there’s a response “somewhere” in the foreman-community on this
  • Katello PR feedback:
  • hstct: publish-time for deb-repos got really long
    • due to on-demand/RemoteArtifact lack of indexing
    • https://github.com/pulp/pulpcore/pull/4836 adds an index
    • publish-time grew from 8 hours to 20+ at large installation
    • AI: hstct to report back on customer experience once applied
    • may cause specific issues in pulp-deb due to its code-path
    • specifically, affects installations w/ large amounts of on_demand content and publishing
  • Support for zstd metadata · Issue #3254 · pulp/pulp_rpm · GitHub
    • need createrepo update as well?
    • yes - need 1.something
    • pulp_rpm patch updates createrepo_c dep as well
    • backported to rpm/3.18, .19, and .22

Action Items:

  • ggainey: schedule mtgs for 2024
  • iballou: summarize discussion RE global-repair into PR
  • iballou: Poke People to get other katello PRs reviewed
  • hstct to report back on customer experience once RemoteArtifact indices applied
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like