Katello/Pulp-Community Integration meeting minutes

Minutes
#31

2023-11-14

Attendees: quba42, iballou, mbucher, hstct, vsedmik, ggainey

Regrets:

Agenda:

  • Previous AIs:
    • quba42: Not yet any progress on structured APT (I got consumed in firefighting…)
  • Heal failed publications PR: https://github.com/Katello/katello/pull/10776
    • can we get some katello eyes to understand the failing test (and understand/approve)
  • Regarding: Slow content proxy sync - #38 by dralley - Support - TheForeman
  • Does Katello provide any features to trigger pulpcore’s artifact repair? Should it? (I am thinking a rake task or similar, or perhaps even a UI button?)
  • pulp_rpm publishes or syncs wrong modules.yaml:
  • UI performance:
    • We are getting customer feedback that it “feels like” things are growing slower and slower with each new version.
    • Note that these reports are mostly about Foreman 3.3, Katello 4.5
    • Question: Have you heard similar complaints? Did anything change (like system requirements, optimization confg)?
    • we have had specific performance issues, mostly on Really Large Instances
      • (list of closed perf-related issues here)
    • All hosts page: According to Bernhard Suttner a lot of data (including a lot of Katello related stuff) is loaded that is not even necessarily displayed, can this be optimized. (Examples: content_facets / concerns / host_managed_extensions)
    • from katello:
      • applicability has had some perf updates recently
      • host-queries can be really slow
      • RE katello/4.5: can’t think of any current complaints?
      • can we get specifics? please?
    • welcome to vsedmik (katello QE)

Action Items:

1 Like
#32

2023-12-05 1000-1030 GMT-5

Attendees: ggainey, mbucher, hstct, sjham iballou, dalley

Regrets: quba42

Agenda:

  • Previous AIs:
  • Meeting Review/Resched for 2024!
    • Still a useful mtg?
    • Still a useful cadence?
    • Still a useful time?
    • Yes to all
    • AI: ggainey to sched for 9-JAN, and then 5 1st-Tues into 2024
  • Question for Katello QE (if present): How do you test for N-1 proxy compatibility?
    • a lot of hand-testing
    • once basic smoke tests pass, move to N-1
    • sync upstream all content-type repos, then SmartProxy sync the results
    • QE uses robotello tests, will be getting enhanced
    • there’s a response “somewhere” in the foreman-community on this
  • Katello PR feedback:
  • hstct: publish-time for deb-repos got really long
    • due to on-demand/RemoteArtifact lack of indexing
    • https://github.com/pulp/pulpcore/pull/4836 adds an index
    • publish-time grew from 8 hours to 20+ at large installation
    • AI: hstct to report back on customer experience once applied
    • may cause specific issues in pulp-deb due to its code-path
    • specifically, affects installations w/ large amounts of on_demand content and publishing
  • Support for zstd metadata · Issue #3254 · pulp/pulp_rpm · GitHub
    • need createrepo update as well?
    • yes - need 1.something
    • pulp_rpm patch updates createrepo_c dep as well
    • backported to rpm/3.18, .19, and .22

Action Items:

  • ggainey: schedule mtgs for 2024
  • iballou: summarize discussion RE global-repair into PR
  • iballou: Poke People to get other katello PRs reviewed
  • hstct to report back on customer experience once RemoteArtifact indices applied
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like
#33

2024-01-09 1000-1030 GMT-5

Attendees: bsuttner, quba42, iballou, ggainey, dalley, pbrochado, sweta

Regrets:

Agenda:

Action Items:

#35

2024-02-13 1000-1030 GMT-5

Attendees: ggainey, mbucher, sja, iballou, quba42, hstct

Regrets:

Agenda:

  • Previous AIs:
  • meeting in person at conference was Really Useful!
  • fun with Forklift - lots of fixing has happened!
    • discussion between ATIX/katello
  • discussion around katello/UI bug in 4.11
    • needs an upstream issue opened?
  • discussing next-pulp-version for katello/4.13
    • prob whatever the latest-Y is at branch-point
    • would be good if a pulp_deb-Y right before katello-branch so katello gets latest
  • pulpcore discussed “supported branches” this week
    • discussion ensues
  • how can we get katello reviews happening more quickly on pulp-deb/ATIX Stuff?
    • can we get ATIX approve-access to the affected repos?
    • sure - there is A Process, let’s do it!
    • iballou to bring up w/ katello team
    • ATIX needs a candidate to start doing regular katello-reviews?
      • work w/ iballou/sja to iron out details
  • ATIX customer is “losing publications”?!?
    • heavy CVV user
    • old CVV “loses” pub?
    • ATIX investigating - rake cleanup task?
    • how can a repo-version exist, but associated publications are gone?!?
    • check if pulp was told to delete?
    • katello/4.7 (we think) is what’s in use
    • might be a related fix, in a newer katello?
  • sja: there exists a PR to “freeze” repo-access while a publication is happening
#36

2024-03-05 1000-1030 GMT-5

Attendees: ggainey, iballou, sjha, sweta, quba42, hstct

Regrets:

Agenda:

#37

2024-05-07 1000-1030 GMT-5

Attendees: ggainey, dosas, jbundesmann, mbucher, quba42, iballou, sjha, pbrochado, dalley, vsedmik, ssingh

Regrets:

Agenda:

  • Katello and robottelo
    • We are experimenting with robottelo tests
    • Since robottelo is owned by SatelliteQE, we wanted to ask (to the extent you can/want to tell us) how Katello and robottelo development interacts?
    • Who adds new tests?
    • Who fixes them if a Katello change breaks them?
    • If we think a test is broken how best to contribute a fix?
    • Who are the best contacts for questions?
    • See also:
    • iballou:
      • dev/qe work very closely together
      • QE makes the required changes/automation in robotello as features/bugs are being addressed by dev
      • long-standing backlog in robotello of “things to fix”, that QE is working their way through
      • process is being improved, but has more work to do
      • bandwidth is always an issue (as it is for everyone, everywhere, forever)
    • ATIX appreciates having someone to ping just to verify something is actually a bug/problem before opening a PR?
  • Fixes #37279 - Re-create remotes when the type needs to change by quba42 · Pull Request #10942 · Katello/katello · GitHub
    • How to add better test coverage here?
    • maybe merge this fix, but plan to add some more tests around these odd edge-cases
    • iballou:
      • good to accept PR as-is
      • happy to do a peer-coding-exercise on how to add more tests, or rubber-duck on a PR
  • Are there plans to use the Pulp replicate features (for smart proxies or otherwise)?
    • some conversations happened at CfgMgt camp
    • sjha:
      • not on the roadmap yet
      • there are some katello-side constraints to be considered
      • still considered desireable
    • quba42: please remember to ping if/when/as we decide to start implementing
    • iballou:
      • N-2 syncing is higher prio
      • maybe can take advantage of replication?
      • pulp-glue?
      • proxies shipping their own bindings?
  • Does Katello have any special plans to handle this?
  • structured-apt-work is still in-progress
    • ATIX getting up to speed on “becoming a katello dev” :slight_smile:
      • Fun!
      • actions often missing the “why” of actions
      • iballou: technical debt is…A Thing :frowning:
    • iballou: absolutely up for getting together for a “community design review”
      • maybe record and get it up on katello’s youtube channel

Action Items:

#38

Pulp_ansible is about to ship a similar migration story. This one has the chance to fail with broken content.

1 Like
#39

2024-06-04 1000-1030 GMT-5

Attendees: hstct, iballou, ggainey, pbrochado, sjha, dalley, vsedmik, quba42

Regrets:

Agenda:

  • Previous AIs:
  • katello is moving to Jira for reporting!
  • pulpcore/3.55 is coming!
  • In Katello content can be “seen” at URL/pulp/repos/, but also at URL/pulp/content/ and by content type for example at URL/pulp/deb/ (does not appear to show just deb content).
    • Are any of these the “preferred” URL?
    • Are there any plans to ever deprecate/remove any of these duplicate locations?
    • discussion ensues
      • /pulp/content/ is the Correct Way
      • /pulp/repos/ exists for very backward compat (ie, to Pulp2)
        • currently no plans to do so
      • see Apache-configs for how these are proxied
      <Location "/pulp/repos">
RequestHeader unset X-CLIENT-CERT
RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
</Location>
  • Slow smart proxy sync in 4.11
  • Are there plans to “containerize smart proxies” (we heard rumblings somewhere)?
    • “plans” is such a strong word :slight_smile:
    • maybe ask ehelms?
  • (If there is time) Structured APT update/challenges
    • progress is happening
    • candlepin-content-id saved in root-repository-model when creating a new repo
    • with structured-apt, need more candlepin-content-ids per repo-model
    • added to katello-repo-model
    • current PR makes sure non-apt-content doesn’t notice this change
    • discussion ensues RE ruby-dev-tools
  • Pulp discourse discussion RE bindings generation
  • don’t forget, foreman-bday-party is coming!
2 Likes
#40

2024-07-02 1000-1030 GMT-5

Attendees: quba42, iballou, sjha, dalley, dosos, hstct, ggainey

Regrets:

Agenda:

  • quba42: Thanks for the review and merge!: https://github.com/Katello/katello/pull/11044
  • quba42: Forklift questions/issues:
    • almalinux8-katello-devel-stable still using defunct CentOS 7 Stream repo files => broken untill next rebuild?
      • may have to purge/delete-image and then box-update maybe?
      • will still be on centos8-stream tho (which will still be broken?)
    • I built centos9-katello-devel successfully, but:
      • Candlepin is creating cp_contents instead of cp2_content?! Is this normal?! Never seen this before on a production system. (but things seem to be working?)
      • guess: cp2-content is Very Old (maybe?) - and decision was to use cp_contents?
      • check in w/ jlenz for clarity
    • I can’t build almalinux8-katello-devel possibly because my host system Ansible is to new? Fails with:
      msg: |-
  The following modules failed to execute: ansible.legacy.setup
    • iballou has hit something like this - check for multiple ansible-installs (ie system- vs pip-installed)
  • quba42: Structured APT: I have a partially working state I would like to present: https://github.com/Katello/katello/pull/11058
    • review/demo/comments/discussion ensue
    • maybe get feedback from jlenz/paji/toledo(Chris Roberts) RE candlepin side of this
    • work w/ iballou RE getting QE-test suite running against
    • katello does rely on candle-in-url to look up Things
    • is there a migration involved for this?
      • non-deb: no
      • deb-content: there is a rake-task to enable structured-app feature
        • rake-task still a work-in-progress
      • when the feature is selected/rake-task run, rake task does the data update
      • run during upgrade?
      • open for alternate proposals
    • The review-ask is “Is there anything that look Really Bad with this approach?”
  • katello upgrading to pulp_rpm/3.26 “now”
  • checkin: is this meeting format/frequency/approach still useful?
    • consensus: let’s keep going!
1 Like
#41

2024-08-06 1000-1030 GMT-5

Attendees: pbrochado, dosas, dralley, ggainey, mdellweg, iballou

Regrets:

Agenda:

  • Previous AIs:
  • Discussion about deprecating supported pulpcore branches.
    • example: https://github.com/pulp/plugin_template/blob/main/scripts/update_core_lowerbound.py#L35
    • (ggainey add example-scenario here)
    • we do semver for API, but not for plugin API
      • we have the deprecation-policy for plugin-api instead
    • we do want to make sure the lowerbounds-declarations are still valid
    • we want to maintain the broadest range (of dependency-requirements in the plugins) that we feel confident Actually Work
    • core gets rapid y-version-updates
      • and we can’t support all of them forever
      • so we have supported-branches for our downstream stakeholders
    • Sooo…removing a branch from supported-branches, should also impact all plugin-branches that currently require that branch
      • ie, any plugin-lowerbound that specifies (or is older-than) a now-unsupported branch, can now be updated to match the now-lowest-supported core-branch
      • if a supported-plugin-branch “breaks” as a result of a core-branch becomeing unsupported - then the implication is, that branch of the plugin is actually also unsupported
    • AI: [ggainey] can we get ocharino into a product-stakeholder-requires-versions matrix
    • AI: [dosas] bring this discussion to the Rest Of ATIX as well
#42

2024-09-18 1000-1030 GMT-5

Attendees: ggainey, sjha, mdellweg, iballou, markusb, quba42

Regrets:

Agenda:

  • Previous AIs:

  • Getting “Structured APT” ready for the initial merge: https://github.com/Katello/katello/pull/11058

    • PR is “feature complete”, but still being tweaked

    • Currently working on the suggestions by jeremylenz

    • ATIX has performed extensive testing of the “feature present but disabled” case (ensures the refactor bits don’t break anything):

      • Export/Import was tested
      • Internal robotello tests were run
      • Large syncs general usage

    • So long as the “disabled” case does not break anything we can merge, even it the feature itself is not perfect yet. Call it “tech preview”, "optional or just don’t tell people about it at first.

    • When does Katello next branch?

      • 2 months-ish from “now”
      • next pulp-upgrade will be one-month-ish from now
      • prob want this merged pre-pulp-upgrade

    • Proposal:

      1. Finish the suggestions by jeremylenz
      2. Have Bernhard or Markus perform a final code review and wait for their ack.
      3. Final chance for core Katello review/request for changes
      4. Merge.
      5. Keep improving the feature in small follow up PRs.

    • net: merge state above with “disabled”; improve “enabled” functionality with new (smaller) PRs

  • [markusb] default ContentView with limited Repo visibility or ContentView providing library-instance repos.

    • There is now a thread for this: Transparent Content View for Limiting Repository-Access in Katello - RFCs - TheForeman
    • This had been possible by limiting subscribed products on an ActivationKey-level (SCA removed that possibility).
    • Scenario: a registerd host should
      1. must have access to latest synced content
      2. must only see a subset of Repos
    • net: lots of discussion on associated foreman thread, incl a proposal - is katello happy for ATX to work on implementing this proposal?
      • AI: sjha would like to have a meetup w/ ATIX/katello/subscription-DE to flatten some issues

  • [quba42] Is the following (satellite) issue also planned for upstream? https://issues.redhat.com/browse/SAT-25398

    • Related question: Is red hat bugzilla being replaced?
      • Yes - Satellite is on Jira now
    • not currently high on the backlog to fix
    • needs an upstream-redmine for community-to-work-on

  • [markusb] Should creation of IncrementalCV with not-available content (e.g. rpm not in any repository) throw an error-message?

    • ATIX hit internally
    • katello: prob useful to have an error msg

  • [quba42] I think this can be closed: Feature #32514: Improve deb repo form regarding simple vs. structured publishing - Katello - Foreman

    • old ask for “Structured Apt”

  • [mdellweg] Inquiry RE foreman (?) PRs from ATIX not getting appropriate/timely attention?

    • possibly not an issue - PTO Happens :slight_smile:

  • [quba42] Is Katello planning to somehow apply indices separately from the pulpcore migration? Publish and Promote slow with 3.11/4.13 - #11 by gvde - Support - TheForeman

Action Items:

1 Like
#43

2024-10-01 1000-1030 GMT-5

Attendees: @iballou forgot to record attendees but it was more or less the “usual crew”

Regrets: not recording attendees

Agenda:

  • Structured-APT PR is ready for a final review
    • Tested on an internal production system by quba42
    • Just requires small remaining changes
  • Prototyping transparent/rolling/simplified content views starting soon
  • New release of pulp-deb would be good before Katello upgrades
    • Pain-point: updating pulp-deb Pulp CLI compatability each time Katello packages pulp-deb is cumbersome.
      • Potential fix: loosen the boundaries for pulp-deb Pulp CLI compatability. This could mean breaking changes get in, but it also means packaging happens more smoothly and people can get to testing it sooner.
  • Previous AIs:

Action Items:

3 Likes
#44

2024-11-12 1000-1030 GMT-5

Attendees: ggainey, sjha, hstct, quba42, mbucher

Regrets: iballou

Agenda:

Action Items:

#45

2024-12-03 1000-1030 GMT-5

Attendees: ggainey, quba42, mbucher, dalley, pbrochado, sjha, hstct, qjames, mdellweg

Regrets:

Agenda:

Action Items:

1 Like
#46

2025-01-07 1000-1030 GMT-5

Attendees: sjha ggainey mbucher quba42 vsedmik hstct

Regrets:

Agenda:

Action Items:

1 Like
#47

2025-03-04 1000-1030 GMT-5

Attendees: ggainey, vsedmik, qjames, quba42, mbucher, hstct

Regrets: iballou

Agenda:

  • Previous AIs:
  • [m-bucher]: Katello; Org with trailing . (e.g. ACME Inc.) => {"base_path":["The provided base path contains forbidden characters."]}
    • container-related - ORG with trailing-dot
    • Pulp is complaining, should katello sanitize?
    • is this container-only? pulp-content-app?
    • atix asking their support to experiment w/ deb/rpm
    • qjames: katello may have already fixed in 4.15-container-push
    • mbucher: will try against nightly
    • this may be specifically a not-legal-container-path Thing
    • if pulp is being “overly restrictive” around base-path-legality, we’ll need an issue to get that addressed
  • [quba42] Still open: Handle URL params for AptRepoFiles by quba42 · Pull Request #3454 · candlepin/subscription-manager · GitHub
    • ignored for 5 months?
    • let’s try and get tthis reviewed/merged
    • ptoscano is prob a person to nag
  • [quba42] Need packaging help: Update rubygem-pulp_deb_client to 3.5.1 by quba42 · Pull Request #11793 · theforeman/foreman-packaging · GitHub
    (Known)Issue or already fixed?
    • skip for now - needs a complicated-packaging-dance to address
    • might happen as part of the ongoing katello-branch-and-rebase-to-new-pulp
  • [quba42]: rolling CV: Fixes #38048 - Add rolling content views by quba42 · Pull Request #11240 · Katello/katello · GitHub
    • Katello 4.16 has branched so we want to finalize ASAP.
    • Lots of review and rework activity happening.
    • Open issues:
      • Smart proxy sync not re-syncing rolling CVs in all cases (maybe fixed now)
      • Missing repo names for rolling CVs on the smart proxy view
      • Container push repos need to be disallowed
      • Why is ::Actions::Katello::ContentView::AddToEnvironment async during rolling CV creation?
      • Hammer PR
      • docs PR
    • work is in-progress, will report if/when/as there may be roadblocks
    • PR might get taken over by mbucher
  • Availabilty: quba42 on leave for 4 weeks starting next week, nag hstct for pulp_deb or mbucher for katello-deb issues
  • [quba42] Potential future rolling CV extension: “I don’t want to sync all of Library to smart proxy just to use my hand full of rolling CV repos”
    • Possibility 1: Call it something different for rolling CVs, e.g.: “RollingLibrary” or just “Rolling”.
      • I don’t want to introduce another new “special” name.
    • Possibility 2: Allow “promoting” rolling CVs to other LCENVs?
      • Easy enough to implement, but does not make a lot of semantic sense.
    • Possibility 3: Allow some kind of filtering by CV on the smart proxy sync?
      • Sounds hard and potentially disruptive.
    • Any other ideas?
    • qjames: let’s rule out 1 above
    • qjames: 2 “makes more sense” (imnsho)
    • vsedmik: autosync-after-promotion
    • vsedmik: option 2 - breaks rolling-content-view “concept”
    • vsedmik: what about org-scoping libraries?
    • quba42: customers prob won’t be happy w/ introduced complications of org-scoping
    • consensus: proposal-1 is Right Out. Discussion needs to continue
    • finish the existing PR, and then perhaps open the discussion to a wider audience

Action Items:

1 Like
#48

2025-04-01 1000-1030 GMT-5

Attendees: mbucher, sjha, ggainey

Regrets:

Agenda:

  • Previous AIs:
  • We do not do “April Fools!” jokes in this meeting! :slight_smile:
  • pulp_deb and domains
  • [m-bucher] container-image naming in katello revisited
    • see last mtg’s discussion
    • katello’s container-name-regex more permissive than Pulp’s
    • Pulp’s regex is different than OCI’s? - why?
    • current katello-PR should fix immediate issue
      • it’s OK if katello is stricter-than Pulp’s regex

Action Items:

1 Like
#49

2025-05-06 1000-1030 GMT-5

Attendees: quba42, mbucher, hstct, sjha, ggainey, vsedmik

Regrets:

Agenda:

Action Items:

  • quba42: open a UX issue RE “green task that didn’t actually DO anything is…confusing”
  • quba42: update rolling-content-view/environments issue w/ discussion about approach being taken
  • hstct: work w/ SUSE to fix their repo?
  • ggainey: extend mtg to remainder of 2025
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like
#50

2025-06-03 1000-1030 GMT-5

Attendees: pbrochado, manisha, sjha, ggainey, mbucher, quba42, vsedmik, hstct

Regrets:

Agenda:

Action Items:

1 Like
#51

2025-07-01 1000-1030 GMT-5

Attendees: ggainey, mbucher, qjames, quba42

Regrets:

Agenda:

  • Previous AIs:
    • None
  • quba42: Update on “rolling environments”: https://github.com/Katello/katello/pull/11407
    • sync-to-proxy fix almost ready
    • needs more unit-tests to cover all the state-changes
    • quba42 will undraft and ping when he thinks it’s ready
    • katello pushing hard towards 4.18 dev-freeze - be aware
  • Q: ATIX customer issue w/ update-capsule-repository task “starts, task-finds-no-capsules, dynflow shows green and then ‘eventually’ foreman-task turns red”?!?
    • transient/not-reproduceable
    • have we seen this before? (alas, no)
    • no issue yet, just reported
  • katello-dev-server on a Mac VM (aarch64)
    • everything works except pulpcore, because RPMs only avail for x86_64
    • Pulp team doesn’t build RPMs, TheForeman Build Gang does
    • Can you point foreman/katello at an “external/containerized” Pulp?
    • discussion ensues