Katello/Pulp-Community Integration meeting minutes

Minutes
#21

2022-12-13 1000-1030 GMT-5

Attendees: ggainey, sjha

Regrets:

Agenda:

  • meeting called
  • will schedule first 2023 meeting for 10-JAN!
    • meeting value/format to be evaluated at this mtg

Action Items:

#22

2023-01-10 1000-1030 GMT-5

Attendees: ggainey, iballou, sjha, quba42, hstct

Regrets:

Agenda:

Action Items:

  • AI: [ggainey] schedule 6 months starting first tues in March
  • AI: [ggainey] clean up mtg-doc
1 Like
#23

2023-03-07 1000-1030 GMT-5

Attendees: ggainey, iballou, sjha, quba42, hstct

Regrets:

Agenda:

Action Items:

#24

2023-04-04 1000-1030 GMT-5

Attendees: quba42, hstct, sjha, ggainey

Regrets:

Agenda:

  • Previous AIs:

  • Possibility of releasing pulp_deb orphan cleanup fix for released versions of Katello?

    • See: https://community.theforeman.org/t/katello-4-8-0-rc1-feedback/33039/4
    • fix includes a migration
    • it’s the only “new” migration since release
    • should be “safe” in all respects (idempotent, no other possibly-skipped migrations “in the way”)
    • which version backporting to?
      • pulp_deb/2.20.1, for katello/4.7 users

  • Any plans to drop Puppet 6 support? If so, with what Foreman/Katello version?

  • Adding a mix of synced and uploaded content in a single repository

    • ATIX opinion:
      • It is a best practice to use separate repos for the two use cases. (Especially in the case of deb/APT content)
      • Katello should be opinionated about enforcing best practice here.
      • Pulp probably does not need to be as opinionated about this as Katello should be.
        • Pulp opinion: Pulp should not/cannot be opinionated in this area
      • See also: RFC: Separating Sync and Upload workflows in pulp_deb (and others?) - #9 by Stephen_Herr
      • can we get a discussion opened on theforeman discourse to gather user-feedback?
      • At a minimum, can we get the katello UI to give some guidance to keep users from accidentally shoot themselves in the foot

Action Items:

1 Like
#25

2023-05-02 1000-1030 GMT-5

Attendees: hstct, sjha, quba42, ggainey, iballou

Regrets:

Agenda:

  • Previous AIs:

  • core/3.24 releasing TODAY

  • core/3.25 #soon - mid/end of May?

    • 3.25.0 blockers Milestone ¡ GitHub
    • django-4.2
    • psycopg3
    • any deprecation warnings? fix em now!
    • next breaking-release: core/3.40
    • discussion
      • packaging?
      • where do plugins look to find dperecation-warnings?
        • quba42 to ask in OpenFloor
    • AI: [ggainey] is there a list of core-releases we backport stuff to?
      • write this down somewhere
      • eg, “what releases does katello depend on”

  • pulp_deb release: Pulp_deb 2.20.2 has been released!

    • bindings change implied?
      • makes katello-packaging harder
      • serializer-change - almost certainly yes
    • can this be ready for katello/4.9 release?
      • should be possible
      • quba42 to investigate/do/poke katello if more info needed

  • SLES advisory export/import problem

  • Working with the Katello upload implementation:

    • https://github.com/Katello/katello/blob/master/app/lib/actions/pulp3/orchestration/repository/upload_content.rb
    • Katello is open to using the repository parameter on content endpoints instead of separately calling the modify API after creating the content.
    • If the upload workflow needs to change for plugins that implement “create or retireve” logic on the Pulp side, there are two options:
      • A flag in the existing workflow (if the two workflows are not very different)
      • A separate workflow that is called for some plugins
      • The goal should be to ultimately move all plugins to the new workflow and move “create or retrieve” logic from Katello to Pulp.

Action Items:

#26

2023-06-06 1000-1030 GMT-5

Attendees: hstct, sjha, quba42, iballou

Regrets: ggainey

Agenda:

Action Items:

#27

2023-07-11 1000-1030 GMT-5

Attendees: quba42, hstct, bernhards, ggainey, mbucher

Regrets: all of katello

Agenda:

Action Items:

2 Likes
#28

2023-08-01 1000-1030 GMT-5

Attendees: ggainey, quba42, hstct, iballou, sjha

Regrets:

Agenda:

  • Previous AIs:
  • We had a strange edge case where a Pulp publication failed because of full disk. (The Pulp sync was successfull) This means the repo was synced and all the packages showed up in Katello, but the content wasn’t actually being served in the CV. Re-syncing did nothing, because the Pulp re-sync reported no new packages, so Katello did not bother to re-publish. Ultimately we solved it by doing a “republish repository metadata” using hammer.
    • Are there plans for “republish repository metadata” to come back to the UI, or will it remain hammer only?
    • The above showed that if sync succeeds, but publish fails, regular Katello usage will never repair this situation, can this be improved?
    • there are lots of ways metadata might get corrupted
      • katello is restoring “republish” to the UI for non-mirror-complete repositories
      • there is a push to make it globally available in spite of ways that might Go Worng - discussion in progress
  • review of structured-apt writeup
    • will be in pulp_deb 3.0
    • 2 katello PRs, 1 subscription-manager PR
    • how to make forklist use unreleased versions of “things”?
      • iballou: usually just do a pip-install of source
    • orcharino test-setup - see quba42 for details
    • still edge-cases that need to be solved
      • prob need to get together w/ someone on candlepin to iron out some details
    • who should quba42 work with/demo to, to work thru the pulp_deb/katello/candlepin pieces?
      • paji/jlentz/croberts?
    • AI: [quba42] get forklift-env set up w/ POC uip and running
    • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
    • AI: [iballou] nag more folk to review the structured-content-doc
    • is this something katello/4.10 has-to have?
      • is there a pulp_deb compatible with core/3.28?
        • yes - 2.22 (we think)
        • is that the right version for katello/4.10? If not, tell iballou “now”
      • there exists a long data-migration to get to pulp_deb/3
    • notes from chat

Action Items:

  • RE structured-deb-content
    • AI: [quba42] get forklift-env set up w/ POC uip and running
    • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
    • AI: [iballou] nag more folk to review the structured-content-doc
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like
#29

2023-09-05 1000-1030 GMT-4

Attendees: quba42, sjha, iballou, ggainey

Regrets:

Agenda:

  • Previous AIs:
    • RE structured-deb-content
      • AI: [quba42] get forklift-env set up w/ POC uip and running
        • still in-progress
        • pulp_deb/3.0 gated us - but it has been released!!
      • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
      • AI: [iballou] nag more folk to review the structured-content-doc
        • lots of responses already in thread
  • pulp_deb 3.0 is released.
    • What Katello version should this be added to? (long running DB migration!)
    • requires matching client-gem packaging, implies a potentially long-running migration
    • prob too late for katello/4.10
    • let’s shoot for katello/4.11?
    • how does pulp_deb make sure this ends up with the right katello/foreman packaging dir?
      • foreman-installer will handle updating if/when pulp_deb/3.0 “magically shows up” in the katello/4.10-pulpcore/3.28 repo-structure
    • how to handle N-1 smartproxy setup?
      • can a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
        • this needs to be flattened first
        • quba42 to work on investigation here
  • pulp_deb bugfix releases, e.g.: https://github.com/theforeman/pulpcore-packaging/pull/713
    • should “just work”
    • add required PRs to approproiaite release-branches
  • What exactly is the state of pulp_rpm dependency solving?
  • would be Really Nifty to have pulp_deb participation in foreman-community-demos
    • every 3 weeks on Thurs?
    • this is a fine, fine idea

Action Items:

  • RE structured-deb-content
    • AI: [quba42] get forklift-env set up w/ POC uip and running
      • still in-progress
      • pulp_deb/3.0 gated us - but it has been released!!
    • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
  • quba42: investigate whether a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like
#30

[previous month’s minutes]

2023-10-03 1000-1030 GMT-4

Attendees: sjha, quba42

Regrets: iballou, ggainey, hstct

Agenda:

  • Previous AIs:
    • RE structured-deb-content
      • AI: [quba42] get forklift-env set up w/ POC uip and running
        • still in-progress
        • pulp_deb/3.0 gated us - but it has been released!!
      • AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
    • quba42: investigate whether a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
  • Remaining pulp_deb 3.0 packaging PRs:
  • Import/Export nil vs empty string issues. Has anyone else encountered this as a problem?
  • sjha: Question regarding smart proxy syncs for deb content:
    • Working on a new feature to display content counts for smart proxy syncs.
    • Noticed that Debian appeared to have two contents on the smart proxy for every one content on the server CV/Lcenv.
    • It appears every .deb package is duplicated on the smart proxy with two alternate relative paths.
    • Is this a bug?
      • quba42: This is an unintended design consequence of having both structured and simple publication mode on the server.
      • This will no longer be the case if we eventually transition to using only structued APT.
      • For now, we will just have to live with the fact that the content counts for .deb content will be double for the proxy.
      • Side note: For synced content the content count on the proxy should be exactly twice what it is on the server, however this may not be true for uploaded packages.

Action Items:

#31

2023-11-14

Attendees: quba42, iballou, mbucher, hstct, vsedmik, ggainey

Regrets:

Agenda:

  • Previous AIs:
    • quba42: Not yet any progress on structured APT (I got consumed in firefighting…)
  • Heal failed publications PR: https://github.com/Katello/katello/pull/10776
    • can we get some katello eyes to understand the failing test (and understand/approve)
  • Regarding: Slow content proxy sync - #38 by dralley - Support - TheForeman
  • Does Katello provide any features to trigger pulpcore’s artifact repair? Should it? (I am thinking a rake task or similar, or perhaps even a UI button?)
  • pulp_rpm publishes or syncs wrong modules.yaml:
  • UI performance:
    • We are getting customer feedback that it “feels like” things are growing slower and slower with each new version.
    • Note that these reports are mostly about Foreman 3.3, Katello 4.5
    • Question: Have you heard similar complaints? Did anything change (like system requirements, optimization confg)?
    • we have had specific performance issues, mostly on Really Large Instances
      • (list of closed perf-related issues here)
    • All hosts page: According to Bernhard Suttner a lot of data (including a lot of Katello related stuff) is loaded that is not even necessarily displayed, can this be optimized. (Examples: content_facets / concerns / host_managed_extensions)
    • from katello:
      • applicability has had some perf updates recently
      • host-queries can be really slow
      • RE katello/4.5: can’t think of any current complaints?
      • can we get specifics? please?
    • welcome to vsedmik (katello QE)

Action Items:

1 Like
#32

2023-12-05 1000-1030 GMT-5

Attendees: ggainey, mbucher, hstct, sjham iballou, dalley

Regrets: quba42

Agenda:

  • Previous AIs:
  • Meeting Review/Resched for 2024!
    • Still a useful mtg?
    • Still a useful cadence?
    • Still a useful time?
    • Yes to all
    • AI: ggainey to sched for 9-JAN, and then 5 1st-Tues into 2024
  • Question for Katello QE (if present): How do you test for N-1 proxy compatibility?
    • a lot of hand-testing
    • once basic smoke tests pass, move to N-1
    • sync upstream all content-type repos, then SmartProxy sync the results
    • QE uses robotello tests, will be getting enhanced
    • there’s a response “somewhere” in the foreman-community on this
  • Katello PR feedback:
  • hstct: publish-time for deb-repos got really long
    • due to on-demand/RemoteArtifact lack of indexing
    • https://github.com/pulp/pulpcore/pull/4836 adds an index
    • publish-time grew from 8 hours to 20+ at large installation
    • AI: hstct to report back on customer experience once applied
    • may cause specific issues in pulp-deb due to its code-path
    • specifically, affects installations w/ large amounts of on_demand content and publishing
  • Support for zstd metadata ¡ Issue #3254 ¡ pulp/pulp_rpm ¡ GitHub
    • need createrepo update as well?
    • yes - need 1.something
    • pulp_rpm patch updates createrepo_c dep as well
    • backported to rpm/3.18, .19, and .22

Action Items:

  • ggainey: schedule mtgs for 2024
  • iballou: summarize discussion RE global-repair into PR
  • iballou: Poke People to get other katello PRs reviewed
  • hstct to report back on customer experience once RemoteArtifact indices applied
  • ggainey to add minutes to Katello/Pulp-Community Integration meeting minutes
1 Like
#33

2024-01-09 1000-1030 GMT-5

Attendees: bsuttner, quba42, iballou, ggainey, dalley, pbrochado, sweta

Regrets:

Agenda:

Action Items:

#35

2024-02-13 1000-1030 GMT-5

Attendees: ggainey, mbucher, sja, iballou, quba42, hstct

Regrets:

Agenda:

  • Previous AIs:
  • meeting in person at conference was Really Useful!
  • fun with Forklift - lots of fixing has happened!
    • discussion between ATIX/katello
  • discussion around katello/UI bug in 4.11
    • needs an upstream issue opened?
  • discussing next-pulp-version for katello/4.13
    • prob whatever the latest-Y is at branch-point
    • would be good if a pulp_deb-Y right before katello-branch so katello gets latest
  • pulpcore discussed “supported branches” this week
    • discussion ensues
  • how can we get katello reviews happening more quickly on pulp-deb/ATIX Stuff?
    • can we get ATIX approve-access to the affected repos?
    • sure - there is A Process, let’s do it!
    • iballou to bring up w/ katello team
    • ATIX needs a candidate to start doing regular katello-reviews?
      • work w/ iballou/sja to iron out details
  • ATIX customer is “losing publications”?!?
    • heavy CVV user
    • old CVV “loses” pub?
    • ATIX investigating - rake cleanup task?
    • how can a repo-version exist, but associated publications are gone?!?
    • check if pulp was told to delete?
    • katello/4.7 (we think) is what’s in use
    • might be a related fix, in a newer katello?
  • sja: there exists a PR to “freeze” repo-access while a publication is happening
#36

2024-03-05 1000-1030 GMT-5

Attendees: ggainey, iballou, sjha, sweta, quba42, hstct

Regrets:

Agenda:

#37

2024-05-07 1000-1030 GMT-5

Attendees: ggainey, dosas, jbundesmann, mbucher, quba42, iballou, sjha, pbrochado, dalley, vsedmik, ssingh

Regrets:

Agenda:

  • Katello and robottelo
    • We are experimenting with robottelo tests
    • Since robottelo is owned by SatelliteQE, we wanted to ask (to the extent you can/want to tell us) how Katello and robottelo development interacts?
    • Who adds new tests?
    • Who fixes them if a Katello change breaks them?
    • If we think a test is broken how best to contribute a fix?
    • Who are the best contacts for questions?
    • See also:
    • iballou:
      • dev/qe work very closely together
      • QE makes the required changes/automation in robotello as features/bugs are being addressed by dev
      • long-standing backlog in robotello of “things to fix”, that QE is working their way through
      • process is being improved, but has more work to do
      • bandwidth is always an issue (as it is for everyone, everywhere, forever)
    • ATIX appreciates having someone to ping just to verify something is actually a bug/problem before opening a PR?
  • Fixes #37279 - Re-create remotes when the type needs to change by quba42 ¡ Pull Request #10942 ¡ Katello/katello ¡ GitHub
    • How to add better test coverage here?
    • maybe merge this fix, but plan to add some more tests around these odd edge-cases
    • iballou:
      • good to accept PR as-is
      • happy to do a peer-coding-exercise on how to add more tests, or rubber-duck on a PR
  • Are there plans to use the Pulp replicate features (for smart proxies or otherwise)?
    • some conversations happened at CfgMgt camp
    • sjha:
      • not on the roadmap yet
      • there are some katello-side constraints to be considered
      • still considered desireable
    • quba42: please remember to ping if/when/as we decide to start implementing
    • iballou:
      • N-2 syncing is higher prio
      • maybe can take advantage of replication?
      • pulp-glue?
      • proxies shipping their own bindings?
  • Does Katello have any special plans to handle this?
  • structured-apt-work is still in-progress
    • ATIX getting up to speed on “becoming a katello dev” :slight_smile:
      • Fun!
      • actions often missing the “why” of actions
      • iballou: technical debt is…A Thing :frowning:
    • iballou: absolutely up for getting together for a “community design review”
      • maybe record and get it up on katello’s youtube channel

Action Items:

#38

Pulp_ansible is about to ship a similar migration story. This one has the chance to fail with broken content.

1 Like
#39

2024-06-04 1000-1030 GMT-5

Attendees: hstct, iballou, ggainey, pbrochado, sjha, dalley, vsedmik, quba42

Regrets:

Agenda:

  • Previous AIs:
  • katello is moving to Jira for reporting!
  • pulpcore/3.55 is coming!
  • In Katello content can be “seen” at URL/pulp/repos/, but also at URL/pulp/content/ and by content type for example at URL/pulp/deb/ (does not appear to show just deb content).
    • Are any of these the “preferred” URL?
    • Are there any plans to ever deprecate/remove any of these duplicate locations?
    • discussion ensues
      • /pulp/content/ is the Correct Way
      • /pulp/repos/ exists for very backward compat (ie, to Pulp2)
        • currently no plans to do so
      • see Apache-configs for how these are proxied
      <Location "/pulp/repos">
RequestHeader unset X-CLIENT-CERT
RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
</Location>
  • Slow smart proxy sync in 4.11
  • Are there plans to “containerize smart proxies” (we heard rumblings somewhere)?
    • “plans” is such a strong word :slight_smile:
    • maybe ask ehelms?
  • (If there is time) Structured APT update/challenges
    • progress is happening
    • candlepin-content-id saved in root-repository-model when creating a new repo
    • with structured-apt, need more candlepin-content-ids per repo-model
    • added to katello-repo-model
    • current PR makes sure non-apt-content doesn’t notice this change
    • discussion ensues RE ruby-dev-tools
  • Pulp discourse discussion RE bindings generation
  • don’t forget, foreman-bday-party is coming!
2 Likes
#40

2024-07-02 1000-1030 GMT-5

Attendees: quba42, iballou, sjha, dalley, dosos, hstct, ggainey

Regrets:

Agenda:

  • quba42: Thanks for the review and merge!: https://github.com/Katello/katello/pull/11044
  • quba42: Forklift questions/issues:
    • almalinux8-katello-devel-stable still using defunct CentOS 7 Stream repo files => broken untill next rebuild?
      • may have to purge/delete-image and then box-update maybe?
      • will still be on centos8-stream tho (which will still be broken?)
    • I built centos9-katello-devel successfully, but:
      • Candlepin is creating cp_contents instead of cp2_content?! Is this normal?! Never seen this before on a production system. (but things seem to be working?)
      • guess: cp2-content is Very Old (maybe?) - and decision was to use cp_contents?
      • check in w/ jlenz for clarity
    • I can’t build almalinux8-katello-devel possibly because my host system Ansible is to new? Fails with:
      msg: |-
  The following modules failed to execute: ansible.legacy.setup
    • iballou has hit something like this - check for multiple ansible-installs (ie system- vs pip-installed)
  • quba42: Structured APT: I have a partially working state I would like to present: https://github.com/Katello/katello/pull/11058
    • review/demo/comments/discussion ensue
    • maybe get feedback from jlenz/paji/toledo(Chris Roberts) RE candlepin side of this
    • work w/ iballou RE getting QE-test suite running against
    • katello does rely on candle-in-url to look up Things
    • is there a migration involved for this?
      • non-deb: no
      • deb-content: there is a rake-task to enable structured-app feature
        • rake-task still a work-in-progress
      • when the feature is selected/rake-task run, rake task does the data update
      • run during upgrade?
      • open for alternate proposals
    • The review-ask is “Is there anything that look Really Bad with this approach?”
  • katello upgrading to pulp_rpm/3.26 “now”
  • checkin: is this meeting format/frequency/approach still useful?
    • consensus: let’s keep going!
1 Like
#41

2024-08-06 1000-1030 GMT-5

Attendees: pbrochado, dosas, dralley, ggainey, mdellweg, iballou

Regrets:

Agenda:

  • Previous AIs:
  • Discussion about deprecating supported pulpcore branches.
    • example: https://github.com/pulp/plugin_template/blob/main/scripts/update_core_lowerbound.py#L35
    • (ggainey add example-scenario here)
    • we do semver for API, but not for plugin API
      • we have the deprecation-policy for plugin-api instead
    • we do want to make sure the lowerbounds-declarations are still valid
    • we want to maintain the broadest range (of dependency-requirements in the plugins) that we feel confident Actually Work
    • core gets rapid y-version-updates
      • and we can’t support all of them forever
      • so we have supported-branches for our downstream stakeholders
    • Sooo…removing a branch from supported-branches, should also impact all plugin-branches that currently require that branch
      • ie, any plugin-lowerbound that specifies (or is older-than) a now-unsupported branch, can now be updated to match the now-lowest-supported core-branch
      • if a supported-plugin-branch “breaks” as a result of a core-branch becomeing unsupported - then the implication is, that branch of the plugin is actually also unsupported
    • AI: [ggainey] can we get ocharino into a product-stakeholder-requires-versions matrix
    • AI: [dosas] bring this discussion to the Rest Of ATIX as well