Katello/Pulp3 Integration meeting

[ggainey]

2024–01-10

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
• RPM
  • Working on Sat-Next BZs as our priority
  • Dalley out Tuesday - Friday next week
  • 2 weeks before BZ freeze (24-JAN)
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Time to think about Pulp versions for:
  • Katello 4.12 (need decision around Feb 1st)
  • Katello 4.13 / Satellite 6.16
  • Currently at core/3.43, next potential breaking-change release is core/3.55
  • Suggestion: Pick whatever is current as of 30-JAN
    • Are there any features that are not already in Pulp that are desired/required for katello-4.12/13?
    • Iballou to investigate
  • Def need to understand build-implications (if any)
    • Python-version-issues (3.11) already resolved
      • Modularity-fun ensued, but is resolved

QE

• not many updates - Sat-Next BZs under verification - 5 ON_QA / 3 VERIFIED

2024-01-24

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • 3.45 released
    • Certguard now “part of” core, just like pulp_file
  • core/3.28 wasn’t getting backports for Nov/Dec - fixed now, core/3.28.20 released
  • HTTPS proxy support now complete / merged, release incoming for 3.39
• RPM
  • Working through hyu’s suggested patch for 2253381
  • Pedro investigating whether there is anything to do re: 2252662 – satellite-6.11.5.6-1.el7sat.noarch has complete content-view module filter failure for RHEL 8 between 8.8 > 8.9 minor releases
    • There may not be, in which case Sat-Next has one less issue
  • The migration blocking issue turned out to be caused by manual index deletions on both community users
    • • another issue, both remove from Sat-Next
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica
• Pulp docs rework
  • See "Unified Docs" Plan - #4 by dkliban

Katello

• It’s unlikely that we’ll be able to squeeze in a Pulp upgrade for Katello 4.12 since time was lost due to shutdown + 4.11 delays
  • Delivery is working on “pulp nightly” and a new Pulp installer method at the same time, which would make building a new version at the same time difficult
  • When Katello 4.12 branches, we will start testing Katello nightly with Pulpcore nightly to make it easier to upgrade for Katello 4.13.
    • Pulp cheers wildly
  • More info: Release Team Meeting 2024-01-17 - #2 by ekohl - Releases - TheForeman
• 2219885 – Satellite using the feature 'Alternate Content Source' is getting errors - blocker for Sat-Next?
  • Discussion ensues
• Using public URLS for testing?
  • Sometimes you need the Filthy Public Repo Data to test edge cases (rpm ecosystem esp)
  • fixtures.pulpproject.org is a good set of “standard” repos to test with
  • QE already uses them for some tests

QE

• we are approx in the first third of Sat-Next verification

[ggainey]

2024-02-07

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • HTTPS proxy support has been verified
    • A new BZ has been filed for Katello - manifest refresh isn’t working 2263159
• RPM
  • 2253381 investigation continues, now w/ reproducer on customer site
  • 2246247 repo-delete can cause a large memory-spike, dalley investigating
  • 2207867 2151657 repositories can contain duplicate NEVRA if the file paths are different
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• next-Y seems to be moving along well, curr-X has been verified
• next-Y+1 features in-progress (repo-repair, container-upload for examples )
  • core/3.39 dev-boxes for katello devs - work in progress
• Flatpak support being discussed

QE

Build Team

[quba42]

@ggainey

pulpcore: HTTPS proxy support has been verified

Can you expand on this? In particular I would be interested in the versions involved.

[ggainey]

This PR Taught https-download to trust system certstore. by ggainey · Pull Request #4951 · pulp/pulpcore · GitHub taught Pulp to know about the system-truststore-certs. Prior to that, trying to access a Remote via an HTTPS proxy that used a different CA than the Remote (which is the usual case!) would fail. (A long time back, you couldn’t even try, because aiohttp didn’t support https-proxies at all until…3.8? Maybe?)

This change requires py3.11 - prior to that, one had to depend on basically monkeypatching an internal attribute to get it to work. See fixes #3036 -Setup the SSL Context correctly for https proxies by parthaa · Pull Request #3038 · pulp/pulpcore · GitHub for what that looked like.

[ggainey]

2024-02-21

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • Update the table of versions
    • No changes for 4.12, 4.13 not set yet
• RPM
  • 2253381 fixed and released to All The Branches
• Ansible
• Pulp Container
  • Flatpak support should be done/available for katello to start consuming
  • question: are “flatpak images” different than OCI images?
    • As a user I want to differentiate between different nature of images · Issue #1437 · pulp/pulp_container · GitHub (introducing a flag for the Content ViewSet to better distinguish between flatpak-based or OCI-based images; easier consumption/filtering for Katello)
• OSTree
  • domains/rbac support added
• Pulp CLI
• Pulp Replica

Katello

• Work ongoing to integrate pulp repair on Capsules
• Ansible import BZ: 2262379 – Library import of an Ansible Collection repo fails
  • https://redhat-internal.slack.com/archives/C031713779N/p1706800620522639
• Flatpak support discussion, is being discussed
• Container push is refined and ready to be worked on for Katello 4.13

QE

• sat-Next - 11/13 verified, 2 circled back to DEV
  • 2223069 – Selinux denials are reported after following "Chapter 13. Managing Custom File Type Content" chapter step by step
  • 2263069 – pulp generated config.repo file lacks GPG public key

[ggainey]

2024-03-06

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • core/3.49 released
  • Reminder: core/3.55 is coming!
• RPM
  • Investigating some reported issues
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Choosing pulp-versions for katello413
  • Will wait till post-4.12-GA (19-MAR) to choose
  • Will grab whatever-is-newest
• Pulp-container: exposing labels/flags?
  • Pulp hoping to merge the work this week
    • Will update supported branches based on the Katello requirements if they pick up the latest available pulp-container version
  • Should make it into katello/4.14 UI
  • Ping iballou as soon as it’s released

QE

• mostly done with next-Y (two more BZs should land in today’s snap)
• one to be verified in next-Z
• Katello-flatpak-test-support still under discussion

[ggainey]

2024-03-20

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
• RPM
  • What timeout does orphan cleanup use by default?
    • "00000000" modular context chomped to "0" · Issue #3454 · pulp/pulp_rpm · GitHub
  • Looking at 2247646 – Capsule sync gets stuck forever after specific connection issues
• Ansible
• Pulp Container
  • 2.19 released with the labels support (As a user I want to differentiate between different nature of images · Issue #1437 · pulp/pulp_container · GitHub)
    • Will be in katello/4.13
  • NOT SET IN STONE: There is an idea to enable sync by manifests’ digests instead of tags
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Orphaned pulp content - #19 by gvde - Support - TheForeman discussion
• Locking in Katello 4.13 Pulpcore versions imminently
  • Planning to pickup pulp-container after 2.19 assuming it releases not too close to Katello 4.13 branching
  • core/3.49
  • May upgrade to container/2.20 if we can
  • Check in with pulp_rpm at the same time - there may be a new release that can be grabbed as well
• Plan has begun for integrating with new container manifest labels et. al.

QE

• Sat-Next/stream : everything VERIFIED!
  • Cheering happened

[ggainey]

2024-04-03

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • Released core/3.50 (and a lot of Zs) last week
• RPM
  • Released rpm/3.25.2
    • Includes repo_config gpgcheck value lost ? (mirror_complete option) · Issue #3462 · pulp/pulp_rpm · GitHub
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Good discussions w/ ATIX/pulp_deb
• When is the next pulp_container happening?
  • Iballou to ping lmjachky directly
• Upgrades looking good so far
• Query RE “protected repo version” : Github PR ?
  • Can we add a force/cascade flag?
  • Pro: Hard to get from repo-versions to have-distributions for some plugins
  • Con: Distribution is not hard-linked to repo-version
  • Suggestion: can we add an API to make it easier to go from “list of repo-versions” to “distributions pointing to them”
  • Observation: current error-msg doesn’t give us any help
  • AI: sjha to open an RFE - let’s move discussion there

[ggainey]

2024-04-17

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • core/3.51 released
  • Gunicorn fun NVD - CVE-2024-1135
    • Updated gunicorn-UB, backports released for core/3.22-28-39-49-51
• RPM
  • Nothing upstream should be affecting katello
    • Fix modulemd defaults upload by pedro-psb · Pull Request #3503 · pulp/pulp_rpm · GitHub
    • Might need backporting for upstream user
• Ansible
• Pulp Container
  • Still waiting for container/2.19 for katello
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Core-upgrade is going well
  • Waiting for releases to be RPM-d
  • Should have 3.49 branch soon
  • Looking into capsule-sync issues - may not be Pulp
• Container-push questions
  • Talk to lmjachky

QE

• Sat-Current BZs looking good
• Current testing looks good for Pulp
• Artifact-to-file linkage question

[ggainey]

2024-05-01

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • core/3.53 and 3.49.6 released
  • Q: recent pulp-container NFS benchmarks
    • New perf-test, or just reporting what Pulp team discovered?
    • Unsure - but probably based on Pulp numbers
• RPM
  • RE Performance
    • RHEL repos grow over time so can affect numbers
    • Some discussion ensued
• Ansible
• Pulp Container
  • Upgrade issue - v1-manifest-support bit us unexpectedly
    • Needed a katello fix
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• QE will be picking up container-push testing “soon”
• Sat-6.15 release may result in new issues (due to Actual Users)
• Sat-next testing starting
• Discussion RE Jira transition
  • Slack integration exists - talk to iballou!

[ggainey]

2024-05-15

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • Issue involving large exports (and imports) discovered (katello-4.12/core-3.39)
    • Export of large amount of packages will result in errors · Issue #5375 · pulp/pulpcore · GitHub
• RPM
  • Release this week or next week (rpm/3.26)
• Ansible
• Pulp Container
  • Released 2.20 with artifactless manifests (Changelog — Pulp container Support 2.21.0.dev documentation)
    • Shipping an updated django-management command to handle the migration of both labels/annotations and manifests
    • Already accepted/merged into katello
• OSTree
• Pulp CLI
• Pulp Replica
• Mtg 29-MAY - ggainey on PTO
  • Iballou to run the mtg

Katello

• Pulp-related feature dev is “all going fine”
  • On-track for Sat-Next features!
• RFE for filtering on manifests/labels? (pulp-container)
  • Maybe for next release?
• Foreman-birthday-party in DE coming
  • Foreman Birthday Event 2024 - Celebrate the 15th birthday - Events - TheForeman
  • Would be cool if anyone form Pulp could attend

QE

• Not much to report
• label/annotation testing of manifests starting next week
  • Expect incoming pings from QE

[ggainey]

2024-06-12

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • core/3.55 to be released #soon (next Tues?)
  • Next breaking-change for core will be 3.70
  • Working on how Jira integration will work w/ pulp’s process
• RPM

  • hyu@ addressed another advisory-merge-edge-case (#3587)

  • Publishing checksums - SHA1 Fun in katello

    • Katello addressed the issue
    • Katello now off of SHA1 and also removed deprecated functionality
• Ansible
• Pulp Container
  • As a user I want to mirror container manifests by digests · Issue #1563 · pulp/pulp_container · GitHub (revisiting sync by digest; compatibility with OSP and Satellite for existing customers)
    • Need to hear from PM about what/where/when/how - who should we be nagging?
• OSTree
• Pulp CLI
  • Reverted allowing requests-2.32 due to https regressions
• Pulp Replica
• Virtual PulpCon will be happening in November-ish
• Possible in-person team mtg - location/feasibility tbd
  • Sync up w/ local katello/foreman/qe folk?
• Bindings-generation-discussion continues apace in discourse
  • Current decision: not going to do this “now”
  • won’t block core/3.55
  • Isn’t a “breaking change” in the context of the plugin-api - so doesn’t really need to wait on a core-breaking-change release
  • Lots of discussion still to be had in discourse thread

Katello

• Container push support nearing completion
  • Under “heavy testing”, yay!
• Next up: upgrading to rpm/3.26

QE

• Verifications for Sat-Next are in-train

[ggainey]

2024-06-26

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • core/3.55 released
    • Plugin-compat-releases are largely available
    • The release train left the station! 3.55 and other new versions of Pulpcore are comming to town!
• RPM
  • Once it’s locked in that Satellite-Next will use pulp_rpm 3.26, let me know so I can update the supported versions list accordingly
    • Esp important since rpm/3.27 is out
• Ansible
• Pulp Container
  • Downstream supposedly wants us to start working on As a user I want to mirror container manifests by digests · Issue #1563 · pulp/pulp_container · GitHub (manifest sync by digest). Is there any deadline for it?
    • We need input from roadmap/product-planning Soon
    • FWIW, pulp-team has escalated from our side
• OSTree
• Pulp CLI
• Pulp Replica
• Discussion around Pulp no longer building bindings continues
  • See latest addition for discovering one of the expected problems “in the wild”

Katello

• Repo creation errors out with Ruby bindings · Issue #3639 · pulp/pulp_rpm · GitHub
  • Blocking RPM 3.27 upgrade
  • I’d offer to regenerate the bindings myself but we’d need to upload them somewhere, and they’ll likely get overwritten by the next z-stream upgrade of pulp-rpm.
  • Collides with delivery-pipeline process
  • Can we get some more problem-determination from the Ruby side?
• Katello would like to be on more-recent-rpm if possible

QE

• Sat-Current verifications in progress, no blockers AFAIK
• Some discussion about/around automating testing a scenario that requires a HUGE amount of data - prob will end up in performance-test-harness

[ggainey]

2024-07-10

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
• RPM
  • ACS/refresh bug updated, work in progress
• Ansible
• Pulp Container
  • core/3.55 compat-release incoming
• OSTree
• Pulp CLI
• Pulp Replica
• “Fun with Jira” continues - please nag us if it looks like we’re missing/messing up something (poke dalley)
  • Discussion ensues
• Pulp team will be in Brno 2-6 SEP
• Thinking about a Virtual PulpCON, first week(ish) of NOV

Katello

• rpm/3.26 packaged, being consumed by katello
  • AI: Dalley to add 3.26 to pulp_rpm supported versions

QE

• kudos to Ina, Luboš, Ian for fast fix of the header issue in Sat-Curr-Z

[ggainey]

2024-07-24

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
• RPM
  • RPM advanced-copy supported in CLI now
  • Ruby bindings patch (https://github.com/pulp/pulp_rpm/pull/3679)
• Ansible
• Pulp Container
  • Released pulp-container v2.21.0 compatible with pulpcore v3.55
  • Humberto started working on the non-existing blob HTTP 500 error (Requesting blob that doesn't exist results in 500 · Issue #1703 · pulp/pulp_container · GitHub)
    • Ian working with Humberto to get it reproduced
• OSTree
• Pulp CLI
  • Added support for prune and advanced-copy in rpm
• Pulp Replica

Katello

• Helping with Requesting blob that doesn't exist results in 500 · Issue #1703 · pulp/pulp_container · GitHub
  • Likely need to create a reproducer
• Planning to upgrade versions after Katello 4.14 branching. We’ll need a new line in the matrix
• Flatpak syncing in ostree from Flathub is coming up for discussion

[ggainey]

2024-08-07

Overview

• Katello Schedule

Pulp

• Pulpcore

  • Query of all changes from last two weeks
  • core/3.57 released

• RPM

  • Investigating publication-checksum-behavior issue
    • Syncing sha1-repo and then publishing resulted in sha1-publication
    • This is “expected behavior” - but may not make sense with current reality
      • Future behavior : https://github.com/pulp/pulp_rpm/pull/3461
    • Katello will adjust by requesting specific-csum-on-publish

• Ansible

• Pulp Container

• OSTree

• Pulp CLI

• Pulp Replica

Katello

• Workaround for https://issues.redhat.com/browse/SAT-26901 (SHA1 publication workaround)
• Busy w/ Sat-Next freeze
• Sat-Future features being discussed
  • will discuss any RFEs for Pulp as they are considered

QE

• the workaround ^ should unblock me at https://issues.redhat.com/browse/SAT-25515

[ggainey]

2024-08-21

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
• RPM
  • Discuss use of is_modular flag in katello
    • To be discussed “soon”, but not needed immediately
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Publish and Promote slow with 3.11/4.13 - #11 by gvde - Support - TheForeman
  • Pointers for adding Pulp DB indexes? We need to make sure it won’t break the next DB migration which will officially create the indexes.
• Container content questions
  • Do child manifests and manifest lists both get tags?
    • Seems both can be tagged
    • Lubos: Depends
  • Diff between labels and annotations?
    • Labels come from image configs
    • Annotations come from image manifests
    • Annotations were created for the OCI spec
  • Do child manifests and manifest lists get annotations
    • Only child manifests seem to get labels and annotations in Satellite, but maybe that’s not always true
    • https://github.com/opencontainers/image-spec/blob/main/image-index.md#image-index-property-descriptions (annotations get be set on indexes as well)
  • Should image-mode hosts be attached to manifests or manifest lists?
    • bootc images used – perhaps they can be manifests or lists?
    • Lubos+Ina: bootc implements container native tooling, so whatever the tag is pointing to it will figure things out; thus, we can operate on lists and regular manifests
  • Should is-flatpak be shown on manifest lists or just manifests?
    • It can apply to either
• Container bug (?)
  • Katello will open an issue w/ reproducer for below so it can be investigated

QE

• https://issues.redhat.com/browse/SAT-23051 - bumped into this one - looks like the fix is present in stream snap 71 but the card is still InProgress. Should be in Review instead?
  • There is a Big Automation Hole that needs to be filled by Not-The-Pulp-Team

[ggainey]

2024-09-18

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • Repo-delete memory/speed improvement has been merged
  • Trying to get a good reproduction of the ACS refresh issue https://issues.redhat.com/browse/SAT-18815
• RPM
  • Investigating high-memory-use when syncing third-party repo
    • Prob needs a jira-ticket if reproducible
• Ansible
  • https://issues.redhat.com/browse/SAT-27939 - ansible-capsule-sync-deadlock
• Pulp Container
  • Enforcing data migration for manifests soon-ish
    • Just a heads-up - may not affect katello (since they’ve already done it)
  • Need Ian’s input on https://issues.redhat.com/browse/SAT-24030 (as a user I want to mirror container images by digest): pulp_container: As a user I want to mirror container manifests by digests · Issue #1563 · pulp/pulp_container · GitHub
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Pulpcore upgrade “soon” - will be coordinating w/ some community needs
  • Decision in “a month(ish)”
• RHEL10 client/repo testing in progress with early releases

QE

• last (hopefully) 6.16.0 bug has been verified
  • thanks to Luboš for help with pulpcore-manager to seed 100k pulp tasks

[ggainey]

2024-10-30

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • PulpCON is next week!
    • Draft schedule
  • Pulp4 (!) discussions happening
    • PulpCON preso: Don’t Panic!
• RPM
  • Backports released for some fixes
    • Which caused bindings-issues that also needed a backport
  • Security review done
• Ansible
• Pulp Container
  • Cutting the release 2.22.0 on this Friday!
    • Diligently working on Add manifest arch, os, and compressed layers size fields to get merged so Katello can pick up a new version.
  • Leadership announcement
    • Gerrod is going to be a new lead for the plugin
• OSTree
  • Released a fix for the bindings issue(?)
    • Please, test the build with pulp-ostree 2.4.4.
  • Humberto will be the main contact.
• Pulp CLI
• Pulp Replica

Katello

• Katello has a PR open to add “Sync dependencies” support for ansible collection repos: Fixes #37958 - Add "Sync Dependencies" option to Ansible collection repositories by Thorben-D · Pull Request #11195 · Katello/katello · GitHub
• What does it mean for a plugin to not support not optimized sync?
  REST API - Pulp Project vs REST API - Pulp Project - Answered by pulp: Katello will continue to pass optimized flag to plugins where applicable and ignore for others.
• Need a release of core/3.63 for bindings fix - ggainey started

[ggainey]

2024-11-13

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
• RPM
  • Jira/github communications have been suboptimal
  • Discussion ensues
• Ansible
• Pulp Container
  • 2.22 released
• OSTree
• Pulp CLI
• Pulp Replica
• Pyproject.toml is being experimented with
  • In discussion with odilon

Katello

• Ian pushing for self-built API bindings for next(ish)-katello-release
  • ggainey cheers wildly
• Phoenix very likely soon to start team-triaging Pulp Jiras
  • Satellite vs pulp, do the subtask creation, etc

QE

• Nothing to mention yet