Katello/Pulp3 Integration meeting

March 02, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last week
    • Releases for 3.18.1, 3.16.4, 3.17.4 (latter have MANY bugfixes)
    • Decreasing memory-use during sync in progress (see bmbouters for details)
    • Support of metalink URLs for remotes starting discussion - get your opinions in now!
  • RPM
    • 3.17.4 released
      • Fixes ruby bindings, one bugfix
    • BZ work continues
    • “Very close” to having “a” sync-memory mitigation ready
      • A few hundred megabytes per sync + improves the time by 60-120 seconds
      • Also pulpcore work in train for same
  • Migration
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI

Katello

QE

  • next-z bz verification, snap-2, most ppl done
  • nexy-Y bz verification snap 11
  • next-Y feature testing
  • Fixing automation failures

Build Team

March 09, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • 3.14.13 released yesterday
    • Working on a few issues reported on foreman forums
    • RBAC work continues, see ppicka for details
  • Migration
  • Ansible
    • ggainey to touch base w/ x9c4 RE adding lines here
  • Pulp Container
    • Support for cross repo blob mount is in
    • Support for upload of foreign layers is in
    • Support for manifest list push is in
    • Preparing for 2.11 release that will have content signing support plus mentioned features ^
    • Remote user auth fix is in, what branch do you want it backported to?
      • No need to backport, will take with core/3.18
    • Fixed a customer bug “ Can’t sync private repos from Azure registry”, what branch do you want it backported to? (2.8?)
  • OSTree
  • Pulp CLI

Katello

  • core/3.18 builds requested
  • “Move to core/3.17” has made it “on the board”
  • rpm/3.17 - in upstream, will be in next product snap
  • Working on client bindings update

QE

  • next-Z snap 1 tomorrow
  • prev-Y bz verification in progress for snap 2
  • next-X bz verification in progress for snap 12
  • next-X feature testing
  • Fixing automation failures

March 16, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • How critical is https://github.com/pulp/pulpcore/issues/2327 ?
    • GA’ing Katello 4.4 would mean forcing people up to Pulpcore 3.16 through the EOL’ing of Katello 4.2
    • It’s Important
  • Permission requirements for /var/lib/pulp/media/* and /var/lib/pulp/tmp/* ?
    • Is 0750 okay?
    • 755 would work as well - owner needs to be “user who’s running pulp services”
      • Or 770, if apache/pulp

QE

  • next-Z snap 2 bz verification in progress
  • next-Y bz verification in progress for snap 13
  • next-Y feature testing
  • Fixing automation failures

March 23, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last week
    • Significant memory reduction bugfix merged, and being backported
      • 27% reduction in memory with EL7 on_demand sync and 5% less runtime
      • Backporting will be happening #soon
    • RemoteArtifact duplicate key error bugfix merged, and being backported
    • Pulpcore now publishes its bindings docs
      • Pulpcore Client
      • Should we publish ruby-bindings-docs as well?
        • “Handy”, more for new folk coming on
  • RPM
  • Migration
    • 0.11.10 released w/ one bugfix (511)
  • Ansible
  • Pulp Container
    • Released 2.11
    • post-release work happening
    • Sync-pipeline refactoring in progress
  • OSTree
  • Pulp CLI
  • Pulp-openapi-generator
    • patch to expose timeouts merged
  • Release cadence - we might pick a “release day” once per week
  • Query: how does katello setup auth-backends?
    • Taken up post-mtg

Katello

QE (In QE all hands, won’t make it in this meeting)

  • prev-Z snap 1 bz verification. Waiting on snap templatization before verification
  • curr-Z snap 3 bz verification in progress
  • nexy-Y bz verification in progress for snap 14
  • next-Y feature testing
  • Fixing automation failures

March 30, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • Memory improvements to be merged released imminently (finally)
      • Combined with some pulpcore improvements that Brian worked on, the biggest syncs stay under 0.9 gb of memory consumption, down from ~3.5 at the release of Sat 6.10
    • Working on other small fixes
    • RBAC work progressing/finishing ip
  • Migration
    • Should consider EOL for 2to3 (core/3.20?)
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI
    • No updates
  • Can we update the pulpcore version being used in Sat-Next-Y?
    • Where/with whom should we discuss this?
    • core/3.17 heading for katello/4.5
    • dkliban/dalley to work w/ iballou on what this implies

Katello

QE

April 06, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last week
    • Should logs be i8ln?
    • Working to get functional and unit tests to be easy to run
      • Can we get pulp-tests run against a) katello-upstream, and b) downstream?
      • QE and/or build-team should be able to take advantage of this
      • Dkliban to bring up in Team Lead liaison mtg
    • Pulcore 3.19 is pushed back a week April 12, 2022
    • Pulpcore 3.20 will require you to upgrade your plugins to use it. 3.20 contains breaking changes in the plugin API (not the rest API) so plugins need to be upgraded to match. Each plugin will have a new release that is compat w/ pulpcore==3.20
  • RPM
    • Several bugfixes
    • Misparsed createrepo_c metadata
      • Been around for a while, no external reports yet
      • fix/cleanup script incoming
    • release(s) coming next Tues
    • Investigating removing sqlite metadata feature
      • Only used by repoview (which is unmaintained py2 code…)
    • Createrepo_c memory-use changes currently only planned for rpm/3.17
      • Could backport to rpm/3.14 but no current request
  • Migration
    • No updates
  • Ansible
  • Pulp Container
    • 2.8.z and 2.10.z releases are out
      • 2.8.z fixes axure-container-registry fix
      • 2.10.z fixes missing groups/perms
  • OSTree
    • dev-BZ-freeze #soon
    • Needs a GA-release
  • Pulp CLI
    • No updates

Katello

QE

  • prev-Y snap 2 bz verification in progress.
  • next-Y bz verification in progress for snap 15
  • next-Y feature testing
  • curr-Y BZ verification completed!
  • Fixing automation failures

April 13, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • 3.14.15 and 3.17.5 released
      • Data cleanup scripts, changelogs and ampersand-in-xml repair
      • Where/who runs the cleanup script?
      • Note RE changelogs: existing repos will not have their changelogs trimmed
        • When/who would run a cleanup script for existing installs?
    • Dralley to focus on depsolving issues (eg, 2070537)
  • Migration
    • When is (should be?) 2to3 removed?
    • 2to3 EOLs as of core/3.19
  • Ansible
    • 0.13.0, 0.12.1 released
  • Pulp Container
    • Lubos is giving a shot at enabling ‘pull-through cache’ on pull
      • Magical auto-discovery on pulp side
  • OSTree
  • Pulp CLI

Katello

QE

  • prev-Y snap 2 bz verification in progress.
    • A couple BZs left, signoff will be Monday
  • next-Y bz verification in progress for snap 16
  • next-Y feature testing
  • Fixing automation failures

April 20, 2022

Overview

  • Katello Schedule

Pulp

Katello

QE

  • curr-Z bz verification
  • next-Y bz verification in progress for snap 16
  • next-Y feature testing
  • Fixing automation failures
1 Like

[missed posting last week’s meeting, catching up]

April 27, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • Requesting feedback on https://github.com/pulp/pulp_rpm/issues/2479 and whether Katello / product has a desire for something like this
    • RBAC nearly complete, will land in 3.18
      • only works for core/3.19
      • Pulp_rpm 3.17 will remain supported for the foreseeable future, downstream should stick with it for now
  • Migration
    • No updates
  • Ansible
    • No updates
  • Pulp Container
    • No updates
  • OSTree
    • No updates
  • Pulp CLI
    • No updates

Katello

  • Almost fully switched to Pulpcore 3.17 now
    • 3.18 is next
    • Someone else needs 3.18, 3.19 will be “soon”
      • core/3.19 diffs vs 3.18 is “relatively small”
    • 4.5 dev-freeze next-week, branch is 10-May
  • Debian upload fix: Bug #34804: Debian package upload is broken - Katello - Foreman
    • Quba42 is best SME for info
  • Content Filter Overview
    • Pulp team (dalley, ggainey) need to review/comment this week
    • this is rpm-specific

QE

  • curr-Z bz verification (almost done!)
  • next-Y bz verification in progress for snap 18
  • next-Y feature testing
  • Fixing automation failures
    • Looking better and better

Build Team

May 4th, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • RBAC merged (will be in rpm/3.18)
    • SLES repo issue
      • SUSE metadata exceeds a DB field-len constraint
    • core/rpm - email to satellite-delivery@ to update zstreams for next 6.10.6 snap
  • Migration
  • Ansible
    • Experimentation w/ import/export happening right now
  • Pulp Container
    • Pulp-container 2.12 is released and is compat with pulpcore 3.19
  • OSTree
  • Pulp CLI
  • django-CVEs for “old” versions of Pulp (core/3.7)
    • Support-timelines for downstream are being extended
    • core/3.7 currently requires 2.2.16(17?)
    • What would it take for katello to upgrade to latest 2.2-LTS release (2.2.28?)
      • pulpcore/3.7 impact is noon-trivial
    • Is this “worth doing” at this point, given that 2.2 is already at EOL (as of April)
    • There will be a discussion/decision conversation happening #soon
    • Bmbouter to make sure iballou gets an invite

Katello

  • Katello will likely stick with 3.18 for Katello 4.5
    • Retaining el7 support, packaging resource constraints are the bottleneck
  • Reminder: If/when are publications/distributions destroyed in Pulp?
    • Publication-metadata isn’t (supposed to be) affected by orphan-cleanup
    • Removing the repo-version that a pub is associated with, removes the publication
  • Content Filter Overview
    • Pulp team (dalley, ggainey) REALLY need to review/comment
  • ACS getting to katello, so we will see user-contact upstream

QE

Build Team

May 11, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last week
    • Releases Happened 10-May (3.18.5, 3.16.8, 3.14.17)
    • Proposal: pulpcore==3.18 be declared the LTS and only release to other versions on a case-by-case basis. Next LTS will be 3.20 or greater.
  • RPM
    • Working w/ iballou/sjha RE depsolving docs
    • rpm/3.17 is main backport target
    • Features coming in rpm/3.18 (prob be core/3.20+ release)
  • Migration
    • No update
  • Ansible
    • No major changes (mostly for other stakeholders)
  • Pulp Container
    • Pulp-container 2.1 compat with django 2.2.28 is on the way (container/2.1.3)
      • Compat w/ core/3.7
  • OSTree
    • No update
  • Pulp CLI
    • No update
  • Other topics
    • Libdnf5 and product thoughts
      • who should libdnf team talk to?
      • Build? pm?

Katello

QE

  • prev-Z respin Just one upgrade bug, and it is verified
  • next-Z bz verification when build is dropped (e.t.a. 11th of May)
  • next-Y bz verification in progress
  • next-Y feature testing

Build Team

[missed posting minutes due to being on PTO (and brain going on PTO a day before I did…) Thanks to the folks who took notes from the meetings I wasn’t around for!]

June 1, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • Still waiting for Foreman packaging for Pulpcore 3.18 bindings

QE

  • next-Y (snap 22) bz verification
  • prev-Z is GA tomorrow
  • Bug 2082209 - Another deadlock issue when syncing repos with high concurrency <= Moved to next release
  • Automation failure fixes
  • Automation for feature areas

Build Team

May 25, 2022

Overview

  • Katello Schedule

Pulp

Katello

QE

Build Team

May 18, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • Regarding some questions from before:
    • LibDNF 5 future planning: Katello developers should be the first to talk with the LibDNF team, not managers or PMs. Ian is happy to do this and invites any others as well :-).
    • SLES repo issue: this isn’t a big deal for Satellite/Katello because SLES isn’t specifically supported any more.
    • Pulpcore 3.18 LTS: we should be all set from a Satellite and Katello standpoint. No further discussion with managers / PMs should be necessary

QE

Build Team

June 8, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • Libmodulemd, RHEL7, and EPEL/F35+ discussion
    • Libsolv discussion
    • Depsolving work ongoing
      • PR submitted
    • Retain package versions sync optimization work nearly finished
      • “Don’t download RPMs you’re going to throw away”
    • pulp_rpm/3.18 coming “soon”
  • Migration
  • Ansible
    • No updates
  • Pulp Container
  • OSTree
  • Pulp CLI
    • No updates (still awesome)

Katello

QE

  • next-Y (snap 23) bz verification
  • Automation failure fixes
  • Automation for feature areas
  • prev-Z first snap expected Monday 13th

June 15, 2022

Overview

  • Katello Schedule

Pulp

Katello

QE

Build Team

June 22, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • 3.17.6 released
    • 3.18 probably in the next 2 weeks
    • Working with Waldirio on some customer cases / BZs
    • Module obsoletes support - will probably need to be backported to 3.17
      • Will eventually show up in RHEL9
  • Migration
    • Discussion around RH IT and their enormous Sat6.9 and migration
  • Ansible
    • Under discussion: Expanding the “Ansible Roles” API to allow for uploads
    • Is this useful for katello? discussion/brainstorming in order
      • Currently not supported at all, not sure if it’s “on the roadmap”
  • Pulp Container
    • Prep for 3.20 compat release
  • OSTree
  • Pulp CLI

Katello

  • How was ACS with RHUI tested?
    • Q: How does one get access to test when inside AWS?
    • Correct certs and correct header to talk to RHUI
  • Mtg RE Sat-IT-setup, discussion ongoing
    • Will work together “if weirdness shows up”
  • Ansible client consumption of collections?
    • AI: sjha to set up call, sjha/bmbouters/newswangerd to connect on What’s Changing
    • See #pulp-ansible in Matrix

QE

  • next-Y (snap 26) bz verification
  • prev-Z (snap 1) bz verification
  • Automation failure fixes
  • Automation for feature areas

June 29, 2022

Overview

  • Katello Schedule

Pulp

Katello

QE

  • next-Y (snap 26) bz verification
  • prev-Z (snap 1) bz verification
  • Automation failure fixes
  • Automation for feature areas

July 06, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • Orphan cleanup issue
    • Which pulp_rpm versions: rpm/3.14 and rpm/3.17 will be supported
    • Need a BZ for this to get it triaged and assigned to release(s)
  • Lots of discussion around releases/builds/process-expectations - fun!
    • Would be good to have current understanding written out

QE

  • 6.11 GA yesterday
  • 6.10.7 GA due today
  • next-Y Snap 1 expected today
  • Waiting for next-Z (no date set for snap 1)
  • Automation failure fixes
  • Automation for feature areas

July 13, 2022

Overview

  • Katello Schedule
    • Katello 4.5 - core/3.18, rpm/3.17, container/2.10, file/1.10

Pulp

Katello

QE

  • next-Z snap 1 bz verification
  • next-Y snap 1 bz verification
  • next-Y snap 2 is delayed till tomorrow
  • Automation failure and fixes for prev-Yand pre-next-Y branch
  • Automation for feature area

July 20, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • Which version of Sat should the ACS go to?
      • Katello reports “very tech preview in katello/4.5, full support katello/4.7”
    • Support for module-obsoletes is approved
      • Testing being worked on
      • Prob needs to get into curr-Y - needs PM approval [dalley to nag]
      • Discussion ensues
      • Mirrored metadata would be fine, but publishing own will lose the tag
    • BZ work continues
    • Can we consider limits/rethinking usecases around on repo-copy?
      • Discussion ensues
    • libmodulmd/zabbix question raised by a customer case
      • Ggainey to fwd to dalley
  • Migration
    • BSON size-problem BZ - Hao has potential PR drafted to address
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI
    • New release #soon (3.20 compat) - 0.15

Katello

QE

July 27, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • Display total size of a yum\docker\iso\file\custom repository in Satellite GUI after syncing them - https://issues.redhat.com/browse/SAT-2512
  • Processing metadata task should fail with an appropriate error message, whenever a task is stuck due to insufficient file system space. - https://issues.redhat.com/browse/SAT-2354
  • Can Pulp be configured to log outgoing requests for content downloads, for example, RPMs or metadata files? We’re trying to figure out how to verify that Alternate Content Sources are working.
    • No code currently, could possibly be added and backported
    • Alternatively: a man-in-the-middle proxy or firewall could be configured to log where requests are going
  • Should Katello use Pulpcore 3.20 or wait for 3.21?
    • Katello 4.7 stabilizes in ~ 3 months (November-ish). We estimate we need 1 month to upgrade Pulp versions (generous estimate). The delivery team needs about 2 weeks.
    • We can wait for Pulpcore 3.21, it should release around September
  • ACS issue: https://github.com/pulp/pulp_rpm/issues/2667
    • Figured it out, details are on the issue.

QE

  • curr-Z snap 1 testing
  • next-Y snap 3 testing
  • Feature testing
  • Automation failure fixes
  • Waiting on static ouia-id for get into a build

August 3, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • Quick chat on 1993917 – Syncing repositories with https proxy set ends with warning Katello::Errors::Pulp3 Error Only http proxies are supported
    • 2 cacerts are needed when talking to CDN via https proxy, one for proxy and one for cdn
      • The proxy cert is typically in the systems default CA trust store
      • Sounds like pulp2 requests library trusted it in 6.9 but pulp3 doesn’t seem to want to load it from the trust store and instead wants to be explicit on ca certificates to use.
      • If we have to stay explicit then it’ll be a lot of work from a katello perspective.
      • Let’s see if we can come up with approaches.
      • Paji doing a POC on the pulp-side, will report on progress

QE

  • curr-Z snap 1 build is scheduled for Friday
  • next-Y snap 4 testing
  • Feature testing
  • Automation failure fixes
  • Waiting on static ouia-id for get into a build

August 10, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last week
    • Paji’s HTTPS-Proxy PR under intense discussion
      • Pulp will reach out to paji to schedule
    • 3.18.6 and 3.16.12 released and already picked up by bldteam
  • RPM
    • New releases out (3.14.18, 3.17.9)
      • Include dup-nevra BZs that are def affecting customers w/ smart-proxies
    • Continuing to bang on BZs
    • Working with EXD to reduce the size of their metadata
    • BZ filed due to CDN not updating “updated_date” on some republished advisories
      • This breaks several workflows
      • There are Unhappy Customers as a result
  • Migration
    • BSON size fix that has been merged into 0.11 and 0.12
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI

Katello

  • Creating Discourse thread for Pulpcore backporting strategy to loop-in the community
  • Katello and pulp should talk about the simultaneous in-person NA meetups
  • Chatting w/ Odilon RE katello-4.7/AH syncing up on core/3.21

QE

  • curr-Z snap 1 bz verification
    • 4 pulp BZs showed up unexpectedly, being verified
    • Sign-off needs to happen by 26-AUG
  • next-Y snap 5 testing
    • Snap 6 on schedule for today (from email)
  • Feature testing
  • Automation failure fixes
  • Waiting on static ouia-id for get into a build
1 Like