Katello/Pulp3 Integration meeting

June 8, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • Libmodulemd, RHEL7, and EPEL/F35+ discussion
    • Libsolv discussion
    • Depsolving work ongoing
      • PR submitted
    • Retain package versions sync optimization work nearly finished
      • “Don’t download RPMs you’re going to throw away”
    • pulp_rpm/3.18 coming “soon”
  • Migration
  • Ansible
    • No updates
  • Pulp Container
  • OSTree
  • Pulp CLI
    • No updates (still awesome)

Katello

QE

  • next-Y (snap 23) bz verification
  • Automation failure fixes
  • Automation for feature areas
  • prev-Z first snap expected Monday 13th

June 15, 2022

Overview

  • Katello Schedule

Pulp

Katello

QE

Build Team

June 22, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • 3.17.6 released
    • 3.18 probably in the next 2 weeks
    • Working with Waldirio on some customer cases / BZs
    • Module obsoletes support - will probably need to be backported to 3.17
      • Will eventually show up in RHEL9
  • Migration
    • Discussion around RH IT and their enormous Sat6.9 and migration
  • Ansible
    • Under discussion: Expanding the “Ansible Roles” API to allow for uploads
    • Is this useful for katello? discussion/brainstorming in order
      • Currently not supported at all, not sure if it’s “on the roadmap”
  • Pulp Container
    • Prep for 3.20 compat release
  • OSTree
  • Pulp CLI

Katello

  • How was ACS with RHUI tested?
    • Q: How does one get access to test when inside AWS?
    • Correct certs and correct header to talk to RHUI
  • Mtg RE Sat-IT-setup, discussion ongoing
    • Will work together “if weirdness shows up”
  • Ansible client consumption of collections?
    • AI: sjha to set up call, sjha/bmbouters/newswangerd to connect on What’s Changing
    • See #pulp-ansible in Matrix

QE

  • next-Y (snap 26) bz verification
  • prev-Z (snap 1) bz verification
  • Automation failure fixes
  • Automation for feature areas

June 29, 2022

Overview

  • Katello Schedule

Pulp

Katello

QE

  • next-Y (snap 26) bz verification
  • prev-Z (snap 1) bz verification
  • Automation failure fixes
  • Automation for feature areas

July 06, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • Orphan cleanup issue
    • Which pulp_rpm versions: rpm/3.14 and rpm/3.17 will be supported
    • Need a BZ for this to get it triaged and assigned to release(s)
  • Lots of discussion around releases/builds/process-expectations - fun!
    • Would be good to have current understanding written out

QE

  • 6.11 GA yesterday
  • 6.10.7 GA due today
  • next-Y Snap 1 expected today
  • Waiting for next-Z (no date set for snap 1)
  • Automation failure fixes
  • Automation for feature areas

July 13, 2022

Overview

  • Katello Schedule
    • Katello 4.5 - core/3.18, rpm/3.17, container/2.10, file/1.10

Pulp

Katello

QE

  • next-Z snap 1 bz verification
  • next-Y snap 1 bz verification
  • next-Y snap 2 is delayed till tomorrow
  • Automation failure and fixes for prev-Yand pre-next-Y branch
  • Automation for feature area

July 20, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • Which version of Sat should the ACS go to?
      • Katello reports “very tech preview in katello/4.5, full support katello/4.7”
    • Support for module-obsoletes is approved
      • Testing being worked on
      • Prob needs to get into curr-Y - needs PM approval [dalley to nag]
      • Discussion ensues
      • Mirrored metadata would be fine, but publishing own will lose the tag
    • BZ work continues
    • Can we consider limits/rethinking usecases around on repo-copy?
      • Discussion ensues
    • libmodulmd/zabbix question raised by a customer case
      • Ggainey to fwd to dalley
  • Migration
    • BSON size-problem BZ - Hao has potential PR drafted to address
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI
    • New release #soon (3.20 compat) - 0.15

Katello

QE

July 27, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • Display total size of a yum\docker\iso\file\custom repository in Satellite GUI after syncing them - https://issues.redhat.com/browse/SAT-2512
  • Processing metadata task should fail with an appropriate error message, whenever a task is stuck due to insufficient file system space. - https://issues.redhat.com/browse/SAT-2354
  • Can Pulp be configured to log outgoing requests for content downloads, for example, RPMs or metadata files? We’re trying to figure out how to verify that Alternate Content Sources are working.
    • No code currently, could possibly be added and backported
    • Alternatively: a man-in-the-middle proxy or firewall could be configured to log where requests are going
  • Should Katello use Pulpcore 3.20 or wait for 3.21?
    • Katello 4.7 stabilizes in ~ 3 months (November-ish). We estimate we need 1 month to upgrade Pulp versions (generous estimate). The delivery team needs about 2 weeks.
    • We can wait for Pulpcore 3.21, it should release around September
  • ACS issue: https://github.com/pulp/pulp_rpm/issues/2667
    • Figured it out, details are on the issue.

QE

  • curr-Z snap 1 testing
  • next-Y snap 3 testing
  • Feature testing
  • Automation failure fixes
  • Waiting on static ouia-id for get into a build

August 3, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • Quick chat on 1993917 – Syncing repositories with https proxy set ends with warning Katello::Errors::Pulp3 Error Only http proxies are supported
    • 2 cacerts are needed when talking to CDN via https proxy, one for proxy and one for cdn
      • The proxy cert is typically in the systems default CA trust store
      • Sounds like pulp2 requests library trusted it in 6.9 but pulp3 doesn’t seem to want to load it from the trust store and instead wants to be explicit on ca certificates to use.
      • If we have to stay explicit then it’ll be a lot of work from a katello perspective.
      • Let’s see if we can come up with approaches.
      • Paji doing a POC on the pulp-side, will report on progress

QE

  • curr-Z snap 1 build is scheduled for Friday
  • next-Y snap 4 testing
  • Feature testing
  • Automation failure fixes
  • Waiting on static ouia-id for get into a build

August 10, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last week
    • Paji’s HTTPS-Proxy PR under intense discussion
      • Pulp will reach out to paji to schedule
    • 3.18.6 and 3.16.12 released and already picked up by bldteam
  • RPM
    • New releases out (3.14.18, 3.17.9)
      • Include dup-nevra BZs that are def affecting customers w/ smart-proxies
    • Continuing to bang on BZs
    • Working with EXD to reduce the size of their metadata
    • BZ filed due to CDN not updating “updated_date” on some republished advisories
      • This breaks several workflows
      • There are Unhappy Customers as a result
  • Migration
    • BSON size fix that has been merged into 0.11 and 0.12
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI

Katello

  • Creating Discourse thread for Pulpcore backporting strategy to loop-in the community
  • Katello and pulp should talk about the simultaneous in-person NA meetups
  • Chatting w/ Odilon RE katello-4.7/AH syncing up on core/3.21

QE

  • curr-Z snap 1 bz verification
    • 4 pulp BZs showed up unexpectedly, being verified
    • Sign-off needs to happen by 26-AUG
  • next-Y snap 5 testing
    • Snap 6 on schedule for today (from email)
  • Feature testing
  • Automation failure fixes
  • Waiting on static ouia-id for get into a build
1 Like

August 17, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last week
    • 2 core/3.18 releases to update ruby-clients for faraday
      • Dkliban working w/ ewoud/zhunting
      • All the plugins also getting releases to match (in order to make installation work)
      • Hoping to get this fun resolved today
    • Need to meet w/ partha RE https proxy
      • AI: paji to schedule
    • Discussion w/ Galaxy RE “shared remote configurations”
  • RPM
    • Bug fixed for syncing SUSE repos (SLES)
    • Continuing to work on BZs, Foreman discourse reports
    • Ppicka working on missing-metadata w/ modules (reported by users)
      • Profiles, description
    • New releases out for 3.14 and 3.17
    • dalley taking over as Satellite Liason
    • fao89 joining the RPM miniteam!
  • Migration
    • Hao’s PR to address 2to3 query
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI

Katello

  • What’s up with Faraday? (NameError: uninitialized constant Faraday::FilePart → 0.17.5)
    • Faraday needs to be 0.16 (or 1.0.1, <2.0) to have the feature needed
    • Should prob be able to install/pin faraday >=1.0 <2.0, on dev/test boxes
  • https-proxy update
    • Foreman fix exists that katello can take advantage of (we think)
    • Paji to finish investigation before calling mtg w/ pulp-team
  • Iballou demoing at devconf - hope to see new user feedback for us!
  • Paji demoing fs-exporter work to Support team on 25-AUG

QE

  • prev-Z snap 1 bz verification
    • Sign-off needs to happen by 26-AUG
    • Still a half-dozen or so left to verify
  • next-Y snap 6.1 testing
  • Helping Grant https://bugzilla.redhat.com/show_bug.cgi?id=2062526
    • Deadlocks, ew
  • Feature testing
  • Automation failure fixes
  • Waiting on static ouia-id to get into a build
    • “This katello sprint” - or at least #soon

August 24, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • https proxy discussion continues

QE

  • curr-Z snap 2 bz verification
    • Sign-off this Friday, 26-AUG
  • next-Y snap 7 testing
  • Helping Grant 2062526 – Another deadlock issue when syncing repos with high concurrency
    • Fix is under-review and will be backported
  • Feature testing
  • Automation failure fixes
  • Waiting on static ouia-id to get into a build
    • CV OUIA-ID PR already merged, waiting for it to be in a build
    • Pending - Host details page OUIA-ID PR
  • Question: how long is “too long” to publish a CV where we have added 1 erratum and incrementally updating (ie, we’re doing depsolving)

August 31, 2022

Overview

  • Katello Schedule

Pulp

Katello

QE

  • next-Z snap 1 bz verification
    • Waiting on snap
  • next-Y snap 7 testing
    • Waiting on snap 8
  • Testathon is happening at the moment
  • Feature testing
  • Automation failure fixes
  • Waiting on static ouia-id to get into a build
    • CV OUIA-ID PR already merged, waiting for it to be in a build
    • Pending - Host details page OUIA-ID PR
  • Question: how long is “too long” to publish a CV where we have added 1 erratum and incrementally updating (ie, we’re doing depsolving)
    • Answer: 10-15m for rhel8_bos + rhel8_app for both incremental update and publishing cv

September 07, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • 3.18 blockers in the process of being merged
      • Still minor cleanup to finish up pre-release
    • Considering some changes that would allow pulp_rpm 3.18 to be used alongside pulpcore 3.18
      • Currently, as mentioned last week, it contains a feature that relies on pulpcore 3.19
    • Coordination going on with certain BZs / customer cases, hotfixes
  • Migration
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI

Katello

  • Testathon in-progress, so changes are on-hold for two weeks
  • Waiting patiently for core/3.21
  • RDU Tean meetups next week - should coord w/ pulp RDU meetup

QE

  • next-Z snap 1 bz verification underway
  • next-Y snap 9 bz verification
  • Feature testing
  • Automation failure fixes
  • Waiting on static ouia-id to get into a build
    • Some missing OUIA-ID for cv, sent BZ back to dev
    • Pending - Host details page OUIA-ID PR
  • Question: how long is “too long” to publish a CV where we have added 1 erratum and incrementally updating (ie, we’re doing depsolving)
    • Rhel8_baseos + Rhel8_appstream
      • Created a cv with a filter of before June 30th
      • Publish and did incremental update
        • Incremental update time: 10min 15 sec
        • Publishing cv after inc update: 10min 15 sec
      • Created a filter that includes up to July 31st
      • Publish and did incremental update
        • Incremental update time: 10min 15 sec
        • Publishing cv after inc update: 10min 15 sec
    • rhel7
      • Created a cv with a filter of before June 30th
      • Publish and did incremental update
        • Incremental update time: 9min 9 sec
        • Publishing cv after inc update: 13min 23 sec
      • Created a filter that includes up to July 31st
      • Publish and did incremental update
        • Incremental update time: 8min 49 sec
        • Publishing cv after inc update: 18min 35 sec
        • [Iballou] It would be interesting to see the breakdown between “pulp time” and “katello time” for this
        • Katello-indexing should be much faster now, would be good to know if that isn’t the case
    • Note: depSolving is not selected for these runs, but iirc, doing filtering and publishing does the depsolving?
      • In general usage, it does not - katello only asks Pulp to depsolve, if the user asks for it explicitly
      • Incrementals depsolve by default, so they do

September 21, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • 3.18.2 is out, I had a preliminary agreement from Sat that they would be OK with having this in 6.12 - 3.18 is just a generally a small bugfix release that includes some migrations
    • 3 still aligned to sat-next that aren’t complete, 1 in danger of not making the cutoff
  • Migration
    • New release 0.17 , that doesn’t affect katello (plugin-eol-dates)
  • Ansible
    • 0.15 releasing today
      • Requires core/3.21
      • Bugfixes and 3 new features
  • Pulp Container
    • Pulp-container 2.9 and 2.10 is out with bugfixes for sat-next
  • OSTree
  • Pulp CLI

Katello

  • core/3.21 RPMs “almost done”
    • This will be for katello/4.7
  • Question: what needs to happen for https://github.com/pulp/pulpcore/pull/3188?
    • Multiple issues, needs multiple CHANGES/ entries
    • Discussion ensued around multiple-commits or not

QE

September 28, 2022

Overview

  • Katello Schedule

Pulp

Katello

QE

  • next-Z snap 2 bz verification underway
    • Sign off this Friday 9/30
  • next-Y snap 12 bz verification
    • Expecting 2 more snaps
    • ggainey to work w/ ltran to test potential deadlock fixes in ‘real’ environments
  • Feature testing & automation
  • Automation failure fixes
  • Ouia-id for both cv and host details are in
    • Any missing IDs will prob be in later releases

October 5, 2022

Overview

  • Katello Schedule

Pulp

Katello

QE

  • next-next-Z is scoped
  • next-Y snap 13 bz verification
    • One last snap incoming
  • Feature testing & automation
  • Automation failure fixes
  • Ltran last day is 10/7
    • Lai to tell ggainey who the right QE contact will be for this mtg

October 12, 2022

Overview

  • Katello Schedule

Pulp

Katello

QE

  • Um…who will be covering Pulp/this mtg going forward?
    • Katello, Pulp Sat Liason to work w/ QE mgt to get definite answer

Build Team

October 19, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • Pulp Telemetry – are you more interested in users outside of Katello?

    • We’re defaulting it to be off for now, but it can be turned on via custom hiera (Puppet).
    • If you do want Katello user telemetry, we’ll need to figure out how to advertise turning it on.
    • Pulp is interested in katello installs, yes
    • Let’s start with “exposed but opt-in via custom hiera”, sociallize w/ katello/foreman community, and see where we go from there
  • Working on pulpcore 3.21 upgrades. Gems are available. Thanks! Osousa looking into pulp_deb release.

    • Branching for katello-next is Monday(ish)
1 Like

November 2, 2022 (26-OCT skipped)

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • Continuing to work on errata issues
    • Progress being made on concurrent repo syncs
    • Some discussion about deprecating default profiles upstream
      • Thinking about how that impacts us as we’ve just added a related feature (to show profile metadata)
    • Lots of BZ cleanup happening (yay dralley!)
  • Migration
    • QE testing for 2to3-migration-version release
    • ggainey working w/ vsedmik
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI

Katello

  • AWS RHUI – What is missing in Katello for basic support?
    • Only the two special headers?
  • https://github.com/pulp/pulp_container/issues/1122
    • Updated client-libraries can’t talk to old-smart-proxy
    • E.g., 3.21-client can’t talk back to 3.18-pulp
    • AI: sjha to pull a group together to talk through this complicated issue
      • Include vsedmik
  • Reminder – does Pulp always copy an erratum if all of its packages are copied too?
  • Telemetry changes are merged and will be off by default in katello 4.7

QE

  • 3 BZs to be verified for 2to3-migration-version (1 FailedQA but fix PR is already up (thanks ggainey), 2 under test)

November 16, 2022 (9-NOV skipped for PulpCON)

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • Domains-in-rpm planning started
    • BZs in progress
  • Migration
    • 0.11.3 released for last 6.9.10 zstream
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI

Katello

  • Conflicting Meetings make moderators Sad

QE

November 30, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • More progress on dropping the libmodulemd dependency
    • Lots of BZs being escalated by support, trying to investigate
    • ppicka will be leaving RH, RPM Team somewhat undercapacity for a bit
  • Migration
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI

Katello

QE

Build Team

November 23, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last week
    • Pulpcore 3.22.0 scheduled for Nov 30, but will likely slip by one week
      • All tests converted to Pytest and build team should be able to run them on the RPMs they build.
  • RPM
    • Considering allowing users to skip treeinfo file. How is katello using the skip types?
  • Migration
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI

Katello

  • Pulp_deb monkey patch in the works. Should make it to 4.7 GA.