Katello/Pulp3 Integration meeting

2023-08-09

Overview

  • Katello Schedule

Pulp

Katello

QE

  • some help needed to reproduce/test 2125366 – Capsule sync fails with Error: It is not possible to add more than one advisory with the same id to a repository version

    • We have an upstream test - let’s get the link for QE to look at
  • How to track 2122872 and 2229788 (gunicorn memory leak Fun)

    • 2122872 originally had a workaround that we were going to move forwards with
      • It was cloned to 6.13.2 as 211957 and pushed, now CLOSED-ERRATA
      • Then we found a better fix, and we want to push that to Next-Z? And Sat-Next
      • We also intend to revert the workaround as it’s no longer needed
    • 2229788 was created for the “actual” fix
    • TL;DR how should this be tracked
      • Dalley suggestion:
        • Leave 2122872 tracking the “meta-issue” of gunicorn memory consumption, including the new lru_cache fix, and the reversion, for Sat-Next
        • Use 2122872 to track the improved fix and the reversion for Next-Z?
        • In both cases, we just test that the gunicorn memory consumption issue is resolved as expected
    • ‘2872 - used to handle the mitigation/restart Fun
    • ‘9788 - used for the ‘real’ fix
    • ‘1957 - cloned ‘2872 mitigation for 6.13 release
    • ‘9788 won’t fix 6.11/6.12 cases (since it’s fixing something that didn’t happen until 6.13)
    • What we settled on:
      • Keep 9788 for separately tracking the lru_cache fix (and set flags to clone it for 6.13)
      • Create a new bug to track the relaxed installer defaults (6.13 cloning optional)
        • Created 2230584
        • flags: 6.14.0?, 6.13.z?
      • QE these together with the original bug, so that QE effort is not duplicated
        • (the “actual” fix + relaxation) for each release
      • Email Brad, Justin and Vijay Singh (QE mngrs), Shweta (QE contact for already delivered mitigations on 6.13) and Vlad (QE just for tracking), and Odilon (delivery) so everyone knows what to expect
      • The new BZs should be triaged on Aug 10th triage meeting, Vlad or William will provide the info.

2023-09-06

Overview

  • Katello Schedule

Pulp

Katello

  • Quba42 and pulp_deb/3.0
    • discussion/investigation in progress Right Now
    • Trying to see if we can fit into katello/4.10 release
    • Stay tuned!
  • BZ RE “tell users if module-streams have expired”?
    • Is this even available? (looks like Not)

QE

  • Couple BZs still being verified

(Catching up on notes - usual poster was unavailable)

2023-10-18

Overview

  • Katello Schedule

Pulp

Katello

  • 4.10 GA “imminent”, a lot of new core/3.28 users incoming!
  • 4.11 will be branching Real Soon Now
    • Once that happens, will be upgrading Pulp again
    • ggainey to update support-matrix - 4.11 versions same as 4.10

QE

  • Reviewed import/export perf-BZs from a test standpoint, looks good
    • QE ready to get the releases w/ the fixes

2023-10-04

Overview

  • Katello Schedule

Pulp

Katello

2023-09-20

Overview

  • Katello Schedule

Pulp

Katello

  • Need backport of https://github.com/pulp/pulp_ansible/pull/1561/ to fix syncing ansible content to smart proxies for Katello 4.10 (GAing next week)
    • Chatting with Gerrod about it now, awaiting opinions from Matthias
    • A backport to 0.18.0 would be ideal, but we could work with a release if it’s compatible with Pulpcore 3.28.

2023-11-01

Overview

  • Katello Schedule

Pulp

Katello

  • PulpCon schedule shared to the rest of the team, yay!
  • Pulpcore upgrade
    • Trying to get 3.40 into katello/4.11
    • Timeline is VERY SHORT
    • Client-api needs to work for N-1
    • any surprises will prob abort the effort

QE

  • Concern about core-update and N/N-1 api access requirement?
    • Iballou to experiment pre-snap/pre-QE
    • Vsedmik to coord w/iballou to get test-scenarios run automatically
    • Dalley proposes to run old-bindings against new-core and give feedback
    • Also katello VCR runs

Build Team

  • If core/3.40 is a Bridge Too Far, should we just choose core/3.39 from the start?
1 Like

2023-11-15

Overview

  • Katello Schedule

Pulp

Katello

  • “Goin crazy on core/3.39 pkg-building Fun”
  • Final fit-and-finish in progress on that effort

QE

  • “So far so good”
  • Testing against N-1 setup w/ core/3.39, also “so far so good”

2023-12-13

Overview

  • Katello Schedule

Pulp

  • Last meeting of the year!
    • Is this meeting still useful?
    • Is the cadence still useful?
    • Does the meeting time still work?
    • ggainey to schedule 6 months in 2024, if so
  • Pulpcore
  • RPM
    • Zstd publishing support added
      • Works for RHEL8.1+/9/10 (nut not RHEL7, no yum support there)
      • Not merged/released into pulp_rpm
      • Not the default, just a new option available
      • Could be exposed by a katello-UI widget “at some point”
    • Publishing with checksums weaker than sha256 removed
      • see 13-NOV minutes
      • Will be in pulp_rpm/3.25
      • Katello will need to decide what to do w/ the csum-choice-pulldown at publish button
      • EL5 may not support sha256 - but it has other, larger, issues w/ “modern” katello
      • Mirroring metadata works fine for older/ancient distributions
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI
  • Pulp Replica
  • N-1 problem
    • Is the communication between Satellite and its Capsules isolated at all?
    • There is a decent amount of separation - maybe we can make changes in just-smart-proxy comms to deal with this

Katello

  • Pulpcore nightly builds & Katello’s new stricter upgrade plan
    • Pulp upgrade was WAY more than anticipated
    • In future, will select the pulp-release for a branch at the branch-point, or very close to
    • Pretty close to having up-to-date pulpcore “nightly”
    • Env updates (ie, python3.next-X) - even more conservative

QE

2024–01-10

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • Working on Sat-Next BZs as our priority
    • Dalley out Tuesday - Friday next week
    • 2 weeks before BZ freeze (24-JAN)
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI
  • Pulp Replica

Katello

  • Time to think about Pulp versions for:
    • Katello 4.12 (need decision around Feb 1st)
    • Katello 4.13 / Satellite 6.16
    • Currently at core/3.43, next potential breaking-change release is core/3.55
    • Suggestion: Pick whatever is current as of 30-JAN
      • Are there any features that are not already in Pulp that are desired/required for katello-4.12/13?
      • Iballou to investigate
    • Def need to understand build-implications (if any)
      • Python-version-issues (3.11) already resolved
        • Modularity-fun ensued, but is resolved

QE

  • not many updates - Sat-Next BZs under verification - 5 ON_QA / 3 VERIFIED

2024-01-24

Overview

  • Katello Schedule

Pulp

Katello

QE

  • we are approx in the first third of Sat-Next verification

2024-02-07

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • 2253381 investigation continues, now w/ reproducer on customer site
    • 2246247 repo-delete can cause a large memory-spike, dalley investigating
    • 2207867 2151657 repositories can contain duplicate NEVRA if the file paths are different
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI
  • Pulp Replica

Katello

  • next-Y seems to be moving along well, curr-X has been verified
  • next-Y+1 features in-progress (repo-repair, container-upload for examples )
    • core/3.39 dev-boxes for katello devs - work in progress
  • Flatpak support being discussed

QE

Build Team

Can you expand on this? In particular I would be interested in the versions involved.

This PR Taught https-download to trust system certstore. by ggainey · Pull Request #4951 · pulp/pulpcore · GitHub taught Pulp to know about the system-truststore-certs. Prior to that, trying to access a Remote via an HTTPS proxy that used a different CA than the Remote (which is the usual case!) would fail. (A long time back, you couldn’t even try, because aiohttp didn’t support https-proxies at all until…3.8? Maybe?)

This change requires py3.11 - prior to that, one had to depend on basically monkeypatching an internal attribute to get it to work. See fixes #3036 -Setup the SSL Context correctly for https proxies by parthaa · Pull Request #3038 · pulp/pulpcore · GitHub for what that looked like.

1 Like

2024-02-21

Overview

  • Katello Schedule

Pulp

Katello

QE

2 Likes

2024-03-06

Overview

  • Katello Schedule

Pulp

Katello

  • Choosing pulp-versions for katello413
    • Will wait till post-4.12-GA (19-MAR) to choose
    • Will grab whatever-is-newest
  • Pulp-container: exposing labels/flags?
    • Pulp hoping to merge the work this week
      • Will update supported branches based on the Katello requirements if they pick up the latest available pulp-container version
    • Should make it into katello/4.14 UI
    • Ping iballou as soon as it’s released

QE

  • mostly done with next-Y (two more BZs should land in today’s snap)
  • one to be verified in next-Z
  • Katello-flatpak-test-support still under discussion

2024-03-20

Overview

  • Katello Schedule

Pulp

Katello

  • Orphaned pulp content - #19 by gvde - Support - TheForeman discussion
  • Locking in Katello 4.13 Pulpcore versions imminently
    • Planning to pickup pulp-container after 2.19 assuming it releases not too close to Katello 4.13 branching
    • core/3.49
    • May upgrade to container/2.20 if we can
    • Check in with pulp_rpm at the same time - there may be a new release that can be grabbed as well
  • Plan has begun for integrating with new container manifest labels et. al.

QE

  • Sat-Next/stream : everything VERIFIED!
    • Cheering happened

2024-04-03

Overview

  • Katello Schedule

Pulp

Katello

  • Good discussions w/ ATIX/pulp_deb
  • When is the next pulp_container happening?
    • Iballou to ping lmjachky directly
  • Upgrades looking good so far
  • Query RE “protected repo version” : Github PR ?
    • Can we add a force/cascade flag?
    • Pro: Hard to get from repo-versions to have-distributions for some plugins
    • Con: Distribution is not hard-linked to repo-version
    • Suggestion: can we add an API to make it easier to go from “list of repo-versions” to “distributions pointing to them”
    • Observation: current error-msg doesn’t give us any help
    • AI: sjha to open an RFE - let’s move discussion there
1 Like

2024-04-17

Overview

  • Katello Schedule

Pulp

Katello

  • Core-upgrade is going well
    • Waiting for releases to be RPM-d
    • Should have 3.49 branch soon
    • Looking into capsule-sync issues - may not be Pulp
  • Container-push questions
    • Talk to lmjachky

QE

  • Sat-Current BZs looking good
  • Current testing looks good for Pulp
  • Artifact-to-file linkage question

2024-05-01

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last two weeks
    • core/3.53 and 3.49.6 released
    • Q: recent pulp-container NFS benchmarks
      • New perf-test, or just reporting what Pulp team discovered?
      • Unsure - but probably based on Pulp numbers
  • RPM
    • RE Performance
      • RHEL repos grow over time so can affect numbers
      • Some discussion ensued
  • Ansible
  • Pulp Container
    • Upgrade issue - v1-manifest-support bit us unexpectedly
      • Needed a katello fix
  • OSTree
  • Pulp CLI
  • Pulp Replica

Katello

  • QE will be picking up container-push testing “soon”
  • Sat-6.15 release may result in new issues (due to Actual Users)
  • Sat-next testing starting
  • Discussion RE Jira transition
    • Slack integration exists - talk to iballou!

2024-05-15

Overview

  • Katello Schedule

Pulp

Katello

QE

  • Not much to report
  • label/annotation testing of manifests starting next week
    • Expect incoming pings from QE
1 Like

2024-06-12

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • hyu@ addressed another advisory-merge-edge-case (#3587)

    • Publishing checksums - SHA1 Fun in katello

      • Katello addressed the issue
      • Katello now off of SHA1 and also removed deprecated functionality
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI
    • Reverted allowing requests-2.32 due to https regressions
  • Pulp Replica
  • Virtual PulpCon will be happening in November-ish
  • Possible in-person team mtg - location/feasibility tbd
    • Sync up w/ local katello/foreman/qe folk?
  • Bindings-generation-discussion continues apace in discourse
    • Current decision: not going to do this “now”
    • won’t block core/3.55
    • Isn’t a “breaking change” in the context of the plugin-api - so doesn’t really need to wait on a core-breaking-change release
    • Lots of discussion still to be had in discourse thread

Katello

  • Container push support nearing completion
    • Under “heavy testing”, yay!
  • Next up: upgrading to rpm/3.26

QE

  • Verifications for Sat-Next are in-train
1 Like

2024-06-26

Overview

  • Katello Schedule

Pulp

Katello

  • Repo creation errors out with Ruby bindings · Issue #3639 · pulp/pulp_rpm · GitHub
    • Blocking RPM 3.27 upgrade
    • I’d offer to regenerate the bindings myself but we’d need to upload them somewhere, and they’ll likely get overwritten by the next z-stream upgrade of pulp-rpm.
    • Collides with delivery-pipeline process
    • Can we get some more problem-determination from the Ruby side?
  • Katello would like to be on more-recent-rpm if possible

QE

  • Sat-Current verifications in progress, no blockers AFAIK
  • Some discussion about/around automating testing a scenario that requires a HUGE amount of data - prob will end up in performance-test-harness
1 Like

2024-07-10

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • ACS/refresh bug updated, work in progress
  • Ansible
  • Pulp Container
    • core/3.55 compat-release incoming
  • OSTree
  • Pulp CLI
  • Pulp Replica
  • “Fun with Jira” continues - please nag us if it looks like we’re missing/messing up something (poke dalley)
    • Discussion ensues
  • Pulp team will be in Brno 2-6 SEP
  • Thinking about a Virtual PulpCON, first week(ish) of NOV

Katello

  • rpm/3.26 packaged, being consumed by katello
    • AI: Dalley to add 3.26 to pulp_rpm supported versions

QE

  • kudos to Ina, Luboš, Ian for fast fix of the header issue in Sat-Curr-Z