Katello/Pulp3 Integration meeting

[ggainey]

2023-03-22

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • core/3.23 file/1.13 and certguard/1.6 released
    • Domain support
    • We need to fix some lowerbounds - will be .1 releases Soon
  • Gonna stop publishing nightlies to pypi - is that ok?
    • What about rubygem nightlies?
    • AI: ggainey to inform team “yes we can stop doing that”
• RPM
  • In the process of fixing the API in 3.19, encountered some issues with module depsolving. Seems to have been broken for a while.
    • I’ll try to put the test up at least, but nobody has complained so far, so it’s not a high priority
  • Working on PR to fix modulmd-packages-import-export being broken
    • Hyu has a great PR, will be backporting the fix
    • BZ 2178176 – Satellite showing errata from module streams not installed on client as upgradable/installable when content is imported (not synced)
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica
  • PRs in progress for file and rpm

Katello

• Looks like 4.8 should be out “soon”, is that correct? Yes. Nightlies are green and so is 4.8 after pulp_rpm 3.19.2. We have an action item to re-record our VCRs assuming format of data returned is changed.
  • Does katello turn on pulp-analytics in the upstream? - No
  • Still opt-in on installer
• Working on refining epics for Next-Y
  • Doesn’t look like many Pulp impacts (yet)
• katello/4.8 looks to be unblocked now, woohoo!

QE

• no updates at all - all ON_QA 6.13 BZs have been resolved

[ggainey]

2023-04-05

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • Release
    • core/3.21.6 and 3.18.16
    • file/1.14
• RPM
  • Bugfix for import/export of modulemds
  • Bugfix for uploaded package 404 issue
    • Dalley working on tool to fix already-existing problems
  • 2122872 – pulpcore-api gunicorn process consumes high memory next on prio list
  • rpm/3.20 coming #soon
    • Domain-support and an RFE (restrict acceptable pkg-signing-keys for a repo)
• Ansible
• Pulp Container
  • container/2.14.4
• OSTree
  • Had its first GA release
• Pulp CLI
• Pulp Replica

Katello

• 2155644 – When importing the content in a disconnected satellite, the customer is facing "Error: 502 Bad Gateway"
  • OOM issue with import? Curious if you think there could be an improvement.
  • Import-time fix : 2170485 – Importing Red Hat Repository Import on Disconnected Red Hat Satellite taking huge time around 5 hours (not released yet?)
  • Memory-fix: 2061224 – MemoryError when importing large repo to disconnected Satellite
    • Released last year : Red Hat Customer Portal - Access to 24x7 support and knowledge
• What happens to the contents on disk if a content import fails? Do they stick around?
  • If so, where? And when will Pulp clean it up?
    • There is suspicion that these files stay in the cache. Reportedly there were times when, after a failed import, the next import would require more storage space than the last.
  • Possibly depends on exactly what fails when - need data please
  • Pulp does not touch/erase the import-files themselves
• Sync-up on https://issues.redhat.com/browse/SAT-16225
  • Tips for collecting data about whether or not incremental-update needs dependency solving?
    • Discussion around breaking-scenarios
    • Dalley’s tool can find “rpms not ‘covered’ by errata”
      • Provide timestamp-per-repo of release-timestamp, repo-urls
  • Docs draft will be written by us devs and then passed on to the documentation team
  • Is this an ask for the Pulp team to take on this task? Because if so, that’s new.
    • Nope!
    • Pulp to collaborate on review
• 2174912 – [RFE] add ability to skip syncing .treeinfo files (skip .treeinfo)
  • “Pulp side” released in rpm/3.18 and 3.19 already
  • See 2046195 – Katello::Errors::Pulp3Error: Treeinfo file should have INI format for the original “Broken upstream” error traceback

QE

• Shweta picked up incremental-export-sync after switching from on_demand-to-immediate

[ggainey]

2023-04-19

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • Bugfix releases
  • What does katello think about accepting performance-improvements for which there are not currently BZs?
    • https://github.com/pulp/pulpcore/issues/3711
    • https://github.com/pulp/pulpcore/issues/3714
    • Discussion ensued
      • If upstream would like it backported, go for it
      • If Pulp wants QE’s help to test a thing, it must have a BZ and follow the process
• RPM
  • Bugfix releases
  • Memory-use in gunicorn investigation continues 2122872 – pulpcore-api gunicorn process consumes high memory
  • Pulp_rpm domain work continues
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
  • 0.19.0 released
• Pulp Replica

Katello

• Any upgrade wishes for the next few Katello releases?
  • core/3.23 has domain work
  • core/3.25 will have django4
  • katello/4.9 branching in “a month”, core/3.22
  • Maybe katello/4.10 will aim for core/3.25(ish)

QE

• One BZ pending for z-stream
  • 2184122 – wrong metadata if uploaded rpm have different name than name in rpm

[ggainey]

[missed posting previous minutes]

2023-05-03

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • core/3.24 released
    • Having Some Issues, will update
  • core/3.25 is coming #soon (May-ish)
    • Moves to Django4.2, psycopg3
    • katello/4.Next should prob think about being core/3.25 or later
• RPM
  • Fix for the profiles issue that Ian reported on Slack is ongoing
  • 3.20 release today
    • No domains yet, alas
• Ansible
• Pulp Container
  • Plugin leadership change, goes to Lubos!
  • Ggainey will add him to invite
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Core-repair API will be in-use for capsules
  • Nudge lmjachky about container-issues involving repair
    • This is a core issue https://github.com/pulp/pulpcore/issues/3606
  • Katello will need fixes “soon”
    • Added prio-list label on repair related issues
  • Thoughts about large/manyrepo/huge-dataset systems - per-repo, or global-repair?
  • Discussion needs to happen! Poke lmjachky?

QE

• Nothing to yell about
• Sat6.13 released today!
  • And there was much rejoicing

[ggainey]

2023-05-17

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • core/3.26 released
  • OpenTelemetry basic support merging into pulpcore
    • Will be available for use by perf-test
• RPM
  • 3.17.18 released
  • 3.21.0 release soon
    • Mostly domains, core/3.25 required
• Ansible
• Pulp Container
  • Podman-push-repo-versioning is changing
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• 2142689 – Capsule sync fails when a CV is published and promoted to two capsule's LCEs at the same time
  • Capsule-sync-race-condition?
  • Needs more eyes, from both sides (pulp/katello)
• 2077363 – Fail to sync kickstart repositories with same sub repositories concurrently hotfix requested for Sat 6.11
  • 3.17.18 released with this fix
• How could 2174912 – [RFE] add ability to skip syncing .treeinfo files be given an automated test?
  • It seems it’s not the repo, but the web server that causes the original bug?
  • Idea: test .treeinfo file existence
  • Dalley: just test sync with / without treeinfo skipped and whether it downloads the .treeinfo. As you say, this just just a way of working around webserver problems, we don’t need to test the webserver.
    • We already have an upstream test for this Added 'treeinfo' to available skip-types at sync time. by ggainey · Pull Request #2913 · pulp/pulp_rpm · GitHub
    • I doubt it’s worth writing a 2nd automated test
• Working with OpenStack team on https://issues.redhat.com/browse/RHOSINFRA-4958
  • They’re having potential issues with Satellite & container content – may be just a usage issue but just FYI.
  • Let’s make sure lmjachky has access to test systems to help

QE

Build Team

• core/3.25 and Django4 is coming - heads up!

[ggainey]

2023-05-31

Overview

• Katello Schedule

Pulp

• Pulpcore

  • Query of all changes from last two weeks
    • OpenTelemetry initial support now baked in
  • 3.27 and a lot of X-releases this week
    • Release Day! May 30, 2023: Pulpcore 3.27 and more
  • core/3.25+ OCI images now available
  • Memory use investigation has made progress. Querying and Serializing the changelogs is pretty much the culprit.
    • Workarounds suggested, issues filed
    • https://github.com/theforeman/puppet-pulpcore/issues/286
    • https://github.com/theforeman/puppet-pulpcore/issues/285
    • Will be/is in latest foreman release
  • Potential fix for 2188504 – The "hammer export" command using single thread encryption causes a performance bottleneck. (export-compression and performance)
    • Note for us: if we want to propose API-lvl changes, we can “just open a Jira” for katello and all The Right People will be informed automagically
  • Rerun-chunked-import fix for 2173757 – Can't rerun a failed content-import task if it was exported using chunks under review

• RPM

• Ansible

• Pulp Container

  • 2.15 release compatible with pulpcore 3.25, django 4.2

• OSTree

• Pulp CLI

• Pulp Replica

Katello

• Katello/4.9 release in progress
• https://issues.redhat.com/browse/SAT-16055

QE

• Sat-next Snap1 is out, no pulp issues yet

[ggainey]

2023-06-14

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • Import-rerun merged, backported, released (3.18/21/22)
  • Release Day happened : Release Day! June 14, 2023: Pulpcore 3.28 and more
  • import/export-compression-fix also merged/released
• RPM
  • deadlock fix released, backported to rpm/3.17-18-19-21-22
    • https://github.com/pulp/pulp_rpm/pull/3002
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Does pulp have native backup/restore?
• 2151333 – Mirror complete sync policy no longer allowed with ignored content types in the repo.
• https://issues.redhat.com/browse/SAT-16330 Repair API - Docker content updates if any?
  • Is this https://github.com/pulp/pulpcore/issues/3606 ?
    • Yes, needs to be fixed “soon” (altho mayhap not ASAP)
  • See also 2127537 – [RFE] If an artifact gone missing from filesystem for any reason there is no way to easily redownload the same in Capsule 6.12
• Discussion RE katello/4.10 and core-release
  • Let’s try to get “further up” than core/3.25
  • Ian to look over the changelogs to see if core/3.28 would be good

QE

• Gunicorn “fix”?
  • Cfg-change to auto-restart workers “regularly” (not a pulp-change, a foreman-installer change)

[ggainey]

2023-06-28

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • 3.28/3.28.1 released
• RPM
  • 6.14 upgrade blocker
    • 2213515 – Upgrade to 6.14 fails on rpm.0048_artifacts_dependencies_fix migration with TypeError: 'NoneType' object is not subscriptable
    • Looks like data issue, investigation continues
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Sorting out some docker bugs in katello:
  • PR out for this: https://github.com/Katello/katello/pull/10629
  • 2 open issues: (manifest/tag deletion)
    • Feature #36539: Allow deleting tags and untagging manifests from Katello - Katello - Foreman
    • Bug #36538: Removing manifest referenced by tag doesn't remove it from pulp but deletes it from katello - Katello - Foreman
• Some katello API changes coming down the line (QE< go yell at Quinn)
  • Stems from removing Olde Pulp2 Stuffe (ggainey cheers wildly)

2159105 – Remove deprecated & not working API endpoints from APIdoc (as an example)

QE

[ggainey]

2023-07-12

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
• RPM
  • Backports for 0048 migration fix backported
    • 2213515 – Upgrade to 6.14 fails on rpm.0048_artifacts_dependencies_fix migration with TypeError: 'NoneType' object is not subscriptable
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
  • Fun with click-8.1.4; pinning to 8.1.3 for now
• Pulp Replica
• Starting to discuss how to be able to change REST API
  • very long term discussion - will link to this group when we get, say, a discourse thread

Katello

• 2219885 – Satellite using the feature 'Alternate Content Source' is getting errors
  • Not quite clear on what is going on here
  • Is there some “tuning” we need to/can do here (esp on the capsule side)
    • Eg, don’t currently have access to optimize/no-optimize on acs refreshes
• Finishing up Sat-Next testing
• Still waiting on full pulp-rpms for core/3.28 and friends
  • Some packaging snafu in-progress
• Hey - what is going on with Flatpak support?
  • Talk to pulp_container gang for discussion-status
• katello/satellite smart-proxy-sync discussions
  • Would like to insure capsule-N-2 works against satellite-N, always

QE

• Sat-Next verifications continue apace
• QE adding a bunch of smart-proxy-sync test cases

[ggainey]

2023-07-26

Overview

• Katello Schedule

Pulp

• Pulpcore

  • Query of all changes from last two weeks
  • Z-releases (including migration-fixes)
  • Gunicorn memory-leak fix released
    • 2122872 – pulpcore-api gunicorn process consumes high memory
    • https://github.com/theforeman/puppet-pulpcore/pull/295 - relaxes the number of restarts once ‘2872 is released

• RPM

  • Q: what version of pulp_rpm is being looked at for katello/4.10?
    • Impacts what we do nightly-tests on and backports-to
    • Also - file/container/ansible?
    • Discussion ensues
    • See Request for Pulpcore 3.28 builds - Releases - TheForeman for initial request
    • Will prob choose “most current release” of plugins at the 4.10 branch point (Aug 22ish) and core/3.28

• Ansible

• Pulp Container

  • [ipanova] https://github.com/pulp/pulp_container/issues/1329
    • In attempt to fix Repair API, we broke pulp-to-pulp sync aka SatServer→Capsule sync
    • 2223707 – Capsule docker sync fails with 406

• OSTree

• Pulp CLI

  • Pulp-replication commands merged
  • Prob should do a cli release “soon”

• Pulp Replica

  • Back under active support/development w/ dkliban back

• BZ process

  • See dralley’s ping on internal Slack RE BZs-in-POST that may be further along

Katello

• Does ACS refresh have an optimized option like remote syncs? - 2219885 – Satellite using the feature 'Alternate Content Source' is getting errors
  • By default, yes
  • Katello to do some testing around this BZ
• Pulpcore 3.28 repo in progress
  • We’ll be testing upgrading to 3.28 with Katello soon
  • #soon!! - Pulp r excite

QE

• RE ipanova’s pulp_container issue - QE tests are passing without?
  • Vsedmik to check test-versions, maybe poke ipanova as/if needed
• https://bugzilla.redhat.com/show_bug.cgi?id=2142689
  • May not be reproducible any more - vsedmik to do some investigation
  • Will prob eventually close? Depends on Sat-Current-Y testing

[ggainey]

2023-08-09

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • 3.30 released
  • A lot of performance-related investigation/fixes going on
• RPM
  • See above RE performance investigations
  • 2219885 – Satellite using the feature 'Alternate Content Source' is getting errors investigation (ACS refresh fun)
• Ansible
• Pulp Container
  • 2.16 released w/ flatpak support
• OSTree
• Pulp CLI
  • 0.21 released w/ pulp-replica support
• Pulp Replica
  • Some tls-fixes released

Katello

• core/3.28 - all katello/4.10 tests run, hit one unreproducible error
  • https://github.com/pulp/pulp_rpm/issues/3223
    • Not gonna be hi-prio unless/until we see it again
  • Hoping to bump pulp_container to 2.16 next

QE

• some help needed to reproduce/test 2125366 – Capsule sync fails with Error: It is not possible to add more than one advisory with the same id to a repository version

  • We have an upstream test - let’s get the link for QE to look at

• How to track 2122872 and 2229788 (gunicorn memory leak Fun)

  • 2122872 originally had a workaround that we were going to move forwards with
    • It was cloned to 6.13.2 as 211957 and pushed, now CLOSED-ERRATA
    • Then we found a better fix, and we want to push that to Next-Z? And Sat-Next
    • We also intend to revert the workaround as it’s no longer needed
      • The PR that does the reverting (i.e. we want this commit): https://github.com/theforeman/puppet-pulpcore/pull/295
  • 2229788 was created for the “actual” fix
  • TL;DR how should this be tracked
    • Dalley suggestion:
      • Leave 2122872 tracking the “meta-issue” of gunicorn memory consumption, including the new lru_cache fix, and the reversion, for Sat-Next
      • Use 2122872 to track the improved fix and the reversion for Next-Z?
      • In both cases, we just test that the gunicorn memory consumption issue is resolved as expected
  • ‘2872 - used to handle the mitigation/restart Fun
  • ‘9788 - used for the ‘real’ fix
  • ‘1957 - cloned ‘2872 mitigation for 6.13 release
  • ‘9788 won’t fix 6.11/6.12 cases (since it’s fixing something that didn’t happen until 6.13)
  • What we settled on:
    • Keep 9788 for separately tracking the lru_cache fix (and set flags to clone it for 6.13)
    • Create a new bug to track the relaxed installer defaults (6.13 cloning optional)
      • Created 2230584
      • flags: 6.14.0?, 6.13.z?
    • QE these together with the original bug, so that QE effort is not duplicated
      • (the “actual” fix + relaxation) for each release
    • Email Brad, Justin and Vijay Singh (QE mngrs), Shweta (QE contact for already delivered mitigations on 6.13) and Vlad (QE just for tracking), and Odilon (delivery) so everyone knows what to expect
    • The new BZs should be triaged on Aug 10th triage meeting, Vlad or William will provide the info.

[ggainey]

2023-09-06

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • core/3.33 released
    • Adds complex-filters - see Changelog — Pulp Project 3.33.0 documentation
  • import/export performance investigations happening
• RPM
  • Progress being made on Oracle + RHEL sync data corruption https://bugzilla.redhat.com/show_bug.cgi?id=2232500
    • Probably no patch for existing releases, but we can fix it going forwards
    • Some questions about how the migration should work
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica
• Virtual PulpCon: November 6-10, 2023, 0800-1200 GMT-5 (US EDT)
  • CfP ends 13-SEP
  • Can we get any talks from Katello?
    • [Ian] we’ll think of something!
    • Maybe fs-exporter from paji?
    • container-push design in katello?
  • Add to the CfP page!

Katello

• Quba42 and pulp_deb/3.0
  • discussion/investigation in progress Right Now
  • Trying to see if we can fit into katello/4.10 release
  • Stay tuned!
• BZ RE “tell users if module-streams have expired”?
  • Is this even available? (looks like Not)

QE

• Couple BZs still being verified

[ggainey]

(Catching up on notes - usual poster was unavailable)

2023-10-18

Overview

• Katello Schedule

Pulp

• Pulpcore

  • Query of all changes from last two weeks
  • Lots of performance-improvements in import/export merged and backported
  • See 4545 - we are working to no longer “reassemble” export-files on import
    • Requesting input from katello
    • QE: there may be a test specifically aimed at interrupting the import-process - vsedmik to review
  • [lmjachky] Katello: What are we going to do about this: Slow content proxy sync - #26 by lubosmj - Support - TheForeman
    • User reverted https://github.com/pulp/pulpcore/pull/4275 and things got better
    • User is on postgres12, we can’t reproduce but we’re on pgres-13
    • lmjachky/iballou to work together on getting older-pgres test sys up

• RPM

• Ansible

• Pulp Container

• OSTree

• Pulp CLI

• Pulp Replica

Katello

• 4.10 GA “imminent”, a lot of new core/3.28 users incoming!
• 4.11 will be branching Real Soon Now
  • Once that happens, will be upgrading Pulp again
  • ggainey to update support-matrix - 4.11 versions same as 4.10

QE

• Reviewed import/export perf-BZs from a test standpoint, looks good
  • QE ready to get the releases w/ the fixes

2023-10-04

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • Import/export performance fixes
• RPM
  • Zstd metadata support moving forwards:
    • Now a BZ open as it is in the wild 2241934 – After upstream repo switched to zst compression, Satellite 6.12.5.1 unable to sync
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Community support threads:
  • Slow content proxy sync - #19 by gvde - Support - TheForeman
  • Two streams with matching NSVCA - #11 by leziri - Support - TheForeman
    • Likely workaround: republish using “mirror complete”

2023-09-20

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • Potential performance regression? Slow content proxy sync - #7 by gvde - Support - TheForeman
    • Comparative performance tests would be helpful for catching these issues?
  • Lots of work on import/export performance
• RPM
  • Zstd metadata support
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Need backport of https://github.com/pulp/pulp_ansible/pull/1561/ to fix syncing ansible content to smart proxies for Katello 4.10 (GAing next week)
  • Chatting with Gerrod about it now, awaiting opinions from Matthias
  • A backport to 0.18.0 would be ideal, but we could work with a release if it’s compatible with Pulpcore 3.28.

[ggainey]

2023-11-01

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • 3.40 released! (requires plugin-updates to be compatible, happening this week)
    • Pulp_file now delivered as part-of pulpcore
    • Pulp_file will start versioning same-as core (ie jump to 3.40)
    • Need to verify/have solid answer that client-api rpm/gems still work
  • If katello/4.11 runs into probs w/ core/3.40 will drop back to core/3.39 and friends
  • In core/3.45, pulp-certguard will also be delivered as part of pulpcore
• RPM
  • New Pulp team member joining today - Pedro Pessoa
  • Pulp_rpm 3.23.0 released
    • Small update, one unbackportable BZ addressed here
      • 2232500 – Synchronizing Oracle Linux 9 AppStream breaks modularity in CVs with filters
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica
• PulpCon is NEXT WEEK!
  • PulpCon 2023 Schedule - HackMD

Katello

• PulpCon schedule shared to the rest of the team, yay!
• Pulpcore upgrade
  • Trying to get 3.40 into katello/4.11
  • Timeline is VERY SHORT
  • Client-api needs to work for N-1
  • any surprises will prob abort the effort

QE

• Concern about core-update and N/N-1 api access requirement?
  • Iballou to experiment pre-snap/pre-QE
  • Vsedmik to coord w/iballou to get test-scenarios run automatically
  • Dalley proposes to run old-bindings against new-core and give feedback
  • Also katello VCR runs

Build Team

• If core/3.40 is a Bridge Too Far, should we just choose core/3.39 from the start?

[ggainey]

2023-11-15

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • finding/fixing some subtle side-effects of merging pulp_file into core
  • pulp_file/1.6 released as a “dummy” package
  • Release in progress to address a BZ/issue
    • 2249847 – Syncing file:// repos can delete the original files
• RPM
  • Onboarding a new team member!
  • PR open to deprecate pkg/metadata csum-type, merge into one
    • Evaluate whether we really need to support specifying "metadata_checksum_type" and "package_checksum_type" separately · Issue #2480 · pulp/pulp_rpm · GitHub
    • https://github.com/pulp/pulp_rpm/pull/3306
    • Katello has a selector, and so will be affected
    • Currently katello only allows sha256/sha1
    • Not backportable, so will only impact future satellites
    • No real need to continue providing the option for sha1, so the decision is whether to remove the option entirely or replace sha1 option with sha512
      • Personally I feel sha512 support isn’t necessary and we’ve had no customer demand
• Ansible
• Pulp Container
  • [lmjachky] hosting bootable RHEL images in Pulp/Satellite - katello will need to support building images and pushing images into Pulp.
    • pulp-container has the support now (specifically, container-push)
    • Katello: “push” already in-plan for future release
    • Image-building support should be investigated/integrated into katello
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• “Goin crazy on core/3.39 pkg-building Fun”
• Final fit-and-finish in progress on that effort

QE

• “So far so good”
• Testing against N-1 setup w/ core/3.39, also “so far so good”

[ggainey]

2023-12-13

Overview

• Katello Schedule

Pulp

• Last meeting of the year!
  • Is this meeting still useful?
  • Is the cadence still useful?
  • Does the meeting time still work?
  • ggainey to schedule 6 months in 2024, if so
• Pulpcore
  • Query of all changes from last two weeks
  • Div-by-zero during import?
    • Fix is simple but improving tests before merging
    • How urgent is getting this fix out?
      • QE would like a fix “soon”
      • Next snap is next Weds - can we get it done before?
      • Let’s just take the fix for now, get an issue open for the test-work
  • Investigation ongoing for 2253381 – Failed incremental CV import shows error: duplicate key value violates unique constraint "rpm_updatecollectionname_name_update_record_id_6ef33bed_uniq"
• RPM
  • Zstd publishing support added
    • Works for RHEL8.1+/9/10 (nut not RHEL7, no yum support there)
    • Not merged/released into pulp_rpm
    • Not the default, just a new option available
    • Could be exposed by a katello-UI widget “at some point”
  • Publishing with checksums weaker than sha256 removed
    • see 13-NOV minutes
    • Will be in pulp_rpm/3.25
    • Katello will need to decide what to do w/ the csum-choice-pulldown at publish button
    • EL5 may not support sha256 - but it has other, larger, issues w/ “modern” katello
    • Mirroring metadata works fine for older/ancient distributions
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica
• N-1 problem
  • Is the communication between Satellite and its Capsules isolated at all?
  • There is a decent amount of separation - maybe we can make changes in just-smart-proxy comms to deal with this

Katello

• Pulpcore nightly builds & Katello’s new stricter upgrade plan
  • Pulp upgrade was WAY more than anticipated
  • In future, will select the pulp-release for a branch at the branch-point, or very close to
  • Pretty close to having up-to-date pulpcore “nightly”
  • Env updates (ie, python3.next-X) - even more conservative

QE

• 6.14.next BZs all done
  • Pulp cheers!
• container sync issue - lubos is on it
  • Might be https://github.com/pulp/pulp_container/pull/1133
• zstd 2241934 – After upstream repo switched to zst compression, Satellite 6.12.5.1 unable to sync
  • Fix needs to be released for 6.13 - already available as backports in appropriate upstream branches
  • Vsedmik to bug zstream triage, and/or bbuckingham/odilhao

[ggainey]

2024–01-10

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
• RPM
  • Working on Sat-Next BZs as our priority
  • Dalley out Tuesday - Friday next week
  • 2 weeks before BZ freeze (24-JAN)
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• Time to think about Pulp versions for:
  • Katello 4.12 (need decision around Feb 1st)
  • Katello 4.13 / Satellite 6.16
  • Currently at core/3.43, next potential breaking-change release is core/3.55
  • Suggestion: Pick whatever is current as of 30-JAN
    • Are there any features that are not already in Pulp that are desired/required for katello-4.12/13?
    • Iballou to investigate
  • Def need to understand build-implications (if any)
    • Python-version-issues (3.11) already resolved
      • Modularity-fun ensued, but is resolved

QE

• not many updates - Sat-Next BZs under verification - 5 ON_QA / 3 VERIFIED

2024-01-24

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • 3.45 released
    • Certguard now “part of” core, just like pulp_file
  • core/3.28 wasn’t getting backports for Nov/Dec - fixed now, core/3.28.20 released
  • HTTPS proxy support now complete / merged, release incoming for 3.39
• RPM
  • Working through hyu’s suggested patch for 2253381
  • Pedro investigating whether there is anything to do re: 2252662 – satellite-6.11.5.6-1.el7sat.noarch has complete content-view module filter failure for RHEL 8 between 8.8 > 8.9 minor releases
    • There may not be, in which case Sat-Next has one less issue
  • The migration blocking issue turned out to be caused by manual index deletions on both community users
    • • another issue, both remove from Sat-Next
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica
• Pulp docs rework
  • See "Unified Docs" Plan - #4 by dkliban

Katello

• It’s unlikely that we’ll be able to squeeze in a Pulp upgrade for Katello 4.12 since time was lost due to shutdown + 4.11 delays
  • Delivery is working on “pulp nightly” and a new Pulp installer method at the same time, which would make building a new version at the same time difficult
  • When Katello 4.12 branches, we will start testing Katello nightly with Pulpcore nightly to make it easier to upgrade for Katello 4.13.
    • Pulp cheers wildly
  • More info: Release Team Meeting 2024-01-17 - #2 by ekohl - Releases - TheForeman
• 2219885 – Satellite using the feature 'Alternate Content Source' is getting errors - blocker for Sat-Next?
  • Discussion ensues
• Using public URLS for testing?
  • Sometimes you need the Filthy Public Repo Data to test edge cases (rpm ecosystem esp)
  • fixtures.pulpproject.org is a good set of “standard” repos to test with
  • QE already uses them for some tests

QE

• we are approx in the first third of Sat-Next verification

[ggainey]

2024-02-07

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last two weeks
  • HTTPS proxy support has been verified
    • A new BZ has been filed for Katello - manifest refresh isn’t working 2263159
• RPM
  • 2253381 investigation continues, now w/ reproducer on customer site
  • 2246247 repo-delete can cause a large memory-spike, dalley investigating
  • 2207867 2151657 repositories can contain duplicate NEVRA if the file paths are different
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Pulp Replica

Katello

• next-Y seems to be moving along well, curr-X has been verified
• next-Y+1 features in-progress (repo-repair, container-upload for examples )
  • core/3.39 dev-boxes for katello devs - work in progress
• Flatpak support being discussed

QE

Build Team

[quba42]

@ggainey

pulpcore: HTTPS proxy support has been verified

Can you expand on this? In particular I would be interested in the versions involved.

[ggainey]

This PR Taught https-download to trust system certstore. by ggainey · Pull Request #4951 · pulp/pulpcore · GitHub taught Pulp to know about the system-truststore-certs. Prior to that, trying to access a Remote via an HTTPS proxy that used a different CA than the Remote (which is the usual case!) would fail. (A long time back, you couldn’t even try, because aiohttp didn’t support https-proxies at all until…3.8? Maybe?)

This change requires py3.11 - prior to that, one had to depend on basically monkeypatching an internal attribute to get it to work. See fixes #3036 -Setup the SSL Context correctly for https proxies by parthaa · Pull Request #3038 · pulp/pulpcore · GitHub for what that looked like.