Katello/Pulp3 Integration meeting

ggainey · April 5, 2023, 8:53pm

2023-04-05

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- Release
  - core/3.21.6 and 3.18.16
  - file/1.14
RPM
- Bugfix for import/export of modulemds
- Bugfix for uploaded package 404 issue
  - Dalley working on tool to fix already-existing problems
- 2122872 – pulpcore-api gunicorn process consumes high memory next on prio list
- rpm/3.20 coming #soon
  - Domain-support and an RFE (restrict acceptable pkg-signing-keys for a repo)
Ansible
Pulp Container
- container/2.14.4
OSTree
- Had its first GA release
Pulp CLI
Pulp Replica

Katello

2155644 – When importing the content in a disconnected satellite, the customer is facing "Error: 502 Bad Gateway"
- OOM issue with import? Curious if you think there could be an improvement.
- Import-time fix : 2170485 – Importing Red Hat Repository Import on Disconnected Red Hat Satellite taking huge time around 5 hours (not released yet?)
- Memory-fix: 2061224 – MemoryError when importing large repo to disconnected Satellite
  - Released last year : Red Hat Customer Portal - Access to 24x7 support and knowledge
What happens to the contents on disk if a content import fails? Do they stick around?
- If so, where? And when will Pulp clean it up?
  - There is suspicion that these files stay in the cache. Reportedly there were times when, after a failed import, the next import would require more storage space than the last.
- Possibly depends on exactly what fails when - need data please
- Pulp does not touch/erase the import-files themselves
Sync-up on https://issues.redhat.com/browse/SAT-16225
- Tips for collecting data about whether or not incremental-update needs dependency solving?
  - Discussion around breaking-scenarios
  - Dalley’s tool can find “rpms not ‘covered’ by errata”
    - Provide timestamp-per-repo of release-timestamp, repo-urls
- Docs draft will be written by us devs and then passed on to the documentation team
- Is this an ask for the Pulp team to take on this task? Because if so, that’s new.
  - Nope!
  - Pulp to collaborate on review
2174912 – [RFE] add ability to skip syncing .treeinfo files (skip .treeinfo)
- “Pulp side” released in rpm/3.18 and 3.19 already
- See 2046195 – Katello::Errors::Pulp3Error: Treeinfo file should have INI format for the original “Broken upstream” error traceback

QE

Shweta picked up incremental-export-sync after switching from on_demand-to-immediate

ggainey · April 19, 2023, 4:12pm

2023-04-19

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- Bugfix releases
- What does katello think about accepting performance-improvements for which there are not currently BZs?
  - https://github.com/pulp/pulpcore/issues/3711
  - https://github.com/pulp/pulpcore/issues/3714
  - Discussion ensued
    - If upstream would like it backported, go for it
    - If Pulp wants QE’s help to test a thing, it must have a BZ and follow the process
RPM
- Bugfix releases
- Memory-use in gunicorn investigation continues 2122872 – pulpcore-api gunicorn process consumes high memory
- Pulp_rpm domain work continues
Ansible
Pulp Container
OSTree
Pulp CLI
- 0.19.0 released
Pulp Replica

Katello

Any upgrade wishes for the next few Katello releases?
- core/3.23 has domain work
- core/3.25 will have django4
- katello/4.9 branching in “a month”, core/3.22
- Maybe katello/4.10 will aim for core/3.25(ish)

QE

One BZ pending for z-stream
- 2184122 – wrong metadata if uploaded rpm have different name than name in rpm

ggainey · May 17, 2023, 6:54pm

[missed posting previous minutes]

2023-05-03

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- core/3.24 released
  - Having Some Issues, will update
- core/3.25 is coming #soon (May-ish)
  - Moves to Django4.2, psycopg3
  - katello/4.Next should prob think about being core/3.25 or later
RPM
- Fix for the profiles issue that Ian reported on Slack is ongoing
- 3.20 release today
  - No domains yet, alas
Ansible
Pulp Container
- Plugin leadership change, goes to Lubos!
- Ggainey will add him to invite
OSTree
Pulp CLI
Pulp Replica

Katello

Core-repair API will be in-use for capsules
- Nudge lmjachky about container-issues involving repair
  - This is a core issue https://github.com/pulp/pulpcore/issues/3606
- Katello will need fixes “soon”
  - Added prio-list label on repair related issues
- Thoughts about large/manyrepo/huge-dataset systems - per-repo, or global-repair?
- Discussion needs to happen! Poke lmjachky?

QE

Nothing to yell about
Sat6.13 released today!
- And there was much rejoicing

ggainey · May 17, 2023, 6:55pm

2023-05-17

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- core/3.26 released
- OpenTelemetry basic support merging into pulpcore
  - Will be available for use by perf-test
RPM
- 3.17.18 released
- 3.21.0 release soon
  - Mostly domains, core/3.25 required
Ansible
Pulp Container
- Podman-push-repo-versioning is changing
OSTree
Pulp CLI
Pulp Replica

Katello

2142689 – Capsule sync fails when a CV is published and promoted to two capsule's LCEs at the same time
- Capsule-sync-race-condition?
- Needs more eyes, from both sides (pulp/katello)
2077363 – Fail to sync kickstart repositories with same sub repositories concurrently hotfix requested for Sat 6.11
- 3.17.18 released with this fix
How could 2174912 – [RFE] add ability to skip syncing .treeinfo files be given an automated test?
- It seems it’s not the repo, but the web server that causes the original bug?
- Idea: test .treeinfo file existence
- Dalley: just test sync with / without treeinfo skipped and whether it downloads the .treeinfo. As you say, this just just a way of working around webserver problems, we don’t need to test the webserver.
  - We already have an upstream test for this Added 'treeinfo' to available skip-types at sync time. by ggainey · Pull Request #2913 · pulp/pulp_rpm · GitHub
  - I doubt it’s worth writing a 2nd automated test
Working with OpenStack team on https://issues.redhat.com/browse/RHOSINFRA-4958
- They’re having potential issues with Satellite & container content – may be just a usage issue but just FYI.
- Let’s make sure lmjachky has access to test systems to help

QE

Build Team

core/3.25 and Django4 is coming - heads up!

ggainey · May 31, 2023, 4:01pm

2023-05-31

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
  - OpenTelemetry initial support now baked in
- 3.27 and a lot of X-releases this week
  - Release Day! May 30, 2023: Pulpcore 3.27 and more
- core/3.25+ OCI images now available
- Memory use investigation has made progress. Querying and Serializing the changelogs is pretty much the culprit.
  - Workarounds suggested, issues filed
  - https://github.com/theforeman/puppet-pulpcore/issues/286
  - https://github.com/theforeman/puppet-pulpcore/issues/285
  - Will be/is in latest foreman release
- Potential fix for 2188504 – The "hammer export" command using single thread encryption causes a performance bottleneck. (export-compression and performance)
  - Note for us: if we want to propose API-lvl changes, we can “just open a Jira” for katello and all The Right People will be informed automagically
- Rerun-chunked-import fix for 2173757 – Can't rerun a failed content-import task if it was exported using chunks under review
RPM
Ansible
Pulp Container
- 2.15 release compatible with pulpcore 3.25, django 4.2
OSTree
Pulp CLI
Pulp Replica

Katello

Katello/4.9 release in progress
https://issues.redhat.com/browse/SAT-16055

QE

Sat-next Snap1 is out, no pulp issues yet

ggainey · June 14, 2023, 5:03pm

2023-06-14

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- Import-rerun merged, backported, released (3.18/21/22)
- Release Day happened : Release Day! June 14, 2023: Pulpcore 3.28 and more
- import/export-compression-fix also merged/released
RPM
- deadlock fix released, backported to rpm/3.17-18-19-21-22
  - https://github.com/pulp/pulp_rpm/pull/3002
Ansible
Pulp Container
OSTree
Pulp CLI
Pulp Replica

Katello

Does pulp have native backup/restore?
2151333 – Mirror complete sync policy no longer allowed with ignored content types in the repo.
https://issues.redhat.com/browse/SAT-16330 Repair API - Docker content updates if any?
- Is this https://github.com/pulp/pulpcore/issues/3606 ?
  - Yes, needs to be fixed “soon” (altho mayhap not ASAP)
- See also 2127537 – [RFE] If an artifact gone missing from filesystem for any reason there is no way to easily redownload the same in Capsule 6.12
Discussion RE katello/4.10 and core-release
- Let’s try to get “further up” than core/3.25
- Ian to look over the changelogs to see if core/3.28 would be good

QE

Gunicorn “fix”?
- Cfg-change to auto-restart workers “regularly” (not a pulp-change, a foreman-installer change)

ggainey · June 28, 2023, 5:15pm

2023-06-28

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- 3.28/3.28.1 released
RPM
- 6.14 upgrade blocker
  - 2213515 – Upgrade to 6.14 fails on rpm.0048_artifacts_dependencies_fix migration with TypeError: 'NoneType' object is not subscriptable
  - Looks like data issue, investigation continues
Ansible
Pulp Container
OSTree
Pulp CLI
Pulp Replica

Katello

Sorting out some docker bugs in katello:
- PR out for this: https://github.com/Katello/katello/pull/10629
- 2 open issues: (manifest/tag deletion)
  - Feature #36539: Allow deleting tags and untagging manifests from Katello - Katello - Foreman
  - Bug #36538: Removing manifest referenced by tag doesn't remove it from pulp but deletes it from katello - Katello - Foreman
Some katello API changes coming down the line (QE< go yell at Quinn)
- Stems from removing Olde Pulp2 Stuffe (ggainey cheers wildly)

2159105 – Remove deprecated & not working API endpoints from APIdoc (as an example)

QE

ggainey · July 12, 2023, 10:16pm

2023-07-12

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
RPM
- Backports for 0048 migration fix backported
  - 2213515 – Upgrade to 6.14 fails on rpm.0048_artifacts_dependencies_fix migration with TypeError: 'NoneType' object is not subscriptable
Ansible
Pulp Container
OSTree
Pulp CLI
- Fun with click-8.1.4; pinning to 8.1.3 for now
Pulp Replica
Starting to discuss how to be able to change REST API
- very long term discussion - will link to this group when we get, say, a discourse thread

Katello

2219885 – Satellite using the feature 'Alternate Content Source' is getting errors
- Not quite clear on what is going on here
- Is there some “tuning” we need to/can do here (esp on the capsule side)
  - Eg, don’t currently have access to optimize/no-optimize on acs refreshes
Finishing up Sat-Next testing
Still waiting on full pulp-rpms for core/3.28 and friends
- Some packaging snafu in-progress
Hey - what is going on with Flatpak support?
- Talk to pulp_container gang for discussion-status
katello/satellite smart-proxy-sync discussions
- Would like to insure capsule-N-2 works against satellite-N, always

QE

Sat-Next verifications continue apace
QE adding a bunch of smart-proxy-sync test cases

ggainey · July 26, 2023, 5:06pm

2023-07-26

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- Z-releases (including migration-fixes)
- Gunicorn memory-leak fix released
  - 2122872 – pulpcore-api gunicorn process consumes high memory
  - https://github.com/theforeman/puppet-pulpcore/pull/295 - relaxes the number of restarts once ‘2872 is released
RPM
- Q: what version of pulp_rpm is being looked at for katello/4.10?
  - Impacts what we do nightly-tests on and backports-to
  - Also - file/container/ansible?
  - Discussion ensues
  - See Request for Pulpcore 3.28 builds - Releases - TheForeman for initial request
  - Will prob choose “most current release” of plugins at the 4.10 branch point (Aug 22ish) and core/3.28
Ansible
Pulp Container
- [ipanova] https://github.com/pulp/pulp_container/issues/1329
  - In attempt to fix Repair API, we broke pulp-to-pulp sync aka SatServer→Capsule sync
  - 2223707 – Capsule docker sync fails with 406
OSTree
Pulp CLI
- Pulp-replication commands merged
- Prob should do a cli release “soon”
Pulp Replica
- Back under active support/development w/ dkliban back
BZ process
- See dralley’s ping on internal Slack RE BZs-in-POST that may be further along

Katello

Does ACS refresh have an optimized option like remote syncs? - 2219885 – Satellite using the feature 'Alternate Content Source' is getting errors
- By default, yes
- Katello to do some testing around this BZ
Pulpcore 3.28 repo in progress
- We’ll be testing upgrading to 3.28 with Katello soon
- #soon!! - Pulp r excite

QE

RE ipanova’s pulp_container issue - QE tests are passing without?
- Vsedmik to check test-versions, maybe poke ipanova as/if needed
https://bugzilla.redhat.com/show_bug.cgi?id=2142689
- May not be reproducible any more - vsedmik to do some investigation
- Will prob eventually close? Depends on Sat-Current-Y testing

ggainey · August 9, 2023, 6:10pm

2023-08-09

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- 3.30 released
- A lot of performance-related investigation/fixes going on
RPM
- See above RE performance investigations
- 2219885 – Satellite using the feature 'Alternate Content Source' is getting errors investigation (ACS refresh fun)
Ansible
Pulp Container
- 2.16 released w/ flatpak support
OSTree
Pulp CLI
- 0.21 released w/ pulp-replica support
Pulp Replica
- Some tls-fixes released

Katello

core/3.28 - all katello/4.10 tests run, hit one unreproducible error
- https://github.com/pulp/pulp_rpm/issues/3223
  - Not gonna be hi-prio unless/until we see it again
- Hoping to bump pulp_container to 2.16 next

QE

some help needed to reproduce/test 2125366 – Capsule sync fails with Error: It is not possible to add more than one advisory with the same id to a repository version
- We have an upstream test - let’s get the link for QE to look at
How to track 2122872 and 2229788 (gunicorn memory leak Fun)
- 2122872 originally had a workaround that we were going to move forwards with
  - It was cloned to 6.13.2 as 211957 and pushed, now CLOSED-ERRATA
  - Then we found a better fix, and we want to push that to Next-Z? And Sat-Next
  - We also intend to revert the workaround as it’s no longer needed
    - The PR that does the reverting (i.e. we want this commit): https://github.com/theforeman/puppet-pulpcore/pull/295
- 2229788 was created for the “actual” fix
- TL;DR how should this be tracked
  - Dalley suggestion:
    - Leave 2122872 tracking the “meta-issue” of gunicorn memory consumption, including the new lru_cache fix, and the reversion, for Sat-Next
    - Use 2122872 to track the improved fix and the reversion for Next-Z?
    - In both cases, we just test that the gunicorn memory consumption issue is resolved as expected
- ‘2872 - used to handle the mitigation/restart Fun
- ‘9788 - used for the ‘real’ fix
- ‘1957 - cloned ‘2872 mitigation for 6.13 release
- ‘9788 won’t fix 6.11/6.12 cases (since it’s fixing something that didn’t happen until 6.13)
- What we settled on:
  - Keep 9788 for separately tracking the lru_cache fix (and set flags to clone it for 6.13)
  - Create a new bug to track the relaxed installer defaults (6.13 cloning optional)
    - Created 2230584
    - flags: 6.14.0?, 6.13.z?
  - QE these together with the original bug, so that QE effort is not duplicated
    - (the “actual” fix + relaxation) for each release
  - Email Brad, Justin and Vijay Singh (QE mngrs), Shweta (QE contact for already delivered mitigations on 6.13) and Vlad (QE just for tracking), and Odilon (delivery) so everyone knows what to expect
  - The new BZs should be triaged on Aug 10th triage meeting, Vlad or William will provide the info.

ggainey · September 6, 2023, 3:57pm

2023-09-06

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- core/3.33 released
  - Adds complex-filters - see Changelog — Pulp Project 3.33.0 documentation
- import/export performance investigations happening
RPM
- Progress being made on Oracle + RHEL sync data corruption https://bugzilla.redhat.com/show_bug.cgi?id=2232500
  - Probably no patch for existing releases, but we can fix it going forwards
  - Some questions about how the migration should work
Ansible
Pulp Container
OSTree
Pulp CLI
Pulp Replica
Virtual PulpCon: November 6-10, 2023, 0800-1200 GMT-5 (US EDT)
- CfP ends 13-SEP
- Can we get any talks from Katello?
  - [Ian] we’ll think of something!
  - Maybe fs-exporter from paji?
  - container-push design in katello?
- Add to the CfP page!

Katello

Quba42 and pulp_deb/3.0
- discussion/investigation in progress Right Now
- Trying to see if we can fit into katello/4.10 release
- Stay tuned!
BZ RE “tell users if module-streams have expired”?
- Is this even available? (looks like Not)

QE

Couple BZs still being verified

ggainey · October 18, 2023, 3:59pm

(Catching up on notes - usual poster was unavailable)

2023-10-18

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- Lots of performance-improvements in import/export merged and backported
- See 4545 - we are working to no longer “reassemble” export-files on import
  - Requesting input from katello
  - QE: there may be a test specifically aimed at interrupting the import-process - vsedmik to review
- [lmjachky] Katello: What are we going to do about this: Slow content proxy sync - #26 by lubosmj - Support - TheForeman
  - User reverted https://github.com/pulp/pulpcore/pull/4275 and things got better
  - User is on postgres12, we can’t reproduce but we’re on pgres-13
  - lmjachky/iballou to work together on getting older-pgres test sys up
RPM
Ansible
Pulp Container
OSTree
Pulp CLI
Pulp Replica

Katello

4.10 GA “imminent”, a lot of new core/3.28 users incoming!
4.11 will be branching Real Soon Now
- Once that happens, will be upgrading Pulp again
- ggainey to update support-matrix - 4.11 versions same as 4.10

QE

Reviewed import/export perf-BZs from a test standpoint, looks good
- QE ready to get the releases w/ the fixes

2023-10-04

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- Import/export performance fixes
RPM
- Zstd metadata support moving forwards:
  - Now a BZ open as it is in the wild 2241934 – After upstream repo switched to zst compression, Satellite 6.12.5.1 unable to sync
Ansible
Pulp Container
OSTree
Pulp CLI
Pulp Replica

Katello

Community support threads:
- Slow content proxy sync - #19 by gvde - Support - TheForeman
- Two streams with matching NSVCA - #11 by leziri - Support - TheForeman
  - Likely workaround: republish using “mirror complete”

2023-09-20

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- Potential performance regression? Slow content proxy sync - #7 by gvde - Support - TheForeman
  - Comparative performance tests would be helpful for catching these issues?
- Lots of work on import/export performance
RPM
- Zstd metadata support
Ansible
Pulp Container
OSTree
Pulp CLI
Pulp Replica

Katello

Need backport of https://github.com/pulp/pulp_ansible/pull/1561/ to fix syncing ansible content to smart proxies for Katello 4.10 (GAing next week)
- Chatting with Gerrod about it now, awaiting opinions from Matthias
- A backport to 0.18.0 would be ideal, but we could work with a release if it’s compatible with Pulpcore 3.28.

ggainey · November 1, 2023, 5:58pm

2023-11-01

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- 3.40 released! (requires plugin-updates to be compatible, happening this week)
  - Pulp_file now delivered as part-of pulpcore
  - Pulp_file will start versioning same-as core (ie jump to 3.40)
  - Need to verify/have solid answer that client-api rpm/gems still work
- If katello/4.11 runs into probs w/ core/3.40 will drop back to core/3.39 and friends
- In core/3.45, pulp-certguard will also be delivered as part of pulpcore
RPM
- New Pulp team member joining today - Pedro Pessoa
- Pulp_rpm 3.23.0 released
  - Small update, one unbackportable BZ addressed here
    - 2232500 – Synchronizing Oracle Linux 9 AppStream breaks modularity in CVs with filters
Ansible
Pulp Container
OSTree
Pulp CLI
Pulp Replica
PulpCon is NEXT WEEK!
- PulpCon 2023 Schedule - HackMD

Katello

PulpCon schedule shared to the rest of the team, yay!
Pulpcore upgrade
- Trying to get 3.40 into katello/4.11
- Timeline is VERY SHORT
- Client-api needs to work for N-1
- any surprises will prob abort the effort

QE

Concern about core-update and N/N-1 api access requirement?
- Iballou to experiment pre-snap/pre-QE
- Vsedmik to coord w/iballou to get test-scenarios run automatically
- Dalley proposes to run old-bindings against new-core and give feedback
- Also katello VCR runs

Build Team

If core/3.40 is a Bridge Too Far, should we just choose core/3.39 from the start?

ggainey · November 15, 2023, 4:49pm

2023-11-15

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- finding/fixing some subtle side-effects of merging pulp_file into core
- pulp_file/1.6 released as a “dummy” package
- Release in progress to address a BZ/issue
  - 2249847 – Syncing file:// repos can delete the original files
RPM
- Onboarding a new team member!
- PR open to deprecate pkg/metadata csum-type, merge into one
  - Evaluate whether we really need to support specifying "metadata_checksum_type" and "package_checksum_type" separately · Issue #2480 · pulp/pulp_rpm · GitHub
  - https://github.com/pulp/pulp_rpm/pull/3306
  - Katello has a selector, and so will be affected
  - Currently katello only allows sha256/sha1
  - Not backportable, so will only impact future satellites
  - No real need to continue providing the option for sha1, so the decision is whether to remove the option entirely or replace sha1 option with sha512
    - Personally I feel sha512 support isn’t necessary and we’ve had no customer demand
Ansible
Pulp Container
- [lmjachky] hosting bootable RHEL images in Pulp/Satellite - katello will need to support building images and pushing images into Pulp.
  - pulp-container has the support now (specifically, container-push)
  - Katello: “push” already in-plan for future release
  - Image-building support should be investigated/integrated into katello
OSTree
Pulp CLI
Pulp Replica

Katello

“Goin crazy on core/3.39 pkg-building Fun”
Final fit-and-finish in progress on that effort

QE

“So far so good”
Testing against N-1 setup w/ core/3.39, also “so far so good”

ggainey · December 13, 2023, 5:37pm

2023-12-13

Overview

Katello Schedule

Pulp

Last meeting of the year!
- Is this meeting still useful?
- Is the cadence still useful?
- Does the meeting time still work?
- ggainey to schedule 6 months in 2024, if so
Pulpcore
- Query of all changes from last two weeks
- Div-by-zero during import?
  - Fix is simple but improving tests before merging
  - How urgent is getting this fix out?
    - QE would like a fix “soon”
    - Next snap is next Weds - can we get it done before?
    - Let’s just take the fix for now, get an issue open for the test-work
- Investigation ongoing for 2253381 – Failed incremental CV import shows error: duplicate key value violates unique constraint "rpm_updatecollectionname_name_update_record_id_6ef33bed_uniq"
RPM
- Zstd publishing support added
  - Works for RHEL8.1+/9/10 (nut not RHEL7, no yum support there)
  - Not merged/released into pulp_rpm
  - Not the default, just a new option available
  - Could be exposed by a katello-UI widget “at some point”
- Publishing with checksums weaker than sha256 removed
  - see 13-NOV minutes
  - Will be in pulp_rpm/3.25
  - Katello will need to decide what to do w/ the csum-choice-pulldown at publish button
  - EL5 may not support sha256 - but it has other, larger, issues w/ “modern” katello
  - Mirroring metadata works fine for older/ancient distributions
Ansible
Pulp Container
OSTree
Pulp CLI
Pulp Replica
N-1 problem
- Is the communication between Satellite and its Capsules isolated at all?
- There is a decent amount of separation - maybe we can make changes in just-smart-proxy comms to deal with this

Katello

Pulpcore nightly builds & Katello’s new stricter upgrade plan
- Pulp upgrade was WAY more than anticipated
- In future, will select the pulp-release for a branch at the branch-point, or very close to
- Pretty close to having up-to-date pulpcore “nightly”
- Env updates (ie, python3.next-X) - even more conservative

QE

6.14.next BZs all done
- Pulp cheers!
container sync issue - lubos is on it
- Might be https://github.com/pulp/pulp_container/pull/1133
zstd 2241934 – After upstream repo switched to zst compression, Satellite 6.12.5.1 unable to sync
- Fix needs to be released for 6.13 - already available as backports in appropriate upstream branches
- Vsedmik to bug zstream triage, and/or bbuckingham/odilhao

ggainey · January 24, 2024, 7:40pm

2024–01-10

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
RPM
- Working on Sat-Next BZs as our priority
- Dalley out Tuesday - Friday next week
- 2 weeks before BZ freeze (24-JAN)
Ansible
Pulp Container
OSTree
Pulp CLI
Pulp Replica

Katello

Time to think about Pulp versions for:
- Katello 4.12 (need decision around Feb 1st)
- Katello 4.13 / Satellite 6.16
- Currently at core/3.43, next potential breaking-change release is core/3.55
- Suggestion: Pick whatever is current as of 30-JAN
  - Are there any features that are not already in Pulp that are desired/required for katello-4.12/13?
  - Iballou to investigate
- Def need to understand build-implications (if any)
  - Python-version-issues (3.11) already resolved
    - Modularity-fun ensued, but is resolved

QE

not many updates - Sat-Next BZs under verification - 5 ON_QA / 3 VERIFIED

2024-01-24

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- 3.45 released
  - Certguard now “part of” core, just like pulp_file
- core/3.28 wasn’t getting backports for Nov/Dec - fixed now, core/3.28.20 released
- HTTPS proxy support now complete / merged, release incoming for 3.39
RPM
- Working through hyu’s suggested patch for 2253381
- Pedro investigating whether there is anything to do re: 2252662 – satellite-6.11.5.6-1.el7sat.noarch has complete content-view module filter failure for RHEL 8 between 8.8 > 8.9 minor releases
  - There may not be, in which case Sat-Next has one less issue
- The migration blocking issue turned out to be caused by manual index deletions on both community users
  - - another issue, both remove from Sat-Next
Ansible
Pulp Container
OSTree
Pulp CLI
Pulp Replica
Pulp docs rework
- See "Unified Docs" Plan - #4 by dkliban

Katello

It’s unlikely that we’ll be able to squeeze in a Pulp upgrade for Katello 4.12 since time was lost due to shutdown + 4.11 delays
- Delivery is working on “pulp nightly” and a new Pulp installer method at the same time, which would make building a new version at the same time difficult
- When Katello 4.12 branches, we will start testing Katello nightly with Pulpcore nightly to make it easier to upgrade for Katello 4.13.
  - Pulp cheers wildly
- More info: Release Team Meeting 2024-01-17 - #2 by ekohl - Releases - TheForeman
2219885 – Satellite using the feature 'Alternate Content Source' is getting errors - blocker for Sat-Next?
- Discussion ensues
Using public URLS for testing?
- Sometimes you need the Filthy Public Repo Data to test edge cases (rpm ecosystem esp)
- fixtures.pulpproject.org is a good set of “standard” repos to test with
- QE already uses them for some tests

QE

we are approx in the first third of Sat-Next verification

ggainey · February 15, 2024, 8:16pm

2024-02-07

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- HTTPS proxy support has been verified
  - A new BZ has been filed for Katello - manifest refresh isn’t working 2263159
RPM
- 2253381 investigation continues, now w/ reproducer on customer site
- 2246247 repo-delete can cause a large memory-spike, dalley investigating
- 2207867 2151657 repositories can contain duplicate NEVRA if the file paths are different
Ansible
Pulp Container
OSTree
Pulp CLI
Pulp Replica

Katello

next-Y seems to be moving along well, curr-X has been verified
next-Y+1 features in-progress (repo-repair, container-upload for examples )
- core/3.39 dev-boxes for katello devs - work in progress
Flatpak support being discussed

QE

Build Team

quba42 · February 16, 2024, 12:12pm

Can you expand on this? In particular I would be interested in the versions involved.

ggainey · February 16, 2024, 12:36pm

This PR Taught https-download to trust system certstore. by ggainey · Pull Request #4951 · pulp/pulpcore · GitHub taught Pulp to know about the system-truststore-certs. Prior to that, trying to access a Remote via an HTTPS proxy that used a different CA than the Remote (which is the usual case!) would fail. (A long time back, you couldn’t even try, because aiohttp didn’t support https-proxies at all until…3.8? Maybe?)

This change requires py3.11 - prior to that, one had to depend on basically monkeypatching an internal attribute to get it to work. See fixes #3036 -Setup the SSL Context correctly for https proxies by parthaa · Pull Request #3038 · pulp/pulpcore · GitHub for what that looked like.

ggainey · February 21, 2024, 8:41pm

2024-02-21

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last two weeks
- Update the table of versions
  - No changes for 4.12, 4.13 not set yet
RPM
- 2253381 fixed and released to All The Branches
Ansible
Pulp Container
- Flatpak support should be done/available for katello to start consuming
- question: are “flatpak images” different than OCI images?
  - As a user I want to differentiate between different nature of images · Issue #1437 · pulp/pulp_container · GitHub (introducing a flag for the Content ViewSet to better distinguish between flatpak-based or OCI-based images; easier consumption/filtering for Katello)
OSTree
- domains/rbac support added
Pulp CLI
Pulp Replica

Katello

Work ongoing to integrate pulp repair on Capsules
Ansible import BZ: 2262379 – Library import of an Ansible Collection repo fails
- https://redhat-internal.slack.com/archives/C031713779N/p1706800620522639
Flatpak support discussion, is being discussed
Container push is refined and ready to be worked on for Katello 4.13

QE

sat-Next - 11/13 verified, 2 circled back to DEV
- 2223069 – Selinux denials are reported after following "Chapter 13. Managing Custom File Type Content" chapter step by step
- 2263069 – pulp generated config.repo file lacks GPG public key