Katello/Pulp3 Integration meeting

ggainey · March 2, 2022, 4:55pm

March 02, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Releases for 3.18.1, 3.16.4, 3.17.4 (latter have MANY bugfixes)
- Decreasing memory-use during sync in progress (see bmbouters for details)
- Support of metalink URLs for remotes starting discussion - get your opinions in now!
RPM
- 3.17.4 released
  - Fixes ruby bindings, one bugfix
- BZ work continues
- “Very close” to having “a” sync-memory mitigation ready
  - A few hundred megabytes per sync + improves the time by 60-120 seconds
  - Also pulpcore work in train for same
Migration
Ansible
Pulp Container
- Finished RBAC roles transition
- PR available for the RemoteUserAuth with token_auth=False https://github.com/pulp/pulp_container/pull/604
OSTree
Pulp CLI
- 0.14 released, see Changelog - Pulp 3 CLI

Katello

upgraded to rpm/3.17 in our core/3.16 repos
- Talk to iballou RE making this happen
Moving to core/3.17 “soon”
Should request builds for core/3.18 “soon” (was released last week)
Wanted to mention: 2033853 – Publish content view failed with "PulpRpmClient::ApiError Error message: the server returns an error" (client side timeout in bindings not working)
import/export feature - meeting request incoming
- https://github.com/Katello/katello/pull/9994
- https://github.com/pulp/pulpcore/issues/2269
Discussion RE who will be “The One” for katello as jsherrill moves responsibilities

QE

next-z bz verification, snap-2, most ppl done
nexy-Y bz verification snap 11
next-Y feature testing
Fixing automation failures

Build Team

ggainey · March 9, 2022, 4:52pm

March 09, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- 3.14.13 released yesterday
- Docs and openapi schema are now human readable
  - https://github.com/pulp/pulpcore/pull/2292 (see pics)
- import/export discussions happening w/ regard to potentially newly-uncovered requirements?
  - paji/ggainey/bbuckingham involved
RPM
- 3.14.13 released yesterday
- Working on a few issues reported on foreman forums
- RBAC work continues, see ppicka for details
Migration
Ansible
- ggainey to touch base w/ x9c4 RE adding lines here
Pulp Container
- Support for cross repo blob mount is in
- Support for upload of foreign layers is in
- Support for manifest list push is in
- Preparing for 2.11 release that will have content signing support plus mentioned features ^
- Remote user auth fix is in, what branch do you want it backported to?
  - No need to backport, will take with core/3.18
- Fixed a customer bug “ Can’t sync private repos from Azure registry”, what branch do you want it backported to? (2.8?)
  - 2026151 – Can't sync private Azure registry to Satellite
OSTree
Pulp CLI

Katello

core/3.18 builds requested
“Move to core/3.17” has made it “on the board”
rpm/3.17 - in upstream, will be in next product snap
Working on client bindings update

QE

next-Z snap 1 tomorrow
prev-Y bz verification in progress for snap 2
next-X bz verification in progress for snap 12
next-X feature testing
Fixing automation failures

ggainey · March 16, 2022, 4:01pm

March 16, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- New bugfix reducing memory 27%, e.g. on EL7 on_demand sync
  - Being backported to 3.14+ #soon
- Remote “encrypted fields” data integrity bug fixed, with a repair script
  - https://github.com/pulp/pulp_rpm/issues/2327
  - Katello 4.3 need to take this patch ASAP and document running the repair script, or have the installer trigger it
  - (not released yet, but soon)
  - If you upgraded from core/3.14 to core/3.16 (or later), and had encrypted fields for remotes, you may have been affected by this
- Adds TASK_DIAGNOSTIC setting that records memory usage over time chart
  - https://github.com/pulp/pulpcore/blob/main/docs/plugins/plugin-writer/concepts/tasks/diagnostics.rst
- Task cleanup added to existing 3.14 migration, fixes significant upgrade issue
  - https://github.com/pulp/pulpcore/issues/2101
  - Will be backported to 3.14+ #soon
RPM
- Grant produced a patch for 2024963 – Syncing EPEL repos on Satellite 6.10 fails with: "Incoming and existing advisories have the same id but different timestamps and non-intersecting package lists.." (EPEL advisory fun)
  - Review in progress
  - Will be backported to 3.14+
- Createrepo_c streaming API PR still making progress - limited by PR review
  - Measurements taken in combination with Brian’s pulpcore PR - RHEL 7 uses less than 900mb to sync https://github.com/pulp/pulpcore/pull/2338#issuecomment-1068695320
  - 1994397 – Increased memory usage of pulp-3 workers during repo sync
- Pulp_rpm RBAC update tests being written, should be submitted #soon
  - Prob won’t affect katello (who uses just-admin)
Migration
Ansible
- API refactor underway for the interaction of the ansible-galaxy CLI
  - Backwards compatible
Pulp Container
- Releasing 2.11 today
  - All the signing work, lots of other Stuff
- When 2.8.z release is needed with this bugfix ?2026151 – Can't sync private Azure registry to Satellite
  - Needs to be in “next Z”
OSTree
Pulp CLI

Katello

How critical is https://github.com/pulp/pulpcore/issues/2327 ?
- GA’ing Katello 4.4 would mean forcing people up to Pulpcore 3.16 through the EOL’ing of Katello 4.2
- It’s Important
Permission requirements for /var/lib/pulp/media/* and /var/lib/pulp/tmp/* ?
- Is 0750 okay?
- 755 would work as well - owner needs to be “user who’s running pulp services”
  - Or 770, if apache/pulp

QE

next-Z snap 2 bz verification in progress
next-Y bz verification in progress for snap 13
next-Y feature testing
Fixing automation failures

ggainey · March 23, 2022, 4:53pm

March 23, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Significant memory reduction bugfix merged, and being backported
  - 27% reduction in memory with EL7 on_demand sync and 5% less runtime
  - Backporting will be happening #soon
- RemoteArtifact duplicate key error bugfix merged, and being backported
- Pulpcore now publishes its bindings docs
  - Pulpcore Client
  - Should we publish ruby-bindings-docs as well?
    - “Handy”, more for new folk coming on
RPM
Migration
- 0.11.10 released w/ one bugfix (511)
Ansible
Pulp Container
- Released 2.11
- post-release work happening
- Sync-pipeline refactoring in progress
OSTree
Pulp CLI
Pulp-openapi-generator
- patch to expose timeouts merged
Release cadence - we might pick a “release day” once per week
Query: how does katello setup auth-backends?
- Taken up post-mtg

Katello

Updating to Pulpcore 3.16.5
Db-encrypted-fields script under test for katello-4.4
Still fighting with the repair of repositories after migrating to Katello 3.18 (now on 4.1.2) - #31 by rbremer - Support - TheForeman
- May need some more help.

QE (In QE all hands, won’t make it in this meeting)

prev-Z snap 1 bz verification. Waiting on snap templatization before verification
curr-Z snap 3 bz verification in progress
nexy-Y bz verification in progress for snap 14
next-Y feature testing
Fixing automation failures

ggainey · March 31, 2022, 6:30pm

March 30, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- releases happened last week
  - core/3.14.15 rpm/3.14.14
  - Bindings updated to expose timeout setting
- There has been a request to prioritize 2035873 – Concurrent CV Publish fails with 502 error
- “Tuesday” is going to become Pulp-Release-Day (we hope)
RPM
- Memory improvements to be merged released imminently (finally)
  - Combined with some pulpcore improvements that Brian worked on, the biggest syncs stay under 0.9 gb of memory consumption, down from ~3.5 at the release of Sat 6.10
- Working on other small fixes
- RBAC work progressing/finishing ip
Migration
- Should consider EOL for 2to3 (core/3.20?)
Ansible
- Galaxy API Refactor Stage 1 merged https://github.com/pulp/pulp_ansible/pull/849
- Signature workflows: https://github.com/pulp/pulp_ansible/pull/867
- Signatures in exports still being worked on: https://github.com/pulp/pulp_ansible/pull/845
Pulp Container
OSTree
Pulp CLI
- No updates
Can we update the pulpcore version being used in Sat-Next-Y?
- Where/with whom should we discuss this?
- core/3.17 heading for katello/4.5
- dkliban/dalley to work w/ iballou on what this implies

Katello

https://github.com/pulp/pulp_rpm/issues/2456 (Add profile information to modulemd endpoint) BZ: 2063999 – No profiles are shown for any module streams
Discuss upgrading to Pulpcore 3.17 in Satellite
- RHUI is going GA with 3.17 - less packaging work, less backporting work, more testing if we consolidate
- 3.16 is from October, will be 8 months old on release day. RHUI will GA with a newer version.

QE

curr-Z GA had a missing package: Bug 2069827 - Subscription & Red Hat Repositories pages are blank after upgrade the minor release
prev-Y snap 1 bz verification in progress.
next-Y bz verification in progress for snap 14
- Pulp bz: 2069822 – Syncing RHEL 5 KS repository fails with: "ModelBase object argument after ** must be a mapping, not str"
next-Y feature testing
Fixing automation failures

ggainey · April 6, 2022, 4:04pm

April 06, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Should logs be i8ln?
  - https://github.com/pulp/pulpcore/issues/2477
- Working to get functional and unit tests to be easy to run
  - Can we get pulp-tests run against a) katello-upstream, and b) downstream?
  - QE and/or build-team should be able to take advantage of this
  - Dkliban to bring up in Team Lead liaison mtg
- Pulcore 3.19 is pushed back a week April 12, 2022
- Pulpcore 3.20 will require you to upgrade your plugins to use it. 3.20 contains breaking changes in the plugin API (not the rest API) so plugins need to be upgraded to match. Each plugin will have a new release that is compat w/ pulpcore==3.20
RPM
- Several bugfixes
- Misparsed createrepo_c metadata
  - Been around for a while, no external reports yet
  - fix/cleanup script incoming
- release(s) coming next Tues
- Investigating removing sqlite metadata feature
  - Only used by repoview (which is unmaintained py2 code…)
- Createrepo_c memory-use changes currently only planned for rpm/3.17
  - Could backport to rpm/3.14 but no current request
Migration
- No updates
Ansible
Pulp Container
- 2.8.z and 2.10.z releases are out
  - 2.8.z fixes axure-container-registry fix
  - 2.10.z fixes missing groups/perms
OSTree
- dev-BZ-freeze #soon
- Needs a GA-release
Pulp CLI
- No updates

Katello

Pulpcore 3.17 integration
- Pulpcore 3.17.6 and pulp-container 2.10.3 being built now
- Katello prepping to upgrade soon – all tests passing
Discuss 2067301 – hammer content-import fails with error Export version 3.14.9 of pulpcore does not match installed version 3.14.12 if the z-stream versions are different for the connected and disconnected satellite 6.10 (https://github.com/pulp/pulpcore/issues/2269)
Discuss https://github.com/pulp/pulp_rpm/issues/2456 (Add profile information to modulemd endpoint) BZ: 2063999 – No profiles are shown for any module streams
2034804 – Document (un)supported combinations of CV filters and dependency solving

QE

prev-Y snap 2 bz verification in progress.
next-Y bz verification in progress for snap 15
next-Y feature testing
curr-Y BZ verification completed!
Fixing automation failures

ggainey · April 13, 2022, 4:00pm

April 13, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Down to 5 next-Y BZs in NEW
  - Query
- 3.19.0 is released
- 3.14.16, 3.16.7 and 3.17.7 released
RPM
- 3.14.15 and 3.17.5 released
  - Data cleanup scripts, changelogs and ampersand-in-xml repair
  - Where/who runs the cleanup script?
  - Note RE changelogs: existing repos will not have their changelogs trimmed
    - When/who would run a cleanup script for existing installs?
- Dralley to focus on depsolving issues (eg, 2070537)
Migration
- When is (should be?) 2to3 removed?
- 2to3 EOLs as of core/3.19
Ansible
- 0.13.0, 0.12.1 released
Pulp Container
- Lubos is giving a shot at enabling ‘pull-through cache’ on pull
  - Magical auto-discovery on pulp side
OSTree
Pulp CLI

Katello

pulp_rpm_client 3.17.5 seems to be causing new test failures
- http://pastebin.test.redhat.com/1044701
- Sjha to file GH issues for findings (https://github.com/pulp/pulp_rpm/issues/2490)
- QE to stay on client rpm/3.17.4 to workaround
Dependency solving information needed:
- 2070537 – Solve Dependencies does not add all needed packages to content view with security only filter
- Might be an expectation problem? Iballou/dalley investigating
Content Filter Overview doc
- Requesting Pulp review/comments/additions
- ggainey/dalley to review
Katello is starting to use Pulpcore 3.17 soon
- pulp_deb_client 2.17.1 needed to make this official
- quba42 investigating a problem

QE

prev-Y snap 2 bz verification in progress.
- A couple BZs left, signoff will be Monday
next-Y bz verification in progress for snap 16
next-Y feature testing
Fixing automation failures

ggainey · April 20, 2022, 3:47pm

April 20, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- 3.20 change coming for better error-reporting (not, alas, backportable)
  - Could be cherry-picked downstream if desired
  - https://github.com/pulp/pulpcore/pull/2271
- Some serious pulp_deb performance enhancements (will be in 3.16+)
  - https://github.com/pulp/pulpcore/pull/2558
- Testing the gunicorn worker process recycling
RPM
- Testing patch involving orphan-content-cleanup errors
  - No reproducer, speculative patch
Migration
- BSON too large issue investigation
Ansible
Pulp Container
OSTree
Pulp CLI

Katello

Sounds like pulp_deb 2.17.1 will be released very soon?
2070537 – Solve Dependencies does not add all needed packages to content view with security only filter
- Just a reminder-nag for pulp_rpm to check this out “soon”

QE

curr-Z bz verification
next-Y bz verification in progress for snap 16
next-Y feature testing
Fixing automation failures

ggainey · May 6, 2022, 2:21pm

[missed posting last week’s meeting, catching up]

April 27, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Added nice bugfix when downloads are retried
  - Backported back to 3.14+
- Bugfix for does_batch improving pulp_deb sync by ~5x
  - Backported to 3.14+
- Working on gunicorn memory Fun
  - 2035873 – Concurrent CV Publish fails with 502 error
RPM
- Requesting feedback on https://github.com/pulp/pulp_rpm/issues/2479 and whether Katello / product has a desire for something like this
- RBAC nearly complete, will land in 3.18
  - only works for core/3.19
  - Pulp_rpm 3.17 will remain supported for the foreseeable future, downstream should stick with it for now
Migration
- No updates
Ansible
- No updates
Pulp Container
- No updates
OSTree
- No updates
Pulp CLI
- No updates

Katello

Almost fully switched to Pulpcore 3.17 now
- 3.18 is next
- Someone else needs 3.18, 3.19 will be “soon”
  - core/3.19 diffs vs 3.18 is “relatively small”
- 4.5 dev-freeze next-week, branch is 10-May
Debian upload fix: Bug #34804: Debian package upload is broken - Katello - Foreman
- Quba42 is best SME for info
Content Filter Overview
- Pulp team (dalley, ggainey) need to review/comment this week
- this is rpm-specific

QE

curr-Z bz verification (almost done!)
next-Y bz verification in progress for snap 18
next-Y feature testing
Fixing automation failures
- Looking better and better

Build Team

ggainey · May 6, 2022, 2:22pm

May 4th, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Branch-releases happening (w/ some hiccups)
- Docs added on enabling debugging
  - https://github.com/pulp/pulpcore/pull/2641
RPM
- RBAC merged (will be in rpm/3.18)
- SLES repo issue
  - SUSE metadata exceeds a DB field-len constraint
- core/rpm - email to satellite-delivery@ to update zstreams for next 6.10.6 snap
Migration
Ansible
- Experimentation w/ import/export happening right now
Pulp Container
- Pulp-container 2.12 is released and is compat with pulpcore 3.19
OSTree
Pulp CLI
django-CVEs for “old” versions of Pulp (core/3.7)
- Support-timelines for downstream are being extended
- core/3.7 currently requires 2.2.16(17?)
- What would it take for katello to upgrade to latest 2.2-LTS release (2.2.28?)
  - pulpcore/3.7 impact is noon-trivial
- Is this “worth doing” at this point, given that 2.2 is already at EOL (as of April)
- There will be a discussion/decision conversation happening #soon
- Bmbouter to make sure iballou gets an invite

Katello

Katello will likely stick with 3.18 for Katello 4.5
- Retaining el7 support, packaging resource constraints are the bottleneck
Reminder: If/when are publications/distributions destroyed in Pulp?
- Publication-metadata isn’t (supposed to be) affected by orphan-cleanup
- Removing the repo-version that a pub is associated with, removes the publication
Content Filter Overview
- Pulp team (dalley, ggainey) REALLY need to review/comment
ACS getting to katello, so we will see user-contact upstream

QE

next-Y bz verification in progress for snap 18 and 18.5
next-Y feature testing
Questions on bug-verifications
- 2033940 – Error: AttributeError: 'NoneType' object has no attribute 'cast' thrown while listing repository versions

Build Team

ggainey · May 11, 2022, 7:47pm

May 11, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Releases Happened 10-May (3.18.5, 3.16.8, 3.14.17)
- Proposal: pulpcore==3.18 be declared the LTS and only release to other versions on a case-by-case basis. Next LTS will be 3.20 or greater.
RPM
- Working w/ iballou/sjha RE depsolving docs
- rpm/3.17 is main backport target
- Features coming in rpm/3.18 (prob be core/3.20+ release)
Migration
- No update
Ansible
- No major changes (mostly for other stakeholders)
Pulp Container
- Pulp-container 2.1 compat with django 2.2.28 is on the way (container/2.1.3)
  - Compat w/ core/3.7
OSTree
- No update
Pulp CLI
- No update
Other topics
- Libdnf5 and product thoughts
  - who should libdnf team talk to?
  - Build? pm?

Katello

Katello 4.5 branched, still needs to be upgraded to Pulpcore 3.18
2027484 – Provide info about AppStream unmanaged content

QE

prev-Z respin Just one upgrade bug, and it is verified
next-Z bz verification when build is dropped (e.t.a. 11th of May)
next-Y bz verification in progress
next-Y feature testing

Build Team

ggainey · June 6, 2022, 8:31pm

[missed posting minutes due to being on PTO (and brain going on PTO a day before I did…) Thanks to the folks who took notes from the meetings I wasn’t around for!]

June 1, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Struggling to release pulpcore
- FYI preliminary telemetry support is in-review
  - Proposal: Telemetry
  - https://github.com/bmbouter/analytics.pulpproject.org
RPM
- Continuing to work on dependency solving issues
- Working on a way to skip over downloading older packages when using retain_package_versions
  - https://github.com/pulp/pulp_rpm/pull/2547
  - Substantial network / disk savings, speedups
- Planning to write many additional tests in the near future
Migration
Ansible
- Fix for ansible bindings merged: https://github.com/pulp/pulp_ansible/pull/994
Pulp Container
OSTree
Pulp CLI

Katello

Still waiting for Foreman packaging for Pulpcore 3.18 bindings

QE

next-Y (snap 22) bz verification
prev-Z is GA tomorrow
Bug 2082209 - Another deadlock issue when syncing repos with high concurrency <= Moved to next release
Automation failure fixes
Automation for feature areas

Build Team

May 25, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- LTS discussions
- 3.20 release date - june 14th ( tentative)
RPM
- Working on depsolving issues
  - Evaluating some ways to improve the design of depsolving in the future
Migration
- Discussions around date for EOL
- Pulp-2to3-migration plugin end-of-life (EOL) date
Ansible
Pulp Container
OSTree
- Might look into adding s3 support
Pulp CLI
- Adding rpm commands

Katello

Upgrade to Pulpcore 3.18 is going, only issue is with the pulp_ansible bindings
- https://github.com/pulp/pulp_ansible/issues/973

QE

prev-Z (snap 3) bz verification
- Still Failed ON_QA: 2082209 – Another deadlock issue when syncing repos with high concurrency
next-Y (snap 21) bz verification.
Automation failure fixes
Automation for feature areas

Build Team

May 18, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
RPM
Migration
- No updates
Ansible
Pulp Container
OSTree
Pulp CLI
- Pulp_rpm content being added

Katello

Regarding some questions from before:
- LibDNF 5 future planning: Katello developers should be the first to talk with the LibDNF team, not managers or PMs. Ian is happy to do this and invites any others as well :-).
- SLES repo issue: this isn’t a big deal for Satellite/Katello because SLES isn’t specifically supported any more.
- Pulpcore 3.18 LTS: we should be all set from a Satellite and Katello standpoint. No further discussion with managers / PMs should be necessary

QE

prev-Z snap 2 bz verification
- Failed ON_QA: 2082209 – Another deadlock issue when syncing repos with high concurrency
next-Y bz verification.
- Last snap before bz freeze is this week I believe
Automation failure fixes
Automation for feature areas
Side note: Lai will not be able to attend anymore till mid-July. Stephen will be taking over.

Build Team

ggainey · June 8, 2022, 4:02pm

June 8, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- 3.20 preparing to come out June 14th
  - Pulpore 3.20 planned for June 14th
  - Will require new plugins which will be forward compat to pulpcore<3.25
  - Telemetry will very likely not be present (still dev-only)
RPM
- Libmodulemd, RHEL7, and EPEL/F35+ discussion
- Libsolv discussion
- Depsolving work ongoing
  - PR submitted
- Retain package versions sync optimization work nearly finished
  - “Don’t download RPMs you’re going to throw away”
- pulp_rpm/3.18 coming “soon”
Migration
Ansible
- No updates
Pulp Container
OSTree
Pulp CLI
- No updates (still awesome)

Katello

Who is the current Satellite correspondent?
- We should probably list that somewhere publicly (ohsnap?)
- Dkliban to add ohsnap PR
What sort of actions would cause contents to be changed under /var/lib/pulp ?
- 1925164 – Online backup restarting tarballs even 12 hours after start of process without logging to stdout/console
- Let’s get a pulp-issue opened so we can actually document all the things that change under /var/lib/pulp

QE

next-Y (snap 23) bz verification
Automation failure fixes
Automation for feature areas
prev-Z first snap expected Monday 13th

ggainey · June 15, 2022, 3:53pm

June 15, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- 3.20 to be released on 21-June
  - Schedule change reason
- 3.16.9 released with a fix for 2096198
RPM
- Depsolving work ongoing
  - PR submitted
  - https://github.com/pulp/pulp_rpm/pull/2553 (?)
- Retain package versions sync optimization work nearly finished
  - “Don’t download RPMs you’re going to throw away”
  - https://github.com/pulp/pulp_rpm/pull/2547
- pulp_rpm/3.18 coming “soon” (for core/3.20 support)
Migration
- EOL discussion still ongoing
Ansible
- galaxy-ng/migration discussions in progress
- Namespace model is moving into pulp-ansible
Pulp Container
OSTree
- What’s the status of ostree support in katello?
- Released in upstream katello
- Will be downstream “soon”
Pulp CLI

Katello

Pulp team meetings in September?
- 12-SEP - 16-SEP in Raleigh
- 18-JUL - 22-JUL in Brno
Thoughts on https://issues.redhat.com/browse/SAT-8244 (Priority on pulp tasks)
- Open an upstream feature to get this on pulpcore’s roadmap ASAP, please!
- [sajha] Opened https://github.com/pulp/pulpcore/issues/2850 in pulpcore.
katello/4.5 on core/3.18 “just about done”

QE

Build Team

ggainey · June 22, 2022, 7:08pm

June 22, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Pulpcore 3.20 released
  - All plugins will need compatibility releases
  - Next pulpcore plugin API breaking change update will be 3.25 so plugins should be compatible with pulpcore>=3.20,<3.25
  - Release pulpcore 3.20 release thread and progress can be followed here
- Working on fails-QE BZs in pulpcore
- Allow chunked uploads to turn into content directly
- Fix ?ordering query. Use that instead of sort going forward
RPM
- 3.17.6 released
- 3.18 probably in the next 2 weeks
- Working with Waldirio on some customer cases / BZs
- Module obsoletes support - will probably need to be backported to 3.17
  - Will eventually show up in RHEL9
Migration
- Discussion around RH IT and their enormous Sat6.9 and migration
Ansible
- Under discussion: Expanding the “Ansible Roles” API to allow for uploads
- Is this useful for katello? discussion/brainstorming in order
  - Currently not supported at all, not sure if it’s “on the roadmap”
Pulp Container
- Prep for 3.20 compat release
OSTree
Pulp CLI

Katello

How was ACS with RHUI tested?
- Q: How does one get access to test when inside AWS?
- Correct certs and correct header to talk to RHUI
Mtg RE Sat-IT-setup, discussion ongoing
- Will work together “if weirdness shows up”
Ansible client consumption of collections?
- AI: sjha to set up call, sjha/bmbouters/newswangerd to connect on What’s Changing
- See #pulp-ansible in Matrix

QE

next-Y (snap 26) bz verification
prev-Z (snap 1) bz verification
Automation failure fixes
Automation for feature areas

ggainey · June 29, 2022, 4:24pm

June 29, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Investigation continues on a pair of fails-qa BZs
  - 2061224 (import streaming)
  - 2082209 (new deadlock problem)
    - Possible solution being tested
- Some discussion on the class of service issue
RPM
- Modularity-obsoletes
  - Supported in rpm/3.18 and later
  - Needed whenever modules decide to implement it (not currently used in RHEL9) - have been told it is possible in RHEL9.1
  - Planned to be a backportable change, but will be downstream-only, need feedback on how to go about shipping this
  - Prob should be brought up to PMs
  - probably/possibly will affect katello-applicability?
  - Dalley to send doc to iballou
    - Changes/Module Obsoletes and EOL - Fedora Project Wiki
- Orphan cleanup failure fix https://github.com/pulp/pulp_rpm/issues/2459
- Focusing on tests and BZs
- rpm/3.18 release coming #soon(ish)
Migration
- EOL is picked 31 Dec 2022
Ansible
- 0.14 release pending (3.20 compatibility)
Pulp Container
- 2.13 release is out compat with pulpcore 3.20
- Looking into this BZ 2102078 – podman run returns Error: unexpected end of JSON input on image pulled from satellite
OSTree
- Adding support for s3 storage
Pulp CLI

Katello

Upload RFE from ATIX meeting: https://github.com/pulp/pulpcore/issues/2911
Python smart proxy sync issue – sounds like pulp_python < 3.7.0 needs to include “Includes” to sync Pulp to Pulp

QE

next-Y (snap 26) bz verification
prev-Z (snap 1) bz verification
Automation failure fixes
Automation for feature areas

ggainey · July 8, 2022, 7:10pm

July 06, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- 3.20 and plugin-compat releases are all out
- Planning to comment and close
  - https://github.com/pulp/pulpcore/issues/2850
  - Discussion around this (and tasking in general) happened
- No progress to report on last week’s BZs, alas
- Note on releasing backports
  - Pulp is working to do backport-releases only on “downstream” requests
  - Should pulp “proactively” backport? Or only do so in response to downstream requirements?
RPM
- 3.17.7 released with some bugfixes including the orphan cleanup error
  - No new 3.14 release yet but I can push one if desired
- Focusing on BZs
- 2070168 – Error 'modulemd-yaml-error-quark' while synchronizing fedora modular repository on Satellite 6.10.
  - How to make sure both 6.10 and 6.11 get libmodulemd-2.12 package update?
  - How to make sure package versions don’t get arbitrarily downgraded in the future?
  - Need to investigate to make sure this hasn’t caused a data integrity issue
Migration
Ansible
- Adding role-import-from-github support
Pulp Container
OSTree
Pulp CLI

Katello

Orphan cleanup issue
- Which pulp_rpm versions: rpm/3.14 and rpm/3.17 will be supported
- Need a BZ for this to get it triaged and assigned to release(s)
Lots of discussion around releases/builds/process-expectations - fun!
- Would be good to have current understanding written out

QE

6.11 GA yesterday
6.10.7 GA due today
next-Y Snap 1 expected today
Waiting for next-Z (no date set for snap 1)
Automation failure fixes
Automation for feature areas

ggainey · July 13, 2022, 4:04pm

July 13, 2022

Overview

Katello Schedule
- Katello 4.5 - core/3.18, rpm/3.17, container/2.10, file/1.10

Pulp

Pulpcore
- Query of all changes from last week
- Meeting scheduled to talk about backport policy tomorrow
  - Thanks to iballou for scheduling!
  - Add ppicka to mtg notice please
- Not much to report for core this week
- Ongoing convos from Discourse
  - Improving conflicting dependencies
  - Preventing removal of a remote if it still has Content associated with it
    - Could affect callers who remove remotes - please read and comment if interested
    - See also should-we-keep-using-fk-with-on-delete-set-null
RPM
- [ppicka] module-obsoletes and EXD?
  - [ipanova] some conversation has happened, ppicka to reach out to dalley
Migration
- Iballou/paji investigating a post-migration issue, very poss. All-katello
- Questions RE hardlink-vs-copy
  - 2106530 – Migration to pulp3 requires additional disk space to migrate the docker blobs ( /var/lib/pulp/content/units/docker_blob ) in Satellite 6.9
Ansible
- Verifying signatures is changing, public keys will now be added to repositories
  - Does katello currently use collection-signatures? Sjha should know
- Lots of work going on in pulp-ansible
Pulp Container
- New z stream releases for 2.9 and 2.10
OSTree
Pulp CLI

Katello

Updated Sat-6.12-version-requirements above
- /me cheers
Discussion RE docker/container and migration Fun
- Docker-tags and schema-v1/v2
- 2081560 – ForeignKeyViolation Error with docker_meta_tags
- How does/did pulp2 store tags?

QE

next-Z snap 1 bz verification
next-Y snap 1 bz verification
next-Y snap 2 is delayed till tomorrow
Automation failure and fixes for prev-Yand pre-next-Y branch
Automation for feature area

ggainey · August 3, 2022, 7:30pm

July 20, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Bugfix of redirect URL rewrite that also was backported
  - https://github.com/pulp/pulpcore/pull/2965
- Someone should summarize the outcome/agreement regarding the backporting strategy between Pulp/Katello/QE
RPM
- Which version of Sat should the ACS go to?
  - Katello reports “very tech preview in katello/4.5, full support katello/4.7”
- Support for module-obsoletes is approved
  - Testing being worked on
  - Prob needs to get into curr-Y - needs PM approval [dalley to nag]
  - Discussion ensues
  - Mirrored metadata would be fine, but publishing own will lose the tag
- BZ work continues
- Can we consider limits/rethinking usecases around on repo-copy?
  - Discussion ensues
- libmodulmd/zabbix question raised by a customer case
  - Ggainey to fwd to dalley
Migration
- BSON size-problem BZ - Hao has potential PR drafted to address
Ansible
- Collection Version global uniqueness switching to sha256
  - https://github.com/pulp/pulp_ansible/pull/1119
  - Repository level will still be (name, namespace, version)
  - Does Katello use Git Remotes and Git Syncing?
    - Ansible does
    - Katello currently “assumes” remote-endpoints are Galaxy
- Adding Collection download log
  - https://github.com/pulp/pulp_ansible/pull/1118
  - Maybe for katello/4.7?
- ggainey to update katello-version-matrix with pulp-ansible versions
Pulp Container
- Adding schema validation for manifests
- Adding import/export for ‘push-repo-type’
- https://github.com/pulp/pulp_container/issues/883
  - How critical is this?
OSTree
Pulp CLI
- New release #soon (3.20 compat) - 0.15

Katello

Backporting-discussion report
- Pulp Backporting Strategy for Satellite
  - “Outcomes” section is at the bottom
https://github.com/pulp/pulp_rpm/issues/2456 - Module profiles and Module Profile Rpms. Safe to delete from Katello? Follow-up on 2063999 – No profiles are shown for any module streams
Was there a recent orphan-cleanup bugfix?
- Yes, but sounds like not the one katello is asking about

QE

curr-Z snap 2 bz verification
next-Y snap 2 bz verification
- NeedInfo for katello: 2106091 – Exclude filter may exclude errata and packages that are needed
- 1892218 – Multi-page listing when adding repositories to Content Views confuses the number of repositories to add
Automation failure and fixes for curr-Y pre-next-Y branch
Automation for feature area

July 27, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Merged exporter PR from Partha. Will backport to 3.16.
RPM
- Pavel finished the module obsoletes work
- Working on BZs
Migration
Ansible
Pulp Container
OSTree
Pulp CLI

Katello

Display total size of a yum\docker\iso\file\custom repository in Satellite GUI after syncing them - https://issues.redhat.com/browse/SAT-2512
Processing metadata task should fail with an appropriate error message, whenever a task is stuck due to insufficient file system space. - https://issues.redhat.com/browse/SAT-2354
Can Pulp be configured to log outgoing requests for content downloads, for example, RPMs or metadata files? We’re trying to figure out how to verify that Alternate Content Sources are working.
- No code currently, could possibly be added and backported
- Alternatively: a man-in-the-middle proxy or firewall could be configured to log where requests are going
Should Katello use Pulpcore 3.20 or wait for 3.21?
- Katello 4.7 stabilizes in ~ 3 months (November-ish). We estimate we need 1 month to upgrade Pulp versions (generous estimate). The delivery team needs about 2 weeks.
- We can wait for Pulpcore 3.21, it should release around September
ACS issue: https://github.com/pulp/pulp_rpm/issues/2667
- Figured it out, details are on the issue.

QE

curr-Z snap 1 testing
next-Y snap 3 testing
Feature testing
Automation failure fixes
Waiting on static ouia-id for get into a build

August 3, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Working on core/3.18.6
  - Fixing CI issues
  - Hoping for this afternon/tomorrow
  - 3 BZs (incl FSExporter fix)
RPM
- Multiple high prio BZs in flight
  - 2103522 – Capsule sync fails with "Parsing interrupted: The repository metadata being synced into Pulp is erroneous in a way that makes it ambiguous (duplicate NEVRAs).."
    - In the comments, there is mention of another separate issue, trying to get a new BZ for that
- New releases - 3.14.17, 3.17.8
  - Don’t actually use these, need to release a small new patch
  - Another set of releases will come shortly afterwards (rpm/3.14.18, rpm/3.17.9)
  - From katello: make sure to set fixed-in correctly in BZs to get these picked up
- Libmodulemd data repair
  - Fallout from having shipped wrong libmodulemd version
  - Repair-script will need to be called from katello utility (foreman-maintain? Rake task?)
  - Get katello example affected module/RPM
    - 2105107 – Minor data issue for users on RHEL7 syncing EL8+ EPEL or Fedora Modular repositories
Migration
- One PR being reviewed
- Evaluating open 2to3 BZs for investigation/fixing
Ansible
Pulp Container
- New releases 2.9.6 and 2.10.5
OSTree
Pulp CLI

Katello

Quick chat on 1993917 – Syncing repositories with https proxy set ends with warning Katello::Errors::Pulp3 Error Only http proxies are supported
- 2 cacerts are needed when talking to CDN via https proxy, one for proxy and one for cdn
  - The proxy cert is typically in the systems default CA trust store
  - Sounds like pulp2 requests library trusted it in 6.9 but pulp3 doesn’t seem to want to load it from the trust store and instead wants to be explicit on ca certificates to use.
  - If we have to stay explicit then it’ll be a lot of work from a katello perspective.
  - Let’s see if we can come up with approaches.
  - Paji doing a POC on the pulp-side, will report on progress

QE

curr-Z snap 1 build is scheduled for Friday
next-Y snap 4 testing
Feature testing
Automation failure fixes
Waiting on static ouia-id for get into a build

ggainey · August 10, 2022, 4:36pm

August 10, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Paji’s HTTPS-Proxy PR under intense discussion
  - Pulp will reach out to paji to schedule
- 3.18.6 and 3.16.12 released and already picked up by bldteam
RPM
- New releases out (3.14.18, 3.17.9)
  - Include dup-nevra BZs that are def affecting customers w/ smart-proxies
- Continuing to bang on BZs
- Working with EXD to reduce the size of their metadata
- BZ filed due to CDN not updating “updated_date” on some republished advisories
  - This breaks several workflows
  - There are Unhappy Customers as a result
Migration
- BSON size fix that has been merged into 0.11 and 0.12
Ansible
Pulp Container
OSTree
Pulp CLI

Katello

Creating Discourse thread for Pulpcore backporting strategy to loop-in the community
- Pulp Backport Release Strategy
Katello and pulp should talk about the simultaneous in-person NA meetups
Chatting w/ Odilon RE katello-4.7/AH syncing up on core/3.21

QE

curr-Z snap 1 bz verification
- 4 pulp BZs showed up unexpectedly, being verified
- Sign-off needs to happen by 26-AUG
next-Y snap 5 testing
- Snap 6 on schedule for today (from email)
Feature testing
Automation failure fixes
Waiting on static ouia-id for get into a build