Katello/Pulp3 Integration meeting

Minutes
#28

December 08, 2021

Overview

  • Katello Schedule
    • 3.18
      • pulpcore 3.7
      • pulp_rpm 3.11
    • 4.1, 4.2
      • pulpcore 3.14
      • pulp_rpm 3.14
    • 4.3 ~Nov 2021
      • pulpcore 3.16(?)
      • pulp_rpm 3.16
    • 4.4?

Pulp

Katello

QE

  • Next-X snap 1.1 preview/test
  • Next-Y automation failure fixes
  • Watching pulpcon pulp testing and adding feedback

Build Team

  • No report
#29

December 15, 2021

Overview

  • Katello Schedule
    • 3.18
      • pulpcore 3.7
      • pulp_rpm 3.11
    • 4.1, 4.2
      • pulpcore 3.14
      • pulp_rpm 3.14
    • 4.3 ~Nov 2021
      • pulpcore 3.16(?)
      • pulp_rpm 3.16
    • 4.4?

Pulp

Katello

  • Switching to sync policy in review
  • Fix for content-migration (request-url-too-long)
  • Enhancing uploads thru katello-api (ostree)
  • Ostree working really well so far!

QE

  • Sat-next work in progress
  • Looking at cust-case-bugs for automatable cases

Build Team

  • No report

Other Topics

  • No more meetings in 2021
  • Happy New Year!
1 Like
#30

January 05, 2022

Overview

  • Katello Schedule
    • 3.18
      • pulpcore 3.7
      • pulp_rpm 3.11
    • 4.1, 4.2
      • pulpcore 3.14
      • pulp_rpm 3.14
    • 4.3 ~Nov 2021
      • pulpcore 3.16(?)
      • pulp_rpm 3.16
    • 4.4?

Pulp

  • Pulpcore
    • Moving to github #soon - all other pulp-projects already migrated!
      • BZ sync will be affected
    • Working on backport-request/release process
  • RPM
    • Depsolving / Copy work re: ASML customer case expected
    • Memory consumption investigation, multiple different avenues being worked on
    • DistributionTree support
      • 3.11 and 3.14 getting backports
    • 3.17 will be released without ACS support
      • It is being pushed into 3.18 so that 3.17 can remain compatible with pulpcore 3.15
  • Migration
    • DistTree fixes affect 2to3 as well
      • Needs a code-change, expect another 0.11.z to fix
      • “Conflicting” trees - resolve via tweaking timestamps
      • next-prev-z going out end-of-month(ish) - to QE by 12-JAN
  • Ansible
    • No report today
  • Pulp Container
  • OSTree
    • No updates
  • Pulp CLI
    • No updates, still awesome
  • Move to github issues
    • Pulpcore only active project not yet migrated
    • Auto-link between bugzillas and ‘upstream’ not in place yet
    • Patience please while we iron out the process kinks!

Katello

QE

  • Busy with new Satellite: Planning for new test automation and fixing some UI tests due to changes in the new version.
    • Patternfly4 changes, etc
  • next-X feature testing
  • next-X BZ verification
  • current-Y BZ verification
  • prev-Y BZ verification

Build Team

  • No report
#31

January 12, 2022

Overview

  • Katello Schedule
    • 3.18
      • pulpcore 3.7
      • pulp_rpm 3.11
    • 4.1, 4.2
      • pulpcore 3.14
      • pulp_rpm 3.14
    • 4.3 ~Nov 2021
      • pulpcore 3.16(?)
      • pulp_rpm 3.16
    • 4.4?

Pulp

Katello

  • ACS support being started, expect questions from iballou!
  • Support for on_demand for container

QE

  • next-X feature testing
  • next-X bz verification
  • Checking automation and signing off for curr-Y
  • prev-Y bz verification
  • More automation work w/ new UI infrastructure

Build Team

#32

January 19, 2022

Overview

  • Katello Schedule
    • 3.18
      • pulpcore 3.7
      • pulp_rpm 3.11
    • 4.1, 4.2
      • pulpcore 3.14
      • pulp_rpm 3.14
    • 4.3 ~Nov 2021
      • pulpcore 3.16(?)
      • pulp_rpm 3.16
    • 4.4?

Pulp

Katello

  • Pulp 3.17 being built into rpms (may not make katello 4.4)
  • ACS work still ongoing, initial testing looks good

QE

  • prev-Y bz verification
  • next-X bz verification
  • next-X feature testing
  • Fixing automation failures
#33

January 26, 2022

Overview

  • Katello Schedule
    • 3.18
      • pulpcore 3.7
      • pulp_rpm 3.11
    • 4.1, 4.2
      • pulpcore 3.14
      • pulp_rpm 3.14
    • 4.3 ~Nov 2021
      • pulpcore 3.16(?)
      • pulp_rpm 3.16
    • 4.4?

Pulp

Katello

  • ACS support underway (Ian)
  • Docker/podman push support, seeing issues, may need help (Justin)
    • Auth bug (to be filed)
    • client/server comm complaining - will bug ipanova or matrix #pulp-container
  • 3.17 RPMs still being built by packaging

QE

  • prev-Y automation signoff
  • next-X bz verification
  • next-X feature testing
  • Fixing automation failures

Build Team

  • 3.17 rpm-builds in progress
    • done #soon?
#34

February 02, 2022

Overview

  • Katello Schedule
    • 3.18
      • pulpcore 3.7
      • pulp_rpm 3.11
    • 4.1, 4.2
      • pulpcore 3.14
      • pulp_rpm 3.14
    • 4.3 ~Nov 2021
      • pulpcore 3.16(?)
      • pulp_rpm 3.16
    • 4.4?

Pulp

  • Pulpcore
    • Query of PRs merged in the last week
    • Some backports, see query ^
    • More CI fixes as we improve our backporting process
    • One bugfix about cancelled tasks not being picked up
    • In progress, but not yet merged
      • Improved proxy testing
      • Ability to reroot /pulp/api/v3/
      • Issue-creation-template being created to improve issue-process
    • ggainey working on distribution tree import/export
  • RPM
    • RBAC effort will start this month
    • Katello to open an RFE to be able to export metadata (publications)
  • Migration
  • Ansible
    • 0.12.0 releasing today
      • Contains new signing features
    • Improved proxy testing underway
  • Pulp Container
    • Moving our tests away from dockerhub to github registry ( avoid docker pull limits CI is reaching)
    • 2.8.4 released ( manifest sync bugfix)
    • Container signing mvp more or less done, tests to follow
    • Wrapping up adding redis caching on requests
    • Working on adding cross-repository blob mount (docker push)
    • Working on removing InterrelateStage from the sync pipeline
    • Wrapping up transition to Roles (RBAC)
    • In the queue to look at adding support for PulpRemoteUserAuthentication (docker push)
      • Not (currently) urgent for katello
  • OSTree
  • Pulp CLI
    • Ability to work with a Pulp system that is rerooted

Katello

  • Initial ACS work under review (with file)
  • Success with docker push (one bug filed https://github.com/pulp/pulp_container/issues/558)
    • Need to think through workflow
    • Allow pre-creation?
    • On hold for now due to bugfixing
    • AI: ipanova/jsherrill to brainstorm workflows
  • Ryan giving presentation to CEE around ostree tomorrow
  • AI: dklliban/jsherrill to brainstorm upstream use-cases
  • Pulp_rpm ruby-bindings issue: openapi-gen-dep got bumped
    • Need to release a bindings-version w/ a lower faraday-version req
      • Allow 1.x and 0.7.13
    • Will definitely need testing support
    • VCR testsuite will catch problems
    • AI: Pulp to work w/ build-team ASAP

QE

  • next-Z bz verification
  • next-X bz verification
  • next-X feature testing
  • Fixing automation failures

Build Team

#35

February 09, 2022

Overview

  • Katello Schedule
    • 3.18
      • pulpcore 3.7
      • pulp_rpm 3.11
    • 4.1, 4.2
      • pulpcore 3.14
      • pulp_rpm 3.14
    • 4.3, 4.4 ~Nov 2021
      • pulpcore 3.16(?)
      • pulp_rpm 3.16
    • 4.5 ~May 2022

Pulp

Katello

  • Adjusting remote sync timeouts to specify more specific timeouts
    • Curr dflt is too low
  • Meeting on Monday to talk through container push workflow, let me know if interested
  • Adding container exclude tags, defaulting to exclude source containers

QE

  • next-Z bz verification
  • next-X bz verification
  • next-X feature testing
  • Fixing automation failures

Build Team

#36

February 16, 2022

Overview

  • Katello Schedule
    • 4.5 ~May 2022
      • Pulpcore 3.18(?)

Pulp

  • Pulpcore
    • Query of all changes from last week
    • 3.18 to release on Feb 22
      • New mgmt command to view publication contents
      • SQL deadlock fix merged and backported
      • Import/Export bugfix merged (not backported)
      • Variety of RBAC change (not significant to katello)
      • Scheduled tasks likely to be added to 3.18
        • Not user editable, only for plugin writers currently
      • Re-rooting feature available for pulp-content
        • Currently hard-coded to /pulp/api/v3/
        • Will be /api/v3/
        • Q: Can you change it on an existing system?
        • Comment: katello prob won’t care about this
  • RPM
    • Possible 3.17 migration bug, investigating (dalley)
      • 3.17.4 likely to come out this week if so
    • Update on memory consumption, call for feedback
      • Specifically - once changelog-limiting is available - need specifics on current-memory-use looks when under load/real-data scenarios
    • RBAC work in progress for 3.18
  • Migration
  • Ansible
    • We need @mdellweg invited here to report
      • done
    • Signature Uploads in-progress
    • API refactor in-progress
      • Primarily drop-in replacement
      • Will ease multi-repo context
      • Will be a few weeks before available
  • Pulp Container
    • Bunch of backports and releases 2.5, 2.8, 2.9 and 2.10
    • Looking at adding support for PulpRemoteUserAuthentication (docker push)
  • OSTree
  • Pulp CLI

Katello

QE

Build Team

#37

February 23, 2022

Overview

  • Katello Schedule
    • 4.5 ~May 2022
      • Pulpcore 3.18(?)
        • Pulpcore 3.17 first
      • pulp_rpm 3.17

Pulp

  • Pulpcore
  • RPM
    • No changes needed for the migration issue discussed last week, Postgresql versions >= 12 unaffected, upgrading solves the problem
    • Work on BZs continues
  • Migration
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI

Katello

  • rpm/3.17 still “on the board”
    • [iballou] raise question about priority of this for next-X testing (pretty please)
  • prev-Y-related bugs
  • Migration-related Stuff (mostly UX)

QE

  • next-Z bz verification, snap-2 dropped
    • Unexpected failures being investigated
  • next-X bz verification
  • next-X feature testing
  • Fixing automation failures

Build Team

#38

March 02, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last week
    • Releases for 3.18.1, 3.16.4, 3.17.4 (latter have MANY bugfixes)
    • Decreasing memory-use during sync in progress (see bmbouters for details)
    • Support of metalink URLs for remotes starting discussion - get your opinions in now!
  • RPM
    • 3.17.4 released
      • Fixes ruby bindings, one bugfix
    • BZ work continues
    • “Very close” to having “a” sync-memory mitigation ready
      • A few hundred megabytes per sync + improves the time by 60-120 seconds
      • Also pulpcore work in train for same
  • Migration
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI

Katello

QE

  • next-z bz verification, snap-2, most ppl done
  • nexy-Y bz verification snap 11
  • next-Y feature testing
  • Fixing automation failures

Build Team

#39

March 09, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • 3.14.13 released yesterday
    • Working on a few issues reported on foreman forums
    • RBAC work continues, see ppicka for details
  • Migration
  • Ansible
    • ggainey to touch base w/ x9c4 RE adding lines here
  • Pulp Container
    • Support for cross repo blob mount is in
    • Support for upload of foreign layers is in
    • Support for manifest list push is in
    • Preparing for 2.11 release that will have content signing support plus mentioned features ^
    • Remote user auth fix is in, what branch do you want it backported to?
      • No need to backport, will take with core/3.18
    • Fixed a customer bug “ Can’t sync private repos from Azure registry”, what branch do you want it backported to? (2.8?)
  • OSTree
  • Pulp CLI

Katello

  • core/3.18 builds requested
  • “Move to core/3.17” has made it “on the board”
  • rpm/3.17 - in upstream, will be in next product snap
  • Working on client bindings update

QE

  • next-Z snap 1 tomorrow
  • prev-Y bz verification in progress for snap 2
  • next-X bz verification in progress for snap 12
  • next-X feature testing
  • Fixing automation failures
#40

March 16, 2022

Overview

  • Katello Schedule

Pulp

Katello

  • How critical is https://github.com/pulp/pulpcore/issues/2327 ?
    • GA’ing Katello 4.4 would mean forcing people up to Pulpcore 3.16 through the EOL’ing of Katello 4.2
    • It’s Important
  • Permission requirements for /var/lib/pulp/media/* and /var/lib/pulp/tmp/* ?
    • Is 0750 okay?
    • 755 would work as well - owner needs to be “user who’s running pulp services”
      • Or 770, if apache/pulp

QE

  • next-Z snap 2 bz verification in progress
  • next-Y bz verification in progress for snap 13
  • next-Y feature testing
  • Fixing automation failures
#41

March 23, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last week
    • Significant memory reduction bugfix merged, and being backported
      • 27% reduction in memory with EL7 on_demand sync and 5% less runtime
      • Backporting will be happening #soon
    • RemoteArtifact duplicate key error bugfix merged, and being backported
    • Pulpcore now publishes its bindings docs
      • Pulpcore Client
      • Should we publish ruby-bindings-docs as well?
        • “Handy”, more for new folk coming on
  • RPM
  • Migration
    • 0.11.10 released w/ one bugfix (511)
  • Ansible
  • Pulp Container
    • Released 2.11
    • post-release work happening
    • Sync-pipeline refactoring in progress
  • OSTree
  • Pulp CLI
  • Pulp-openapi-generator
    • patch to expose timeouts merged
  • Release cadence - we might pick a “release day” once per week
  • Query: how does katello setup auth-backends?
    • Taken up post-mtg

Katello

QE (In QE all hands, won’t make it in this meeting)

  • prev-Z snap 1 bz verification. Waiting on snap templatization before verification
  • curr-Z snap 3 bz verification in progress
  • nexy-Y bz verification in progress for snap 14
  • next-Y feature testing
  • Fixing automation failures
#42

March 30, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • Memory improvements to be merged released imminently (finally)
      • Combined with some pulpcore improvements that Brian worked on, the biggest syncs stay under 0.9 gb of memory consumption, down from ~3.5 at the release of Sat 6.10
    • Working on other small fixes
    • RBAC work progressing/finishing ip
  • Migration
    • Should consider EOL for 2to3 (core/3.20?)
  • Ansible
  • Pulp Container
  • OSTree
  • Pulp CLI
    • No updates
  • Can we update the pulpcore version being used in Sat-Next-Y?
    • Where/with whom should we discuss this?
    • core/3.17 heading for katello/4.5
    • dkliban/dalley to work w/ iballou on what this implies

Katello

QE

#43

April 06, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
    • Query of all changes from last week
    • Should logs be i8ln?
    • Working to get functional and unit tests to be easy to run
      • Can we get pulp-tests run against a) katello-upstream, and b) downstream?
      • QE and/or build-team should be able to take advantage of this
      • Dkliban to bring up in Team Lead liaison mtg
    • Pulcore 3.19 is pushed back a week April 12, 2022
    • Pulpcore 3.20 will require you to upgrade your plugins to use it. 3.20 contains breaking changes in the plugin API (not the rest API) so plugins need to be upgraded to match. Each plugin will have a new release that is compat w/ pulpcore==3.20
  • RPM
    • Several bugfixes
    • Misparsed createrepo_c metadata
      • Been around for a while, no external reports yet
      • fix/cleanup script incoming
    • release(s) coming next Tues
    • Investigating removing sqlite metadata feature
      • Only used by repoview (which is unmaintained py2 code…)
    • Createrepo_c memory-use changes currently only planned for rpm/3.17
      • Could backport to rpm/3.14 but no current request
  • Migration
    • No updates
  • Ansible
  • Pulp Container
    • 2.8.z and 2.10.z releases are out
      • 2.8.z fixes axure-container-registry fix
      • 2.10.z fixes missing groups/perms
  • OSTree
    • dev-BZ-freeze #soon
    • Needs a GA-release
  • Pulp CLI
    • No updates

Katello

QE

  • prev-Y snap 2 bz verification in progress.
  • next-Y bz verification in progress for snap 15
  • next-Y feature testing
  • curr-Y BZ verification completed!
  • Fixing automation failures
#44

April 13, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • 3.14.15 and 3.17.5 released
      • Data cleanup scripts, changelogs and ampersand-in-xml repair
      • Where/who runs the cleanup script?
      • Note RE changelogs: existing repos will not have their changelogs trimmed
        • When/who would run a cleanup script for existing installs?
    • Dralley to focus on depsolving issues (eg, 2070537)
  • Migration
    • When is (should be?) 2to3 removed?
    • 2to3 EOLs as of core/3.19
  • Ansible
    • 0.13.0, 0.12.1 released
  • Pulp Container
    • Lubos is giving a shot at enabling ‘pull-through cache’ on pull
      • Magical auto-discovery on pulp side
  • OSTree
  • Pulp CLI

Katello

QE

  • prev-Y snap 2 bz verification in progress.
    • A couple BZs left, signoff will be Monday
  • next-Y bz verification in progress for snap 16
  • next-Y feature testing
  • Fixing automation failures
#45

April 20, 2022

Overview

  • Katello Schedule

Pulp

Katello

QE

  • curr-Z bz verification
  • next-Y bz verification in progress for snap 16
  • next-Y feature testing
  • Fixing automation failures
1 Like
#46

[missed posting last week’s meeting, catching up]

April 27, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • Requesting feedback on https://github.com/pulp/pulp_rpm/issues/2479 and whether Katello / product has a desire for something like this
    • RBAC nearly complete, will land in 3.18
      • only works for core/3.19
      • Pulp_rpm 3.17 will remain supported for the foreseeable future, downstream should stick with it for now
  • Migration
    • No updates
  • Ansible
    • No updates
  • Pulp Container
    • No updates
  • OSTree
    • No updates
  • Pulp CLI
    • No updates

Katello

  • Almost fully switched to Pulpcore 3.17 now
    • 3.18 is next
    • Someone else needs 3.18, 3.19 will be “soon”
      • core/3.19 diffs vs 3.18 is “relatively small”
    • 4.5 dev-freeze next-week, branch is 10-May
  • Debian upload fix: Bug #34804: Debian package upload is broken - Katello - Foreman
    • Quba42 is best SME for info
  • Content Filter Overview
    • Pulp team (dalley, ggainey) need to review/comment this week
    • this is rpm-specific

QE

  • curr-Z bz verification (almost done!)
  • next-Y bz verification in progress for snap 18
  • next-Y feature testing
  • Fixing automation failures
    • Looking better and better

Build Team

#47

May 4th, 2022

Overview

  • Katello Schedule

Pulp

  • Pulpcore
  • RPM
    • RBAC merged (will be in rpm/3.18)
    • SLES repo issue
      • SUSE metadata exceeds a DB field-len constraint
    • core/rpm - email to satellite-delivery@ to update zstreams for next 6.10.6 snap
  • Migration
  • Ansible
    • Experimentation w/ import/export happening right now
  • Pulp Container
    • Pulp-container 2.12 is released and is compat with pulpcore 3.19
  • OSTree
  • Pulp CLI
  • django-CVEs for “old” versions of Pulp (core/3.7)
    • Support-timelines for downstream are being extended
    • core/3.7 currently requires 2.2.16(17?)
    • What would it take for katello to upgrade to latest 2.2-LTS release (2.2.28?)
      • pulpcore/3.7 impact is noon-trivial
    • Is this “worth doing” at this point, given that 2.2 is already at EOL (as of April)
    • There will be a discussion/decision conversation happening #soon
    • Bmbouter to make sure iballou gets an invite

Katello

  • Katello will likely stick with 3.18 for Katello 4.5
    • Retaining el7 support, packaging resource constraints are the bottleneck
  • Reminder: If/when are publications/distributions destroyed in Pulp?
    • Publication-metadata isn’t (supposed to be) affected by orphan-cleanup
    • Removing the repo-version that a pub is associated with, removes the publication
  • Content Filter Overview
    • Pulp team (dalley, ggainey) REALLY need to review/comment
  • ACS getting to katello, so we will see user-contact upstream

QE

Build Team