Katello/Pulp3 Integration meeting

[ggainey]

July 06, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • 3.20 and plugin-compat releases are all out
  • Planning to comment and close
    • https://github.com/pulp/pulpcore/issues/2850
    • Discussion around this (and tasking in general) happened
  • No progress to report on last week’s BZs, alas
  • Note on releasing backports
    • Pulp is working to do backport-releases only on “downstream” requests
    • Should pulp “proactively” backport? Or only do so in response to downstream requirements?
• RPM
  • 3.17.7 released with some bugfixes including the orphan cleanup error
    • No new 3.14 release yet but I can push one if desired
  • Focusing on BZs
    • Mostly 2102331 – [RFE] Sometimes the metadata and PG database information about # of packages are not matching. Satellite should alert about it, it’s not clear yet what the problem is
    • Also 2103522 – Capsule sync fails with "Parsing interrupted: The repository metadata being synced into Pulp is erroneous in a way that makes it ambiguous (duplicate NEVRAs).."
    • Also 2070611 – [Errno 24] too many open files after some time syncing.
  • 2070168 – Error 'modulemd-yaml-error-quark' while synchronizing fedora modular repository on Satellite 6.10.
    • How to make sure both 6.10 and 6.11 get libmodulemd-2.12 package update?
    • How to make sure package versions don’t get arbitrarily downgraded in the future?
    • Need to investigate to make sure this hasn’t caused a data integrity issue
• Migration
• Ansible
  • Adding role-import-from-github support
• Pulp Container
• OSTree
• Pulp CLI

Katello

• Orphan cleanup issue
  • Which pulp_rpm versions: rpm/3.14 and rpm/3.17 will be supported
  • Need a BZ for this to get it triaged and assigned to release(s)
• Lots of discussion around releases/builds/process-expectations - fun!
  • Would be good to have current understanding written out

QE

• 6.11 GA yesterday
• 6.10.7 GA due today
• next-Y Snap 1 expected today
• Waiting for next-Z (no date set for snap 1)
• Automation failure fixes
• Automation for feature areas

[ggainey]

July 13, 2022

Overview

• Katello Schedule
  • Katello 4.5 - core/3.18, rpm/3.17, container/2.10, file/1.10

Pulp

• Pulpcore
  • Query of all changes from last week
  • Meeting scheduled to talk about backport policy tomorrow
    • Thanks to iballou for scheduling!
    • Add ppicka to mtg notice please
  • Not much to report for core this week
  • Ongoing convos from Discourse
    • Improving conflicting dependencies
    • Preventing removal of a remote if it still has Content associated with it
      • Could affect callers who remove remotes - please read and comment if interested
      • See also should-we-keep-using-fk-with-on-delete-set-null
• RPM
  • [ppicka] module-obsoletes and EXD?
    • [ipanova] some conversation has happened, ppicka to reach out to dalley
• Migration
  • Iballou/paji investigating a post-migration issue, very poss. All-katello
  • Questions RE hardlink-vs-copy
    • 2106530 – Migration to pulp3 requires additional disk space to migrate the docker blobs ( /var/lib/pulp/content/units/docker_blob ) in Satellite 6.9
• Ansible
  • Verifying signatures is changing, public keys will now be added to repositories
    • Does katello currently use collection-signatures? Sjha should know
  • Lots of work going on in pulp-ansible
• Pulp Container
  • New z stream releases for 2.9 and 2.10
• OSTree
• Pulp CLI

Katello

• Updated Sat-6.12-version-requirements above
  • /me cheers
• Discussion RE docker/container and migration Fun
  • Docker-tags and schema-v1/v2
  • 2081560 – ForeignKeyViolation Error with docker_meta_tags
  • How does/did pulp2 store tags?

QE

• next-Z snap 1 bz verification
• next-Y snap 1 bz verification
• next-Y snap 2 is delayed till tomorrow
• Automation failure and fixes for prev-Yand pre-next-Y branch
• Automation for feature area

[ggainey]

July 20, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • Bugfix of redirect URL rewrite that also was backported
    • https://github.com/pulp/pulpcore/pull/2965
  • Someone should summarize the outcome/agreement regarding the backporting strategy between Pulp/Katello/QE
• RPM
  • Which version of Sat should the ACS go to?
    • Katello reports “very tech preview in katello/4.5, full support katello/4.7”
  • Support for module-obsoletes is approved
    • Testing being worked on
    • Prob needs to get into curr-Y - needs PM approval [dalley to nag]
    • Discussion ensues
    • Mirrored metadata would be fine, but publishing own will lose the tag
  • BZ work continues
  • Can we consider limits/rethinking usecases around on repo-copy?
    • Discussion ensues
  • libmodulmd/zabbix question raised by a customer case
    • Ggainey to fwd to dalley
• Migration
  • BSON size-problem BZ - Hao has potential PR drafted to address
• Ansible
  • Collection Version global uniqueness switching to sha256
    • https://github.com/pulp/pulp_ansible/pull/1119
    • Repository level will still be (name, namespace, version)
    • Does Katello use Git Remotes and Git Syncing?
      • Ansible does
      • Katello currently “assumes” remote-endpoints are Galaxy
  • Adding Collection download log
    • https://github.com/pulp/pulp_ansible/pull/1118
    • Maybe for katello/4.7?
  • ggainey to update katello-version-matrix with pulp-ansible versions
• Pulp Container
  • Adding schema validation for manifests
  • Adding import/export for ‘push-repo-type’
  • https://github.com/pulp/pulp_container/issues/883
    • How critical is this?
• OSTree
• Pulp CLI
  • New release #soon (3.20 compat) - 0.15

Katello

• Backporting-discussion report
  • Pulp Backporting Strategy for Satellite
    • “Outcomes” section is at the bottom
• https://github.com/pulp/pulp_rpm/issues/2456 - Module profiles and Module Profile Rpms. Safe to delete from Katello? Follow-up on 2063999 – No profiles are shown for any module streams
• Was there a recent orphan-cleanup bugfix?
  • Yes, but sounds like not the one katello is asking about

QE

• curr-Z snap 2 bz verification
• next-Y snap 2 bz verification
  • NeedInfo for katello: 2106091 – Exclude filter may exclude errata and packages that are needed
  • 1892218 – Multi-page listing when adding repositories to Content Views confuses the number of repositories to add
• Automation failure and fixes for curr-Y pre-next-Y branch
• Automation for feature area

July 27, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • Merged exporter PR from Partha. Will backport to 3.16.
• RPM
  • Pavel finished the module obsoletes work
  • Working on BZs
    • 2103522 – Capsule sync fails with "Parsing interrupted: The repository metadata being synced into Pulp is erroneous in a way that makes it ambiguous (duplicate NEVRAs).."
    • 2111159 – Refreshing Alternate Content Source complains about invalid remote URL
    • Sync errors on all syncs including the initial sync between new katello server and content proxy - #7 by gvde - Support - TheForeman
    • Some CI problems currently due to modulemd work
• Migration
• Ansible
• Pulp Container
• OSTree
• Pulp CLI

Katello

• Display total size of a yum\docker\iso\file\custom repository in Satellite GUI after syncing them - https://issues.redhat.com/browse/SAT-2512
• Processing metadata task should fail with an appropriate error message, whenever a task is stuck due to insufficient file system space. - https://issues.redhat.com/browse/SAT-2354
• Can Pulp be configured to log outgoing requests for content downloads, for example, RPMs or metadata files? We’re trying to figure out how to verify that Alternate Content Sources are working.
  • No code currently, could possibly be added and backported
  • Alternatively: a man-in-the-middle proxy or firewall could be configured to log where requests are going
• Should Katello use Pulpcore 3.20 or wait for 3.21?
  • Katello 4.7 stabilizes in ~ 3 months (November-ish). We estimate we need 1 month to upgrade Pulp versions (generous estimate). The delivery team needs about 2 weeks.
  • We can wait for Pulpcore 3.21, it should release around September
• ACS issue: https://github.com/pulp/pulp_rpm/issues/2667
  • Figured it out, details are on the issue.

QE

• curr-Z snap 1 testing
• next-Y snap 3 testing
• Feature testing
• Automation failure fixes
• Waiting on static ouia-id for get into a build

August 3, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • Working on core/3.18.6
    • Fixing CI issues
    • Hoping for this afternon/tomorrow
    • 3 BZs (incl FSExporter fix)
• RPM
  • Multiple high prio BZs in flight
    • 2103522 – Capsule sync fails with "Parsing interrupted: The repository metadata being synced into Pulp is erroneous in a way that makes it ambiguous (duplicate NEVRAs).."
      • In the comments, there is mention of another separate issue, trying to get a new BZ for that
  • New releases - 3.14.17, 3.17.8
    • Don’t actually use these, need to release a small new patch
    • Another set of releases will come shortly afterwards (rpm/3.14.18, rpm/3.17.9)
    • From katello: make sure to set fixed-in correctly in BZs to get these picked up
  • Libmodulemd data repair
    • Fallout from having shipped wrong libmodulemd version
    • Repair-script will need to be called from katello utility (foreman-maintain? Rake task?)
    • Get katello example affected module/RPM
      • 2105107 – Minor data issue for users on RHEL7 syncing EL8+ EPEL or Fedora Modular repositories
• Migration
  • One PR being reviewed
  • Evaluating open 2to3 BZs for investigation/fixing
• Ansible
• Pulp Container
  • New releases 2.9.6 and 2.10.5
• OSTree
• Pulp CLI

Katello

• Quick chat on 1993917 – Syncing repositories with https proxy set ends with warning Katello::Errors::Pulp3 Error Only http proxies are supported
  • 2 cacerts are needed when talking to CDN via https proxy, one for proxy and one for cdn
    • The proxy cert is typically in the systems default CA trust store
    • Sounds like pulp2 requests library trusted it in 6.9 but pulp3 doesn’t seem to want to load it from the trust store and instead wants to be explicit on ca certificates to use.
    • If we have to stay explicit then it’ll be a lot of work from a katello perspective.
    • Let’s see if we can come up with approaches.
    • Paji doing a POC on the pulp-side, will report on progress

QE

• curr-Z snap 1 build is scheduled for Friday
• next-Y snap 4 testing
• Feature testing
• Automation failure fixes
• Waiting on static ouia-id for get into a build

[ggainey]

August 10, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • Paji’s HTTPS-Proxy PR under intense discussion
    • Pulp will reach out to paji to schedule
  • 3.18.6 and 3.16.12 released and already picked up by bldteam
• RPM
  • New releases out (3.14.18, 3.17.9)
    • Include dup-nevra BZs that are def affecting customers w/ smart-proxies
  • Continuing to bang on BZs
  • Working with EXD to reduce the size of their metadata
  • BZ filed due to CDN not updating “updated_date” on some republished advisories
    • This breaks several workflows
    • There are Unhappy Customers as a result
• Migration
  • BSON size fix that has been merged into 0.11 and 0.12
• Ansible
• Pulp Container
• OSTree
• Pulp CLI

Katello

• Creating Discourse thread for Pulpcore backporting strategy to loop-in the community
  • Pulp Backport Release Strategy
• Katello and pulp should talk about the simultaneous in-person NA meetups
• Chatting w/ Odilon RE katello-4.7/AH syncing up on core/3.21

QE

• curr-Z snap 1 bz verification
  • 4 pulp BZs showed up unexpectedly, being verified
  • Sign-off needs to happen by 26-AUG
• next-Y snap 5 testing
  • Snap 6 on schedule for today (from email)
• Feature testing
• Automation failure fixes
• Waiting on static ouia-id for get into a build

[ggainey]

August 17, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • 2 core/3.18 releases to update ruby-clients for faraday
    • Dkliban working w/ ewoud/zhunting
    • All the plugins also getting releases to match (in order to make installation work)
    • Hoping to get this fun resolved today
  • Need to meet w/ partha RE https proxy
    • AI: paji to schedule
  • Discussion w/ Galaxy RE “shared remote configurations”
    • See https://github.com/pulp/pulpcore/issues/2824
• RPM
  • Bug fixed for syncing SUSE repos (SLES)
  • Continuing to work on BZs, Foreman discourse reports
  • Ppicka working on missing-metadata w/ modules (reported by users)
    • Profiles, description
  • New releases out for 3.14 and 3.17
  • dalley taking over as Satellite Liason
  • fao89 joining the RPM miniteam!
• Migration
  • Hao’s PR to address 2to3 query
• Ansible
• Pulp Container
  • ipanova working on 2115970 – Sync container images of existing docker type repositories fail with 404 - Not found
• OSTree
• Pulp CLI

Katello

• What’s up with Faraday? (NameError: uninitialized constant Faraday::FilePart → 0.17.5)
  • Faraday needs to be 0.16 (or 1.0.1, <2.0) to have the feature needed
  • Should prob be able to install/pin faraday >=1.0 <2.0, on dev/test boxes
• https-proxy update
  • Foreman fix exists that katello can take advantage of (we think)
  • Paji to finish investigation before calling mtg w/ pulp-team
• Iballou demoing at devconf - hope to see new user feedback for us!
• Paji demoing fs-exporter work to Support team on 25-AUG

QE

• prev-Z snap 1 bz verification
  • Sign-off needs to happen by 26-AUG
  • Still a half-dozen or so left to verify
• next-Y snap 6.1 testing
• Helping Grant https://bugzilla.redhat.com/show_bug.cgi?id=2062526
  • Deadlocks, ew
• Feature testing
• Automation failure fixes
• Waiting on static ouia-id to get into a build
  • “This katello sprint” - or at least #soon

[ggainey]

August 24, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • Katello: what things must be in core for 4.7?
    • Fs-exporter gaps (e.g., incrementals) not planned for 4.7
    • Is there a date in mind for “latest a core-release can be out for katello to use in 4.7”?
    • AI: paji to review with katello team and get back to us
• RPM
  • Working with Samir on a hotfix / customer case
    • Progress, hotfix may happen #soon
  • Ina has made progress on Sync errors on all syncs including the initial sync between new katello server and content proxy - Support - TheForeman
  • Noticed 2107701 – [Pulp 3] If a modulemd metadata artifact is missing from the filesystem but has an artifact_id associated with it in database, "Verify Content Checksum" cannot fix this problem, will likely prioritize (can’t fix missing modulemd w/ verify-checksums)
• Migration
  • “BSON too large” issues being addressed
• Ansible
  • CollectionVersion upload is getting refactored for better consistency
    • https://github.com/pulp/pulp_ansible/pull/1176/
    • Upload fields changes: removed: name, namespace, version; added: artifact, upload, expected_[sha256, name, namespace, version]
    • AI: paji/sjha to catch up w/ katello-ansible-gang to make sure they’re aware
  • Pulp-ansible 0.13.4 was released
• Pulp Container
  • Ipanova is getting out of the door releases with bugfixes starting with 2.8+
  • There is a new feature release 2.14 compat with 3.20
• OSTree
• Pulp CLI

Katello

• https proxy discussion continues

QE

• curr-Z snap 2 bz verification
  • Sign-off this Friday, 26-AUG
• next-Y snap 7 testing
• Helping Grant 2062526 – Another deadlock issue when syncing repos with high concurrency
  • Fix is under-review and will be backported
• Feature testing
• Automation failure fixes
• Waiting on static ouia-id to get into a build
  • CV OUIA-ID PR already merged, waiting for it to be in a build
  • Pending - Host details page OUIA-ID PR
• Question: how long is “too long” to publish a CV where we have added 1 erratum and incrementally updating (ie, we’re doing depsolving)

[ggainey]

August 31, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • core/3.21 release still scheduled for 8-SEP
• RPM
  • Need to release 3.18 in the 1.5 weeks (hopefully for real this time)
  • Should include the modulemd profile & description data 2109634 – Add module profile information to modulemd enpoints
    • Snafu: it looks like pulp_rpm 3.18 claims to require pulpcore 3.19, that is a problem if we want this fix to go into Satellite-next, which is where the BZ is currently pointing
      • 3.19 is a very small release with no breaking changes and only 6 small quality of life features, is a bump theoretically still possible?
        • https://github.com/pulp/pulpcore/blob/3.19/CHANGES.rst#rest-api-1
        • If not, we need to investigate if there is something we can do to make pulp_rpm 3.18 compatible
        • [Ian] I would argue that that bug is not important enough to bump y-versions of Pulpcore. It isn’t a breaking issue, just missing information from the UI. Even updating the y-version of pulp_rpm is a bit much. To help decide, we’d need to see a changelog diff for either upgrade.
    • Will try to prioritize https://github.com/pulp/pulp_rpm/issues/2643 and https://github.com/pulp/pulp_rpm/issues/2735
• Migration
• Ansible
• Pulp Container
• OSTree
• Pulp CLI

Katello

• Satellite-next capsule sync issue with ansible collections: https://github.com/pulp/pulp_ansible/issues/1201 , 2121583 – Sync of an Ansible collection repo to the Capsule fails
• Dep solving fun (Pulp 3 this time!) 2070537 – Solve Dependencies does not add all needed packages to content view with security only filter
  • (dalley) apologies for the lack of response on this so far. I was meaning to dig into it 6 weeks ago but it got overtaken by other items e.g. duplicate NEVRA

QE

• next-Z snap 1 bz verification
  • Waiting on snap
• next-Y snap 7 testing
  • Waiting on snap 8
• Testathon is happening at the moment
• Feature testing
• Automation failure fixes
• Waiting on static ouia-id to get into a build
  • CV OUIA-ID PR already merged, waiting for it to be in a build
  • Pending - Host details page OUIA-ID PR
• Question: how long is “too long” to publish a CV where we have added 1 erratum and incrementally updating (ie, we’re doing depsolving)
  • Answer: 10-15m for rhel8_bos + rhel8_app for both incremental update and publishing cv

[ggainey]

September 07, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • 3.21.0 will release tomorrow
  • “The” deadlock fix backported for curr-Y Satellite
• RPM
  • 3.18 blockers in the process of being merged
    • Still minor cleanup to finish up pre-release
  • Considering some changes that would allow pulp_rpm 3.18 to be used alongside pulpcore 3.18
    • Currently, as mentioned last week, it contains a feature that relies on pulpcore 3.19
  • Coordination going on with certain BZs / customer cases, hotfixes
• Migration
• Ansible
• Pulp Container
• OSTree
• Pulp CLI

Katello

• Testathon in-progress, so changes are on-hold for two weeks
• Waiting patiently for core/3.21
• RDU Tean meetups next week - should coord w/ pulp RDU meetup

QE

• next-Z snap 1 bz verification underway
• next-Y snap 9 bz verification
• Feature testing
• Automation failure fixes
• Waiting on static ouia-id to get into a build
  • Some missing OUIA-ID for cv, sent BZ back to dev
  • Pending - Host details page OUIA-ID PR
• Question: how long is “too long” to publish a CV where we have added 1 erratum and incrementally updating (ie, we’re doing depsolving)
  • Rhel8_baseos + Rhel8_appstream
    • Created a cv with a filter of before June 30th
    • Publish and did incremental update
      • Incremental update time: 10min 15 sec
      • Publishing cv after inc update: 10min 15 sec
    • Created a filter that includes up to July 31st
    • Publish and did incremental update
      • Incremental update time: 10min 15 sec
      • Publishing cv after inc update: 10min 15 sec
  • rhel7
    • Created a cv with a filter of before June 30th
    • Publish and did incremental update
      • Incremental update time: 9min 9 sec
      • Publishing cv after inc update: 13min 23 sec
    • Created a filter that includes up to July 31st
    • Publish and did incremental update
      • Incremental update time: 8min 49 sec
      • Publishing cv after inc update: 18min 35 sec
      • [Iballou] It would be interesting to see the breakdown between “pulp time” and “katello time” for this
      • Katello-indexing should be much faster now, would be good to know if that isn’t the case
  • Note: depSolving is not selected for these runs, but iirc, doing filtering and publishing does the depsolving?
    • In general usage, it does not - katello only asks Pulp to depsolve, if the user asks for it explicitly
    • Incrementals depsolve by default, so they do

[ggainey]

September 21, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • 3.21.0 is out
• RPM
  • 3.18.2 is out, I had a preliminary agreement from Sat that they would be OK with having this in 6.12 - 3.18 is just a generally a small bugfix release that includes some migrations
  • 3 still aligned to sat-next that aren’t complete, 1 in danger of not making the cutoff
• Migration
  • New release 0.17 , that doesn’t affect katello (plugin-eol-dates)
• Ansible
  • 0.15 releasing today
    • Requires core/3.21
    • Bugfixes and 3 new features
• Pulp Container
  • Pulp-container 2.9 and 2.10 is out with bugfixes for sat-next
• OSTree
• Pulp CLI
  • Filling out “tasks” support, to help ppl doing problem-determination
    • Add filter parameters to task list
    • Support base-object field limits and ordering

Katello

• core/3.21 RPMs “almost done”
  • This will be for katello/4.7
• Question: what needs to happen for https://github.com/pulp/pulpcore/pull/3188?
  • Multiple issues, needs multiple CHANGES/ entries
  • Discussion ensued around multiple-commits or not

QE

• next-Z snap 2 bz verification underway
• next-Y snap 11 bz verification
  • BZ freeze this week
• Feature testing
• Automation failure fixes
• Waiting on static ouia-id to get into a build
  • Pending Host details page OUIA-ID into a build
  • 2060099 – [RFE] ouia-ID for tile cards in the new host details page

[ggainey]

September 28, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
• RPM
  • New releases of pulp_rpm are out - 3.17.13 and 3.18.3
  • Dalley looking at 2070537 – Solve Dependencies does not add all needed packages to content view with security only filter
  • Modulemd profiles/description are available in rpm/3.18, so could conceivably reappear in 6.12
• Migration
• Ansible
• Pulp Container
• OSTree
• Pulp CLI

Katello

• 2 talks on Pulpcon CFP – Pulp-CLI usage and Katello-Pulp integration update
• Filed Bug #35574: Re-enable indexing and displaying module profile info plus few more fields added in pulpcore 3.18.z - Katello - Foreman to re-enable module profiles and add the new fields in katello. Will file a BZ for 6.12.z around this once it goes through upstream triage.

QE

• next-Z snap 2 bz verification underway
  • Sign off this Friday 9/30
• next-Y snap 12 bz verification
  • Expecting 2 more snaps
  • ggainey to work w/ ltran to test potential deadlock fixes in ‘real’ environments
• Feature testing & automation
• Automation failure fixes
• Ouia-id for both cv and host details are in
  • Any missing IDs will prob be in later releases

[ggainey]

October 5, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
• RPM
  • Dalley still planning to respond to 2070537 – Solve Dependencies does not add all needed packages to content view with security only filter
    • Attention currently redirected to 2070692 – yum check-update gives update notice as broken, or a bad duplicate, skipping.
    • If priorities need to be different let me know
  • Working on an effort to potentially drop the libmodulemd dependency
• Migration
• Ansible
• Pulp Container
• OSTree
• Pulp CLI

Katello

• Upgrade to 3.21 is in progress. Need some input on this: http://pastebin.test.redhat.com/1076220
  • There is a pulp-setting to turn off deprecation warnings
  • https://github.com/pulp/pulpcore/issues/3254
• ‘0537 is not the highest of priorities, but would like some feedback on iballou’s commentary

QE

• next-next-Z is scoped
• next-Y snap 13 bz verification
  • One last snap incoming
• Feature testing & automation
• Automation failure fixes
• Ltran last day is 10/7
  • Lai to tell ggainey who the right QE contact will be for this mtg

[ggainey]

October 12, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • Not much to report
• RPM
  • 3.18.7 is out. Fixed a migration
    • Note that most of the 3.18.z releases have empty changelogs due to the only changes being progressively loosening some dependency requirements for the build team
  • Pulp workflow discussion - “prio-list”
    • Issues · pulp/pulp_rpm · GitHub
    • RH Pulp Kanban board · GitHub
• Migration
• Ansible
• Pulp Container
  • Merged but unreleased https://github.com/pulp/pulp_container/issues/1103
• OSTree
• Pulp CLI

Katello

• 3.21 upgrade in progress, still blocked (Samir can share more)
  • https://github.com/pulp/pulp_deb/issues/652
  • https://github.com/pulp/pulp_file/issues/841
• 2070537 – Solve Dependencies does not add all needed packages to content view with security only filter - weekly nag
• Question RE https://github.com/pulp/pulpcore/pull/3188 failures - paji to reach out to dkliban for help

QE

• Um…who will be covering Pulp/this mtg going forward?
  • Katello, Pulp Sat Liason to work w/ QE mgt to get definite answer

Build Team

[ggainey]

October 19, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • Pulpcore 3.21.1 released last night
    • Paji’s fs-export BZ fix, see changelog for other fixed issues
  • Pulp_file client released last night (to unblock 3.21)
• RPM
  • Progress being made on various BZs
    • 2070692 – yum check-update gives update notice as broken, or a bad duplicate, skipping.
    • 2125366 – Capsule sync fails with Error: It is not possible to add more than one advisory with the same id to a repository version
    • 2077363 – Fail to sync kickstart repositories with same sub repositories concurrently
      • MUCH harder problem than we thought it was
  • Progress being made on potentially dropping the libmodulemd dependency
  • Discussion about dependency solving
  • Not sure if this is pulp or hammer:
    • 2135608 – Custom rpms uploaded via hammer in bulk result in same filename inside custom repository
• Migration
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Oct 26 mtg
  • Conflicts w/ another important mtg - ok to skip next week for us?
  • We will skip the 26-OCT mtg

Katello

• Pulp Telemetry – are you more interested in users outside of Katello?

  • We’re defaulting it to be off for now, but it can be turned on via custom hiera (Puppet).
  • If you do want Katello user telemetry, we’ll need to figure out how to advertise turning it on.
  • Pulp is interested in katello installs, yes
  • Let’s start with “exposed but opt-in via custom hiera”, sociallize w/ katello/foreman community, and see where we go from there

• Working on pulpcore 3.21 upgrades. Gems are available. Thanks! Osousa looking into pulp_deb release.

  • Branching for katello-next is Monday(ish)

[ggainey]

November 2, 2022 (26-OCT skipped)

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • Deadlock issue hopefully addressed finally
    • And releases pushed, see BZ, backports to 3.16, 3.18, 3.21
  • 3.21 release happened last week
• RPM
  • Continuing to work on errata issues
  • Progress being made on concurrent repo syncs
    • Overlapping subrepos/kickstart trees is Bad
    • https://github.com/pulp/pulp_rpm/issues/2278
  • Some discussion about deprecating default profiles upstream
    • Thinking about how that impacts us as we’ve just added a related feature (to show profile metadata)
  • Lots of BZ cleanup happening (yay dralley!)
• Migration
  • QE testing for 2to3-migration-version release
  • ggainey working w/ vsedmik
• Ansible
  • Migration fix updating Collection search_vector:
    • https://github.com/pulp/pulp_ansible/pull/1277
    • Will be available in pulp_ansible 0.16
• Pulp Container
• OSTree
• Pulp CLI

Katello

• AWS RHUI – What is missing in Katello for basic support?
  • Only the two special headers?
• https://github.com/pulp/pulp_container/issues/1122
  • Updated client-libraries can’t talk to old-smart-proxy
  • E.g., 3.21-client can’t talk back to 3.18-pulp
  • AI: sjha to pull a group together to talk through this complicated issue
    • Include vsedmik
• Reminder – does Pulp always copy an erratum if all of its packages are copied too?
• Telemetry changes are merged and will be off by default in katello 4.7

QE

• 3 BZs to be verified for 2to3-migration-version (1 FailedQA but fix PR is already up (thanks ggainey), 2 under test)

[ggainey]

November 16, 2022 (9-NOV skipped for PulpCON)

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • 3.22 currently slated for 30-NOV
    • Will include domains (multi-tenancy)
• RPM
  • Domains-in-rpm planning started
  • BZs in progress
• Migration
  • 0.11.3 released for last 6.9.10 zstream
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
  • 0.16 released
  • CHANGELOG

Katello

• Conflicting Meetings make moderators Sad

QE

• “Fun” simultaneity BZ logged (poss a dup?)
  • Oh look - shared-distttree-subrepo problem
    • 2142689 – Capsule sync fails when a CV is published and promoted to two capsule's LCEs at the same time
    • 2103246 – Capsule sync fails with duplicate key value violates unique constraint "core_repositoryversion_repository_id_number
  • Already in main and rpm/3.18
  • Backporting to rpm/3.17 requires a diff patch - need advance warning

[ggainey]

November 30, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • 3.22 release tentatively scheduled for 5-DEC
• RPM
  • More progress on dropping the libmodulemd dependency
  • Lots of BZs being escalated by support, trying to investigate
  • ppicka will be leaving RH, RPM Team somewhat undercapacity for a bit
• Migration
• Ansible
• Pulp Container
• OSTree
• Pulp CLI

Katello

• Pulpcore 3.18 branch upgraded pulp-rpm to 3.18.9 (From 3.18.7)
  • Client bindings are also at 3.18.9
• Escalation around “container-stuff on smart-proxies”
  • https://bugzilla.redhat.com/show_bug.cgi?id=2148813
  • Entirely fixable on katello-side, woo!

QE

• verified 1948930 – Discrepancy in the packages count in a repository on the client and Satellite
  • so many NEEDINFOs - ggainey to investigate/clear
• Will wait for build-team for downstream packaging of fix for https://bugzilla.redhat.com/show_bug.cgi?id=2148813

Build Team

November 23, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • Pulpcore 3.22.0 scheduled for Nov 30, but will likely slip by one week
    • All tests converted to Pytest and build team should be able to run them on the RPMs they build.
• RPM
  • Considering allowing users to skip treeinfo file. How is katello using the skip types?
• Migration
• Ansible
• Pulp Container
• OSTree
• Pulp CLI

Katello

• Pulp_deb monkey patch in the works. Should make it to 4.7 GA.

[ggainey]

December 7, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • 3.22 will be going out next week
• RPM
  • Primarily working on escalated BZs
  • Fedora discussions around XML-modulemd are finishing up
    • [Modularity] XML format for in-repository modules - devel - Fedora Mailing-Lists
• Ansible
  • 0.16.0 is out: Changelog — Pulp Ansible 0.17.0.dev documentation
    • Deprecated V2 and /ansible/collections/ upload endpoints
    • Renamed CollectionVersion upload fields [namespace, name, version] to expected_[namespace, name, version]
• Pulp Container
• OSTree
• Pulp CLI

Katello

• Bug #35829: Mirror complete sync policy no longer allowed with ignored content types in the repo. - Katello - Foreman → https://github.com/pulp/pulp_rpm/issues/2293
  • Katello fix will be released for 4.6 and 4.7

QE

• Question RE reproducing prob involving Azure
  • “Can’t sync from repo-in-azure.cr.io registry”
  • Hard to test fix w/out azure-access
  • 2026151 – Can't sync private Azure registry to Satellite
• Discussion RE deadlock-testing

[ggainey]

(NOTE: this meeting will be held next on 11-JAN-2023! Happy New Year, everyone!)

December 14, 2022

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • core/3.22 discussions
    • go/nogo tomorrow
    • May not contain the domains work?
  • Merged two PRs that that revealed unexpected plugin impacts
    • Labels-as-hstore (addresses user-reported performance issue w/ labels)
      • Fixes in progress
    • content-app redirects (BZ 2104792)
      • Being reverted for now, to be revisited post-3.22
  • Incremental filesystem import/export
    • Not implemented yet
    • Causing support heartaches
• RPM
  • 2 BZs nearing resolution
    • Dup-advisory
    • modulemd-multiple-get
  • Pulp_rpm is mostly ggainey and dalley
• Ansible
• Pulp Container
  • Fix maxing out idle db connections on pulp_container 2.8.x
    • https://github.com/pulp/pulp_container/pull/1176
  • 2.8.9 will be released soon
• OSTree
• Pulp CLI
• No meeting 21/28-DEC or 04-JAN-2023
  • ggainey to cancel invites

Katello

• Bug #35829: Mirror complete sync policy no longer allowed with ignored content types in the repo. - Katello - Foreman → https://github.com/pulp/pulp_rpm/issues/2293 made it to 4.7 and will be part of 4.6.2
• 6.12 upgrade issue
  • Katello db somehow allowed ‘errata’ into “ignore types”

QE

• testing deadlock BZ - reproducible in 6.12.0 with rhscl, NOT reproducible in 6.12.1 with bigger rhel7 repo (5 CVs, on_demand on SAT side, immediate on CAPS side) - so far looks good!

[ggainey]

2023-01-11

Overview

• Katello Schedule

Pulp

• Pulpcore
  • Query of all changes from last week
  • core/3.22 released, 3.22.1 Real Soon Now to fix a migration issue
  • Should katello-next update to 3.22?
    • Leaning towards “no”, depends on features
    • When does katello need to make a call?
  • A couple backports went out yesterday for various BZs
    • core/3.18 and core/3.21 updated
• RPM
  • Modulemd is proposing to add/be-changed-to .xml (instead of yaml) - does anything in katello (applicability?) look at the modulemd yaml directly?
    • Nope! I didn’t even know we had access to that. [Ian]
  • RHEL moving forwards with a change to stop shipping un-compressed comps.xml
    • Or maybe not. We’re talking through some issues with the DNF team
    • May impact which pulp versions support future versions of RHEL 9.Y
    • We can/will do backports if necessary
• Ansible
• Pulp Container
• OSTree
• Pulp CLI
• Meeting Review
  • Is this useful?
  • Is the cadence useful?
  • Is the format useful?
  • Is the invite-list correct?
  • Is “katello issues” link above useful at all? (suspect not)
    • Current query not relevant
    • No current equivalent
  • Think about these questions, decisions next week

Katello

• katello/4.8 branch - when?
  • Tentatively “early Feb”
• Discuss incremental export
  • 2143497 – Can't perform incremental content exports in syncable format
  • Paji to schedule a meeting with dalley/ggainey
• Pulp-replica project?

QE

• No updates
• BZ-verification underway for next-Y and next-Z

Build Team

[ggainey]

2023-01-18

Overview

• Katello Schedule

Pulp

• Meeting Review
  • Is this useful?
    • Does Ansible need to actually be here?
    • Maybe just add report to the doc prior to the mtg
  • Is the cadence useful?
    • Seems to be
  • Is the format useful?
  • Is the invite-list correct?
    • Corrected with input
• Pulpcore
  • Query of all changes from last week
  • Pulp-replica project update?
    • Basic CLI POC in progress
    • “Replicate” starts with repo/distribution on src-pulp, creates remote on downstream-pulp to create/sync/publish/distribute in downstream
    • Experimentation in progres - talk to dkliban to learn/get involved
    • Pulp Replication - HackMD
  • Requesting feedback on “How do you view ‘tech preview’?” discussion
• RPM
  • Scheduling a discussion about possibly restructuring the incremental update features to be more foolproof, possibly eliminate depsolving
    • RHEL8, libsolv, and module Sadness
• Ansible
• Pulp Container
• OSTree
• Pulp CLI

Katello

• Duplicate data with unique indices: 2142270 – Locale change caused by RHEL upgrade results in database index corruption "get() returned more than one Modulemd -- it returned 2!"
• Discuss https://github.com/pulp/pulpcore/pull/3511/

QE