Katello/Pulp3 Integration meeting

ggainey · October 20, 2021, 3:54pm

October 20, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1 - May 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.2 branching ~August 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 branching 5-Nov 2021
  - Pulpcore 3.16
  - Pulp-rpm 3.17

Pulp

Pulpcore
- Nothing to report
RPM
- PRs merged for sync optimization fix, content mirror without metadata mirror
- Investigating a set of related issues, may try to resolve them prior to 3.16 release
  - If not - rpm/3.16 today/tomorrow
- Will soon be investigating if any depsolving changes are needed to support RHEL 9 (static_context implications to copy() api)
- ACS moved to rpm/3.17
  - Need to insure rpm/3.17 is released prior to katello-4.3 dev-freeze
- Advisory-conflict fix backported to rpm/3.14.7 and released
  - 2013320 – CV publish fails with "get() returned more than one UpdateRecord" if multiple advisories with the same errata_id are requested to be published
Migration
- Nothing to report
Ansible
- No updates
Pulp Container
- Merged azure storage support
OSTree
- adding cli support
- Katello would love a new beta release - dkliban to drive
Pulp CLI
- Prob do a 0.13 when ostree has landed

Katello

Pulp 3.15 appearing in nightly
- release-pipeline fixes happening
Ostree upload work
Installer changes needed for core/3.16?
- Pulpcore-selinux new version
- If there are any 3.15-to-3.16 changes - LET JUSTIN KNOW!

QE

Waiting on snap 24 (should be final snap)
Automation failure fixes in preparation for GA
next-Z bz verification

Build Team

Snap24 Real Soon Now (™)

ggainey · October 27, 2021, 3:46pm

October 27, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1 - May 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.2 branching ~August 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 branching ~Nov 2021

Pulp

Pulpcore
- 3.17.0 release scheduled for Nov 16
- Content app db connection issue should be fixed soon as part of 3.14.9 Issue #9515: content app doesn't survive PostgreSQL disconnect in pulpcore 3.14.7 - Pulp
RPM
- Comps.xml upload API in-progress
  - Requested by RHUI, does katello want to expose it?
- RHEL 9 copy / depsolving support next on the list
- 3.14.8, 3.16.1 going out today
  - Fix a sync issue with Amazon-hosted repos, and (for 3.16) a set of issues with sync optimization in combination with metadata mirroring
- ACS Kbase in draft
Migration
- QE found a FIPS-migration blocker
  - Patch for createrepo_c needed (thanks dalley!)
  - https://bugzilla.redhat.com/show_bug.cgi?id=2017489
Ansible
Pulp Container
OSTree
- Another alpha went out earlier this week
  - Repository Modify API, commit-sha returned with refname
Pulp CLI
- Nothing to report

Katello

Still working on getting 3.15 working in nightly
- Install is green, upgrade issues being resolved
Buildteam 3.16 work starting
Ostree plugin packaged
Ostree upload work (lots of questions)
- jjeffers to set up a mtg to discuss
‘Ping’ will now report content app status

QE

next-Z FIPS issue:
- https://bugzilla.redhat.com/show_bug.cgi?id=2017489
- Coordinate w/ zhunting on new createrepo_c build to retest
next-Y bz verification
Automation failure fixes

Build Team

ggainey · November 3, 2021, 3:50pm

November 3, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1 - May 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.2 branching ~August 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 branching ~Nov 2021

Pulp

Pulpcore
- dkliban is now the Satellite Liaison!
- Pulpcon next week
  - Would be cool to have updates form katello
- 3.17 release slated for the week of the 15th
- Chunked-uploads bug - will be backported to 3.14
  - Issue #9550: Ensure correct tmp location is used when reconstructing chunked uploads - Pulp
RPM
- 3.16.1 released
- Will be focusing on:
  - Ensuring compatibility with RHEL 9
  - Reducing memory consumption during sync
Migration
- Nothing to report
Ansible
- No updates
Pulp Container
- No updates
OSTree
- Fixing some bugs (ref/commit required)
  - https://github.com/pulp/pulp_ostree/issues/140
- Will release once fix is avail
Pulp CLI
Pulp-selinux-1.2.7 released
- Squid, http-proxy support

Katello

Ostree upload work continues, sample small tarball to test with for automation?
- Work w/ dklliban, lmjachky
Ran integration tests with 3.16, all green, no changes needed in katello.
- Waiting on builds
Wanted to raise 2018888 – Actions::Pulp3::CapsuleContent::RefreshDistribution fails with NoMethodError: undefined method `pulp_href' for nil:NilClass likely a dupe, may not need any action

QE

next-Z testing
- Verified: 2017489 – Migration fails when migrating content with FIPS mode enabled (createrepo_c)
- New BZ: 2019563 – Missing fields on MD5 repos in repomd.xml on a FIPS enabled satellite
  - Looks like FIPS-mode is broken in Pulp2-2.21.5
- Failed: 1970246 – Publishing a repository in Capsule can take more than an hour to finish if many cloned repositories have been synced to the Capsule
  - Vlad, pmoravec working together on this
  - Hchen has submitted a pair of PRs for this
- Troubleshooting with ggainey on FIPS on pulp2
  - What if we downgrade to pulp-2.21.4 and retest?
  - Jsherrill to work with ltran
Automation failure fixes

Build Team

No updates today

ggainey · November 17, 2021, 7:54pm

November 17, 2021 (November 10 skipped for PulpCon)

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1 - May 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.2 branching ~August 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 branching ~Nov 2021

Pulp

Pulpcore
- Bulk-task-delete in progress : Story #8554: Provide easy task cleanup api - Pulp
  - https://github.com/pulp/pulpcore/pull/1721
  - https://github.com/pulp/pulp-cli/pull/431
- Discussion about plugin API deprecation policy changing to have 3.20 be the next breaking release
- 3.17 scheduled for Nov 23rd
RPM
- RHEL9, static_context modules, advanced use
  - We may be in “no changes needed” territory
- RHEL8, static_context will not happen (whew)
- Oracle Linux sync and memory issues (dralley investigating)
Migration
- 0.11.6 in progress
  - SLES related Issue #9254: Pulp 2to3 Migration of SLES repositories does not work correctly - Migration Plugin - Pulp
  - Errata batch size issue Issue #9451: Set bach_size to mitigate query result exceeding 16mb - Migration Plugin - Pulp
Ansible
- Z-stream fixes for GalaxyNG Gang
- git-sync work continues
- Collection-signing implementation in progress
Pulp Container
- Content-signing planning is finishing up
OSTree
Pulp CLI

Katello

3.16 close to being done packaging wise, pulling it in should be quick
Ostree upload nearing completion
Rhel 9 testing
- Issue #9570: RHEL 9 beta contains one module stream, but it isn't imported into pulp - RPM Support - Pulp (maybe related to Story #8741: Full support for "static_context" in modulemd documents - RPM Support - Pulp ?)
- No luck reproducing against master or core/3.14, see issue for notes

QE

Automation Failure fixes for next-Z
Automation for closed loop bugs
next-Y snap delayed

Build Team

No report

December 01, 2021 (24-Nov cancelled for PTOs)

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1, 4.2
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 ~Nov 2021
  - pulpcore 3.16(?)
  - pulp_rpm 3.16
- 4.4?

Pulp

Pulpcore
- 3.7.9, 3.15.3 released yesterday
  - 3.15 download-everything-always regression fixed
- 3.17 delayed - go/nogo today, currently next week? (but prob not)
- Plugins coming out soon will declare compat with up to 3.19 which should allow upgrades of a single component, e.g. just pulpcore without compatibility issues
  - Increasing Pulpcore Compatibility with Plugin Versions | software repository management
- Bulk delete of task records is on-track to be included in 3.17
RPM
- Two investigations in-progress:
  - Memory-consumption
    - Following up on foreman issue
  - Distribution-tree fun
    - Issue #9566: Create better primary keys for subtrees - RPM Support - Pulp
      - See the scenario when the issue pops up, not sure if it affects katello
      - Is backport needed? (has an impact on implementation)
    - Issue #9583: Distribution tree uniqueness constraint is not enough for a suboptimal .treeinfo - RPM Support - Pulp
      - ATIX sees the issue often with pulp-2to3-migration
      - CentOS 8 and Alma 8 are affected (RHEL is fine)
      - Conversations with centos folks about the way they generate treeinfos
      - No reports except from 2to3 - why sync isn’t complaining?
Migration
- 0.11.6 is out
  - Customers started hitting the errata issue, consider for next-Y
    - 2003888 – The Pulp 2 to Pulp 3 migration stucks and raises 'pymongo.errors.DocumentTooLarge' error in /var/log/messages
Ansible
- Import Collections from Git continues
- Long-term effort to consolidate the APIs between GalaxyNG and pulp_ansible starting in Jan
- Collection signing is in-progress
  - Will sign all unsigned content in a repo or specific collections. Signatures are content. Will sync signatures also. CLI is getting verification capabilities
  - https://github.com/pulp/pulp_ansible/pull/754
Pulp Container
- 2.8.2 and 2.9.1 are out
- Main focus is container signing feature
  - NET 2.11+
  - Jsherrill to add backlog item for full katello support, ticket for PM radar
- Roles for pulp-container
- Adding redis caching for image pull requests
OSTree
- No updates
Pulp CLI
- Adding full pulp_container support
Infrastructure
- Most pulp- components are now using ‘main’ as default, not master

Katello

Upgraded to 3.16 in nightly, katello 4.3 rc 2 (first downstream snap)
- Double-download-issue needs a backport request
  - Issue #9542: Repositories with packages in common (same hash) redownload the file even if it is already in the artifact folder - Pulp
Integration with repo cleaning in review

QE

No report

Build Team

No report

ggainey · December 8, 2021, 5:18pm

December 08, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1, 4.2
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 ~Nov 2021
  - pulpcore 3.16(?)
  - pulp_rpm 3.16
- 4.4?

Pulp

Pulpcore
- 3.17 to release Dec 14th
- Bulk-task delete is merged to be included in 3.17.0
- Hoping to migrate pulpcore issues to github.com/pulp/pulpcore/issues/ next week after 3.17.0 release
- Removing some APIs for managing ContentGuard permissions
- Telemetry working group starting in Jan
  - Proposal: Telemetry
RPM
- Working to improve sync memory-consumption
  - Mostly happening in createrepo_c land
- Distribution-tree fun
  - Issue #9566: Create better primary keys for subtrees - RPM Support - Pulp (continued from last week)
    - Does this scenario impact Katello? Is a backport needed? (has an impact on implementation)
    - Likely yes?
  - Issue #9583: Distribution tree uniqueness constraint is not enough for a suboptimal .treeinfo - RPM Support - Pulp (continued from last week)
    - No change
Migration
- 0.11.7 is out
  - Includes a fix for importer migration on re-runs Backport #9612: Backport #8968 "'NoneType' object has no attribute 'delete'" during migration to 0.11.z - Migration Plugin - Pulp
- Pulp_deb working on deb-2to3
  - ttereshc to share diffs w/ katello
Ansible
Pulp Container
- 2026277 – null value in column "manifest_id" violates not-null constraint error while syncing RHOSP container images
  - 1. source containers can be a problem, including rate limiting
    - 1845471 – [RFE] Add ability to opt out of mirroring source containers.
    - To exclude source container use exclude_tags = [‘latest’, ‘*-source’]
  - 1. Refactor #9424: Remove the interrelate stage from the sync pipeline - Container Support - Pulp
  - Discussion around how katello/pulp3 could cooperate on how to exclude src-containers
OSTree
Pulp CLI

Katello

Switching to sync policy in progress
- May backport downstream-only? Maybe?
- “Let’s backport correctly” seems to be the consensus
Adding ostree testing to nightly/release pipeline
Tried out new openapi version bindings, failed to build properly
- Issue #9619: OpenAPI schema for ModulemdDefault is incorrect - RPM Support - Pulp
Question: https://bugzilla.redhat.com/showhttps://bugzilla.redhat.com/show_bug.cgi?id=2024963_bug.cgi?id=2024963

QE

Next-X snap 1.1 preview/test
Next-Y automation failure fixes
Watching pulpcon pulp testing and adding feedback

Build Team

No report

ggainey · December 16, 2021, 2:20pm

December 15, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1, 4.2
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 ~Nov 2021
  - pulpcore 3.16(?)
  - pulp_rpm 3.16
- 4.4?

Pulp

Pulpcore
- 3.17 is out
  - It has the bulk deletion for tasks endpoint in it here
  - PulpImport /tmp usage also fixed
- Moving from pulp.plan.io to github issues soon, maybe this week, maybe Jan
RPM
- No significant updates from last week
- Backport releases will go out soon for 3.14 and 3.16 streams
- ttereshc is working on 2027127 – Pulp 2 to 3 migration fails on certain repos during the upgrade with FileNotFoundError: [Errno 2] No such file or directory: '' in prepare_metadata_files
Migration
- Shared pulp_deb contributions with katello, anything need from pulp side?
Ansible
- No updates
Pulp Container
- 2.10.0 is out, compat with pulpcore 3.17
  - Has Azure support and rate_limit option on remote
- Sync will download container signatures by default if sigstore is provided or registry natively supports signatures extensions api - starting with 2.11 version
OSTree
Pulp CLI

Katello

Switching to sync policy in review
Fix for content-migration (request-url-too-long)
Enhancing uploads thru katello-api (ostree)
Ostree working really well so far!

QE

Sat-next work in progress
Looking at cust-case-bugs for automatable cases

Build Team

No report

January 05, 2022

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1, 4.2
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 ~Nov 2021
  - pulpcore 3.16(?)
  - pulp_rpm 3.16
- 4.4?

Pulp

Pulpcore
- Moving to github #soon - all other pulp-projects already migrated!
  - BZ sync will be affected
- Working on backport-request/release process
RPM
- Depsolving / Copy work re: ASML customer case expected
- Memory consumption investigation, multiple different avenues being worked on
- DistributionTree support
  - 3.11 and 3.14 getting backports
- 3.17 will be released without ACS support
  - It is being pushed into 3.18 so that 3.17 can remain compatible with pulpcore 3.15
Migration
- DistTree fixes affect 2to3 as well
  - Needs a code-change, expect another 0.11.z to fix
  - “Conflicting” trees - resolve via tweaking timestamps
  - next-prev-z going out end-of-month(ish) - to QE by 12-JAN
Ansible
- No report today
Pulp Container
- Finished planning for sync pipeline refactor to address this BZ 2026277 – null value in column "manifest_id" violates not-null constraint error while syncing RHOSP container images
  - next-Z if possible? (27-Jan-ish)
- Container image Signing effort ongoing - current top prio
OSTree
- No updates
Pulp CLI
- No updates, still awesome
Move to github issues
- Pulpcore only active project not yet migrated
- Auto-link between bugzillas and ‘upstream’ not in place yet
- Patience please while we iron out the process kinks!

Katello

‘Sync policy’ now implemented (as “mirroring policy”), will likely be backported to current-downstream
https-proxy/aiohttp changes
- Might be a problem still? Under investigation
- 1993917 – Syncing repositories with https proxy set ends with warning Katello::Errors::Pulp3 Error Only http proxies are supported

QE

Busy with new Satellite: Planning for new test automation and fixing some UI tests due to changes in the new version.
- Patternfly4 changes, etc
next-X feature testing
next-X BZ verification
current-Y BZ verification
prev-Y BZ verification

Build Team

No report

ggainey · January 12, 2022, 5:23pm

January 12, 2022

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1, 4.2
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 ~Nov 2021
  - pulpcore 3.16(?)
  - pulp_rpm 3.16
- 4.4?

Pulp

Pulpcore
- Lots of backport releases
  - 3.17.2, 3.16.2, 3.14.10
  - Support a django-release that fixed django-CVEs
- import/export memory issues being investigated
- Sync memory issues being investigation
- Discussion RE TaskReservedResource table has orphaned entries at upgrade time
  - Modify existing migration to trunc tables (downstream)?
    - Pulp team to investigate
  - 2031154 – After upgrading to Satellite 6.10, Repository sync randomly fails if a ReservedResource exists in core_taskreservedresource table of pulpcore DB.
RPM
- Change to release plans - 3.17.0 will include ACS support, support pulpcore 3.16+
- Merged “changelog limit” PR
- Couple PRs blocking 3.17 release - #soon
  - https://github.com/pulp/pulp_rpm/pull/2344
  - https://github.com/pulp/pulp_rpm/pull/2185
- Investigating depsolving to see if we can make improvements
- 3.11.3 is available for prev-Y
Migration
- Fun w/ product-customer cases and CEE continues
  - Will drag ttereshc into the conversation
- https://github.com/pulp/pulp-2to3-migration/pulls
  - Some fixes that will want to be in a new 0.11.z
  - #soon would be good - is this stop-the-presses need?
    - Prob not, ttereshc to check
- 0.11.8 is out for prev-Y (requires pulp_rpm 3.11.3)
Ansible
Pulp Container
- Dennis picked up container sync refactor to address 2026277 – null value in column "manifest_id" violates not-null constraint error while syncing RHOSP container images
- Container signing work
OSTree
- Alpha4 release just happened this morning
Pulp CLI
- Ostree plugin released
- Pulp-deb initial release also happened
product-support looking to have an information session from dev
- Would love katello collab

Katello

ACS support being started, expect questions from iballou!
Support for on_demand for container

QE

next-X feature testing
next-X bz verification
Checking automation and signing off for curr-Y
prev-Y bz verification
More automation work w/ new UI infrastructure

Build Team

ggainey · January 19, 2022, 5:00pm

January 19, 2022

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1, 4.2
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 ~Nov 2021
  - pulpcore 3.16(?)
  - pulp_rpm 3.16
- 4.4?

Pulp

Pulpcore
- Moved entirely to github issues. Any feedback or improvements are welcome.
  - https://github.com/pulp/pulpcore/issues
  - No more backport issues, comment on the issue or PR if you need one?
    - “Leave comment on PR itself” to request a backport
  - All changes since last week query: https://github.com/pulp/pulpcore/pulls?q=is%3Apr+closed%3A>2022-01-12+is%3Amerged
- Hoping to finalize patchback use to ensure consistent backporting back to 3.14
  - https://github.com/pulp/plugin_template/pull/551
  - https://github.com/pulp/plugin_template/pull/552
- Retrying on validation errors added
  - https://github.com/pulp/pulpcore/pull/1807
- Making Redis not enabled by default
  - https://github.com/pulp/pulpcore/pull/1752/files
  - Pulp installer still configures it, but katello installer does not
  - core/3.18 will have this
  - Jsherrill to open a foreman-installer ticket
- No longer logging for deprecated API calls, e.g. orphan cleanup
  - https://github.com/pulp/pulpcore/pull/2100
- Bugfix against 3.14 tasking system migration to delete tasks and reserved_resources
  - 2031154 – After upgrading to Satellite 6.10, Repository sync randomly fails if a ReservedResource exists in core_taskreservedresource table of pulpcore DB.
  - https://github.com/pulp/pulpcore/issues/2101
- Improved logging for OOM of tasks
  - https://github.com/pulp/pulpcore/pull/1806
- Import/export OOM error
  - https://github.com/pulp/pulpcore/pull/1782
RPM
- 3.17.0 (.1) released
- 3.14.10 released
- Continuing to work on memory consumption of the RPM first stage
- Nearly finished with depsolving report generation, to help facilitate much easier debugging of depsolving issues in the future
- Nearly finished with a tool for trimming the changelogs of already-synced packages - should allow a lot of disk space to be reclaimed and speed up manual publishes significantly
- Distribution tree issue
Migration
- NFS issue is on the todo list, likely pulpcore problem
  - 2041508 – migration fail if /var/lib/pulp is NFS share
Ansible
- Intermittent Validation error when syncing a community collections
  - https://github.com/pulp/pulp_ansible/issues/773
  - Maybe resolved by this pulpcore change
- Collection signing and the syncing of signatures is in-place with pulp_ansible
Pulp Container
- BZ docker sync escalation solved with a hotfix . Refactor of the pipeline continues, it will be backported.
- Container signing work continues
- Adding redis caching support for container registry
- Porting onto RBAC roles almost complete PR, still needs migration TBD
OSTree
Pulp CLI

Katello

Pulp 3.17 being built into rpms (may not make katello 4.4)
ACS work still ongoing, initial testing looks good

QE

prev-Y bz verification
next-X bz verification
next-X feature testing
Fixing automation failures

ggainey · January 26, 2022, 4:57pm

January 26, 2022

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1, 4.2
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 ~Nov 2021
  - pulpcore 3.16(?)
  - pulp_rpm 3.16
- 4.4?

Pulp

Pulpcore
- Query of PRs merged in the last week
- Many backports
  - 3.14 backports
  - 3.17 backports
- Lots of rollout of patchback to old branches
- A new current_task field is now shown on workers (also in status API)
  - https://github.com/pulp/pulpcore/pull/2117
  - Has a downstream backport BZ now
- Pulpcore 3.18 tentatively scheduled for Feb 8th
  - Pulpcore 3.18.0 release schedule
- Upgrading django-lts versions in 3.14 z-streams
  - Some confusion around motivation/implications
  - Group to re-examine the and agree on the motivations (dkliban, rchan to org)
RPM
- 3.14.11, 3.17.2 have been released
  - Changelog trimming \o/
  - Tool to trim the changelogs of already synced packages (3.17 only)
  - Fixes a distribution tree issue https://github.com/pulp/pulp_rpm/issues/2373
  - Logs decisions made by the dependency solver in /var/tmp/pulp/ (3.17 only)
- A distribution tree issue remains which is difficult to backport past 3.17
  - https://github.com/pulp/pulp_rpm/issues/2304#issuecomment-1019297646
- No updates on memory-use front
Migration
Ansible
- Collection content signing from console.redhat.com
- Syncing these collections
- Ansible-galaxy CLI feature to verify these checksums in progress
Pulp Container
OSTree
Pulp CLI
Process work
- bugzilla/gh-issue automation in place - report any problems to fao89
- Gihub Issues: Process and Procedures - HackMD
  - Stakeholders - please feel free to comment here!
- Jsherrill to review/update katello releases and dates and stuff

Katello

ACS support underway (Ian)
Docker/podman push support, seeing issues, may need help (Justin)
- Auth bug (to be filed)
- client/server comm complaining - will bug ipanova or matrix #pulp-container
3.17 RPMs still being built by packaging

QE

prev-Y automation signoff
next-X bz verification
next-X feature testing
Fixing automation failures

Build Team

3.17 rpm-builds in progress
- done #soon?

ggainey · February 2, 2022, 9:40pm

February 02, 2022

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1, 4.2
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 ~Nov 2021
  - pulpcore 3.16(?)
  - pulp_rpm 3.16
- 4.4?

Pulp

Pulpcore
- Query of PRs merged in the last week
- Some backports, see query ^
- More CI fixes as we improve our backporting process
- One bugfix about cancelled tasks not being picked up
- In progress, but not yet merged
  - Improved proxy testing
  - Ability to reroot /pulp/api/v3/
  - Issue-creation-template being created to improve issue-process
- ggainey working on distribution tree import/export
  - https://github.com/pulp/pulpcore/pull/2193
RPM
- RBAC effort will start this month
- Katello to open an RFE to be able to export metadata (publications)
Migration
Ansible
- 0.12.0 releasing today
  - Contains new signing features
- Improved proxy testing underway
Pulp Container
- Moving our tests away from dockerhub to github registry ( avoid docker pull limits CI is reaching)
- 2.8.4 released ( manifest sync bugfix)
- Container signing mvp more or less done, tests to follow
- Wrapping up adding redis caching on requests
- Working on adding cross-repository blob mount (docker push)
- Working on removing InterrelateStage from the sync pipeline
- Wrapping up transition to Roles (RBAC)
- In the queue to look at adding support for PulpRemoteUserAuthentication (docker push)
  - Not (currently) urgent for katello
OSTree
Pulp CLI
- Ability to work with a Pulp system that is rerooted

Katello

Initial ACS work under review (with file)
Success with docker push (one bug filed https://github.com/pulp/pulp_container/issues/558)
- Need to think through workflow
- Allow pre-creation?
- On hold for now due to bugfixing
- AI: ipanova/jsherrill to brainstorm workflows
Ryan giving presentation to CEE around ostree tomorrow
AI: dklliban/jsherrill to brainstorm upstream use-cases
Pulp_rpm ruby-bindings issue: openapi-gen-dep got bumped
- Need to release a bindings-version w/ a lower faraday-version req
  - Allow 1.x and 0.7.13
- Will definitely need testing support
- VCR testsuite will catch problems
- AI: Pulp to work w/ build-team ASAP

QE

next-Z bz verification
next-X bz verification
next-X feature testing
Fixing automation failures

Build Team

ggainey · February 9, 2022, 9:11pm

February 09, 2022

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1, 4.2
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3, 4.4 ~Nov 2021
  - pulpcore 3.16(?)
  - pulp_rpm 3.16
- 4.5 ~May 2022

Pulp

Pulpcore
- Query for PRs merged in the last week
- analysis/perf-testing on touch()-deadlock issue
  - 2021406 – syncing tens of repos to capsule can cause deadlock: while updating tuple (...) in relation "core_content"
  - touch() and deadlocks - HackMD
- Duplicate key error during add_and_remove content
  - 2044631 – duplicate key value violates unique constraint "core_repositorycontent_repository_id_content_id_df902e11_uniq"
  - https://github.com/pulp/pulpcore/issues/2047
RPM
- No updates
Migration
- No updates
Ansible
- Allow users to upload signatures - new endpoint being designed
Pulp Container
- Moved our tests away from dockerhub to github registry ( avoid docker pull limits CI is reaching)
- 2.9.2 released
- Merged redis caching on requests
- Docs and tests for signing epic
- PR up for cross-repository blob mount (docker push)
- Wrapping up transition to Roles (RBAC)
- Looking at adding support for PulpRemoteUserAuthentication (docker push)
OSTree
Pulp CLI

Katello

Adjusting remote sync timeouts to specify more specific timeouts
- Curr dflt is too low
Meeting on Monday to talk through container push workflow, let me know if interested
Adding container exclude tags, defaulting to exclude source containers

QE

next-Z bz verification
next-X bz verification
next-X feature testing
Fixing automation failures

Build Team

ggainey · February 16, 2022, 7:32pm

February 16, 2022

Overview

Katello Schedule
- 4.5 ~May 2022
  - Pulpcore 3.18(?)

Pulp

Pulpcore
- Query of all changes from last week
- 3.18 to release on Feb 22
  - New mgmt command to view publication contents
  - SQL deadlock fix merged and backported
  - Import/Export bugfix merged (not backported)
  - Variety of RBAC change (not significant to katello)
  - Scheduled tasks likely to be added to 3.18
    - Not user editable, only for plugin writers currently
  - Re-rooting feature available for pulp-content
    - Currently hard-coded to /pulp/api/v3/
    - Will be /api/v3/
    - Q: Can you change it on an existing system?
    - Comment: katello prob won’t care about this
RPM
- Possible 3.17 migration bug, investigating (dalley)
  - 3.17.4 likely to come out this week if so
- Update on memory consumption, call for feedback
  - Specifically - once changelog-limiting is available - need specifics on current-memory-use looks when under load/real-data scenarios
- RBAC work in progress for 3.18
Migration
Ansible
- We need @mdellweg invited here to report
  - done
- Signature Uploads in-progress
- API refactor in-progress
  - Primarily drop-in replacement
  - Will ease multi-repo context
  - Will be a few weeks before available
Pulp Container
- Bunch of backports and releases 2.5, 2.8, 2.9 and 2.10
- Looking at adding support for PulpRemoteUserAuthentication (docker push)
OSTree
Pulp CLI

Katello

Added rate limit option
- Takes advantage of the existing pulp API
- RFE for a speed / bandwidth limit, waiting on a description of the use case: 2035224 – [RFE] Implement a Satellite installer option that provides same functionality as the deprecated '--katello-pulp-max-speed' option
Added block list for container syncing, for blocking source containers
Timeouts adjusted (sock timeout set to 60 seconds, download timeout set to 3600 by default), adjustable by the user
Errata issued_dates not nullable (possible change from pulp2?)

QE

next-Z bz verification
next-X bz verification
next-X feature testing
Fixing automation failures
QE/katello discussion on 2045854 – organization context fails to change in web UI

Build Team

ggainey · February 23, 2022, 8:13pm

February 23, 2022

Overview

Katello Schedule
- 4.5 ~May 2022
  - Pulpcore 3.18(?)
    - Pulpcore 3.17 first
  - pulp_rpm 3.17

Pulp

Pulpcore
- Query of all changes from last week
  - Query for 3.14 updates in last week
- pulpcore/3.18 released - changelog
  - Highlights: plugin-removal, workers have a current-task, analyze-publication cmd
RPM
- No changes needed for the migration issue discussed last week, Postgresql versions >= 12 unaffected, upgrading solves the problem
- Work on BZs continues
Migration
Ansible
Pulp Container
OSTree
Pulp CLI

Katello

rpm/3.17 still “on the board”
- [iballou] raise question about priority of this for next-X testing (pretty please)
prev-Y-related bugs
Migration-related Stuff (mostly UX)

QE

next-Z bz verification, snap-2 dropped
- Unexpected failures being investigated
next-X bz verification
next-X feature testing
Fixing automation failures

Build Team

ggainey · March 2, 2022, 4:55pm

March 02, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Releases for 3.18.1, 3.16.4, 3.17.4 (latter have MANY bugfixes)
- Decreasing memory-use during sync in progress (see bmbouters for details)
- Support of metalink URLs for remotes starting discussion - get your opinions in now!
RPM
- 3.17.4 released
  - Fixes ruby bindings, one bugfix
- BZ work continues
- “Very close” to having “a” sync-memory mitigation ready
  - A few hundred megabytes per sync + improves the time by 60-120 seconds
  - Also pulpcore work in train for same
Migration
Ansible
Pulp Container
- Finished RBAC roles transition
- PR available for the RemoteUserAuth with token_auth=False https://github.com/pulp/pulp_container/pull/604
OSTree
Pulp CLI
- 0.14 released, see Changelog - Pulp 3 CLI

Katello

upgraded to rpm/3.17 in our core/3.16 repos
- Talk to iballou RE making this happen
Moving to core/3.17 “soon”
Should request builds for core/3.18 “soon” (was released last week)
Wanted to mention: 2033853 – Publish content view failed with "PulpRpmClient::ApiError Error message: the server returns an error" (client side timeout in bindings not working)
import/export feature - meeting request incoming
- https://github.com/Katello/katello/pull/9994
- https://github.com/pulp/pulpcore/issues/2269
Discussion RE who will be “The One” for katello as jsherrill moves responsibilities

QE

next-z bz verification, snap-2, most ppl done
nexy-Y bz verification snap 11
next-Y feature testing
Fixing automation failures

Build Team

ggainey · March 9, 2022, 4:52pm

March 09, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- 3.14.13 released yesterday
- Docs and openapi schema are now human readable
  - https://github.com/pulp/pulpcore/pull/2292 (see pics)
- import/export discussions happening w/ regard to potentially newly-uncovered requirements?
  - paji/ggainey/bbuckingham involved
RPM
- 3.14.13 released yesterday
- Working on a few issues reported on foreman forums
- RBAC work continues, see ppicka for details
Migration
Ansible
- ggainey to touch base w/ x9c4 RE adding lines here
Pulp Container
- Support for cross repo blob mount is in
- Support for upload of foreign layers is in
- Support for manifest list push is in
- Preparing for 2.11 release that will have content signing support plus mentioned features ^
- Remote user auth fix is in, what branch do you want it backported to?
  - No need to backport, will take with core/3.18
- Fixed a customer bug “ Can’t sync private repos from Azure registry”, what branch do you want it backported to? (2.8?)
  - 2026151 – Can't sync private Azure registry to Satellite
OSTree
Pulp CLI

Katello

core/3.18 builds requested
“Move to core/3.17” has made it “on the board”
rpm/3.17 - in upstream, will be in next product snap
Working on client bindings update

QE

next-Z snap 1 tomorrow
prev-Y bz verification in progress for snap 2
next-X bz verification in progress for snap 12
next-X feature testing
Fixing automation failures

ggainey · March 16, 2022, 4:01pm

March 16, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- New bugfix reducing memory 27%, e.g. on EL7 on_demand sync
  - Being backported to 3.14+ #soon
- Remote “encrypted fields” data integrity bug fixed, with a repair script
  - https://github.com/pulp/pulp_rpm/issues/2327
  - Katello 4.3 need to take this patch ASAP and document running the repair script, or have the installer trigger it
  - (not released yet, but soon)
  - If you upgraded from core/3.14 to core/3.16 (or later), and had encrypted fields for remotes, you may have been affected by this
- Adds TASK_DIAGNOSTIC setting that records memory usage over time chart
  - https://github.com/pulp/pulpcore/blob/main/docs/plugins/plugin-writer/concepts/tasks/diagnostics.rst
- Task cleanup added to existing 3.14 migration, fixes significant upgrade issue
  - https://github.com/pulp/pulpcore/issues/2101
  - Will be backported to 3.14+ #soon
RPM
- Grant produced a patch for 2024963 – Syncing EPEL repos on Satellite 6.10 fails with: "Incoming and existing advisories have the same id but different timestamps and non-intersecting package lists.." (EPEL advisory fun)
  - Review in progress
  - Will be backported to 3.14+
- Createrepo_c streaming API PR still making progress - limited by PR review
  - Measurements taken in combination with Brian’s pulpcore PR - RHEL 7 uses less than 900mb to sync https://github.com/pulp/pulpcore/pull/2338#issuecomment-1068695320
  - 1994397 – Increased memory usage of pulp-3 workers during repo sync
- Pulp_rpm RBAC update tests being written, should be submitted #soon
  - Prob won’t affect katello (who uses just-admin)
Migration
Ansible
- API refactor underway for the interaction of the ansible-galaxy CLI
  - Backwards compatible
Pulp Container
- Releasing 2.11 today
  - All the signing work, lots of other Stuff
- When 2.8.z release is needed with this bugfix ?2026151 – Can't sync private Azure registry to Satellite
  - Needs to be in “next Z”
OSTree
Pulp CLI

Katello

How critical is https://github.com/pulp/pulpcore/issues/2327 ?
- GA’ing Katello 4.4 would mean forcing people up to Pulpcore 3.16 through the EOL’ing of Katello 4.2
- It’s Important
Permission requirements for /var/lib/pulp/media/* and /var/lib/pulp/tmp/* ?
- Is 0750 okay?
- 755 would work as well - owner needs to be “user who’s running pulp services”
  - Or 770, if apache/pulp

QE

next-Z snap 2 bz verification in progress
next-Y bz verification in progress for snap 13
next-Y feature testing
Fixing automation failures

ggainey · March 23, 2022, 4:53pm

March 23, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- Significant memory reduction bugfix merged, and being backported
  - 27% reduction in memory with EL7 on_demand sync and 5% less runtime
  - Backporting will be happening #soon
- RemoteArtifact duplicate key error bugfix merged, and being backported
- Pulpcore now publishes its bindings docs
  - Pulpcore Client
  - Should we publish ruby-bindings-docs as well?
    - “Handy”, more for new folk coming on
RPM
Migration
- 0.11.10 released w/ one bugfix (511)
Ansible
Pulp Container
- Released 2.11
- post-release work happening
- Sync-pipeline refactoring in progress
OSTree
Pulp CLI
Pulp-openapi-generator
- patch to expose timeouts merged
Release cadence - we might pick a “release day” once per week
Query: how does katello setup auth-backends?
- Taken up post-mtg

Katello

Updating to Pulpcore 3.16.5
Db-encrypted-fields script under test for katello-4.4
Still fighting with the repair of repositories after migrating to Katello 3.18 (now on 4.1.2) - #31 by rbremer - Support - TheForeman
- May need some more help.

QE (In QE all hands, won’t make it in this meeting)

prev-Z snap 1 bz verification. Waiting on snap templatization before verification
curr-Z snap 3 bz verification in progress
nexy-Y bz verification in progress for snap 14
next-Y feature testing
Fixing automation failures

ggainey · March 31, 2022, 6:30pm

March 30, 2022

Overview

Katello Schedule

Pulp

Pulpcore
- Query of all changes from last week
- releases happened last week
  - core/3.14.15 rpm/3.14.14
  - Bindings updated to expose timeout setting
- There has been a request to prioritize 2035873 – Concurrent CV Publish fails with 502 error
- “Tuesday” is going to become Pulp-Release-Day (we hope)
RPM
- Memory improvements to be merged released imminently (finally)
  - Combined with some pulpcore improvements that Brian worked on, the biggest syncs stay under 0.9 gb of memory consumption, down from ~3.5 at the release of Sat 6.10
- Working on other small fixes
- RBAC work progressing/finishing ip
Migration
- Should consider EOL for 2to3 (core/3.20?)
Ansible
- Galaxy API Refactor Stage 1 merged https://github.com/pulp/pulp_ansible/pull/849
- Signature workflows: https://github.com/pulp/pulp_ansible/pull/867
- Signatures in exports still being worked on: https://github.com/pulp/pulp_ansible/pull/845
Pulp Container
OSTree
Pulp CLI
- No updates
Can we update the pulpcore version being used in Sat-Next-Y?
- Where/with whom should we discuss this?
- core/3.17 heading for katello/4.5
- dkliban/dalley to work w/ iballou on what this implies

Katello

https://github.com/pulp/pulp_rpm/issues/2456 (Add profile information to modulemd endpoint) BZ: 2063999 – No profiles are shown for any module streams
Discuss upgrading to Pulpcore 3.17 in Satellite
- RHUI is going GA with 3.17 - less packaging work, less backporting work, more testing if we consolidate
- 3.16 is from October, will be 8 months old on release day. RHUI will GA with a newer version.

QE

curr-Z GA had a missing package: Bug 2069827 - Subscription & Red Hat Repositories pages are blank after upgrade the minor release
prev-Y snap 1 bz verification in progress.
next-Y bz verification in progress for snap 14
- Pulp bz: 2069822 – Syncing RHEL 5 KS repository fails with: "ModelBase object argument after ** must be a mapping, not str"
next-Y feature testing
Fixing automation failures

Katello/Pulp3 Integration meeting

October 20, 2021

Overview

Pulp

Katello

QE

Build Team

October 27, 2021

Overview

Pulp

Katello

QE

Build Team

November 3, 2021

Overview

Pulp

Katello

QE

Build Team

November 17, 2021 (November 10 skipped for PulpCon)

Overview

Pulp

Katello

QE

Build Team

Other Topics

December 01, 2021 (24-Nov cancelled for PTOs)

Overview

Pulp

Katello

QE

Build Team

December 08, 2021

Overview

Pulp

Katello

QE

Build Team

December 15, 2021

Overview

Pulp

Katello

QE

Build Team

Other Topics

January 05, 2022

Overview

Pulp

Katello

QE

Build Team

January 12, 2022

Overview

Pulp

Katello

QE

Build Team

January 19, 2022

Overview

Pulp

Katello

QE

January 26, 2022

Overview

Pulp

Katello

QE

Build Team

February 02, 2022

Overview

Pulp

Katello

QE

Build Team

February 09, 2022

Overview

Pulp

Katello

QE

Build Team