Katello/Pulp3 Integration meeting

ggainey · August 11, 2021, 4:39pm

Regular meeting to discuss integration points and priorities between the Pulp, Katello, and QE teams.

Template:

### date  
#### Overview
* Katello Schedule
  * 3.18
    * pulpcore 3.7
    * pulp_rpm 3.10
  * 4.1 branching ~May 2021 
    * pulpcore 3.14
    * pulp_rpm 3.13.2
  * 4.2 branching ~August 2021
  * 4.3 branching ~Nov 2021
#### Pulp
##### Pulpcore
* 
##### RPM
* 
##### Ansible
* 
##### Pulp Container
* 
##### Pulp CLI
*
#### Katello
*

#### QE
*

ggainey · September 15, 2021, 5:10pm

May 26th, 2021

Overview

Katello Schedule
- 4.1 branching ~May 2021
  - pulpcore 3.10 (or newer)
- 4.2 branching ~August 2021
- 4.3 branching ~Nov 2021

Pulp

Pulpcore

Dropping support for Python 3.6 and Django 2.2
- https://github.com/pulp/pulpcore/discussions/1359
- Feedback def desired
Releases
- https://github.com/pulp/pulpcore/releases
- 3.13.0 going out today
- 3.11.2 going out today
Pulp is testing Github Discussions as replacement for pulp-dev and pulp-list
- https://github.com/pulp/pulpcore/discussions
- Please experiment and give us feedback
Pulp is evaluating using Matrix as replacement for Freenode

RPM

Metadata mirroring MVP finished, pending review
- Some BZs being investigated currently for Pulp 2
  - 1948930 – Discrepancy in the packages count in a repository on the client and Satellite
  - 1959412 – Solve Dependencies does not pull all stream dependencies for kickstart without all streams no errata filter
Expect 3.11.1 with fixes:
- Katello-3.18 will want to consume this
- Issue #8812: syncing a repo with 'skip_types' set to 'srpm' throws an error 'popitem() takes no keyword arguments' - RPM Support - Pulp
- Issue #8814: static_context cannot be None - RPM Support - Pulp
Migration
- Distribution tree issues
  - 1964530 – [pulp3] ValueError: No declared artifact with relative path ".treeinfo" for migrating CentOS 8 kickstart repos - solved
  - 1964593 – [pulp3] on_demand distribution trees with no images are not migrated - in progress

Ansible

0.8.0 to be released
- Later this week (maybe at the beginning of the next week)

Pulp Container

2.2.2 release going out today
- Evgeni owes ipanova even more cookies

Pulp CLI

Katello

Issue #7828: syncing two repos with the same content at the same time results in an error - Pulp
- Katello needs the priority raised on this
More pulp2 code removal (actions)
- Potentially will remove smart proxy syncing
Repository naming during migration (katello bug)
- Multi-org might be a/the problem?
- Two orgs with the same repo sync’d, and then migrated?

QE

HTB Support
next-Y feature testing
Automation failure fixes
Next z-stream coming into QE next week

ggainey · September 15, 2021, 5:11pm

June 2, 2021

Overview

Katello Schedule
- 4.1 branching ~May 2021
  - pulpcore 3.11
  - pulp_rpm 3.10
- 4.2 branching ~August 2021
- 4.3 branching ~Nov 2021

Pulp

Pulpcore

Experimenting with moving from mailing-lists to github Discussions
- https://github.com/pulp/pulpcore/discussions
Dropping Python 3.6 and upgrading to Django 3.2
- Met with build team and we’re aiming for pulpcore 3.15 (late July/Aug)
3.11.2 and 3.13.0 are GA
- https://github.com/pulp/pulpcore/discussions/1369
- https://github.com/pulp/pulpcore/discussions/1371
New Tasking System with 3.14
- Blog post is being made recapping benefits
  - Performance, high availability, reliability
- Review mtg incoming - ping bmbouter/mdellweg if you are interested in attending
- Investigation of current tasking system issues
  - https://github.com/pulp/pulpcore/discussions/1383

RPM

Metadata mirroring MVP finished, merged
Several Pulp 3 bugs being worked on / recently fixed

Migration

Pulp 3.13+ compatible release coming soon
Fixed migration when one migrates one plugin at a time, under review
- Issue #8686: Migrating one plugin removes publications/distributions of others. - Migration Plugin - Pulp
On_demand kickstart migration issue, in progress
- Issue #8817: on_demand distribution trees with no images are not migrated - Migration Plugin - Pulp
Removing the plugin
- Not done yet on katello-side
- The ‘simple’ answer might be Too Simple (table-inheritance is an issue?)
- Jsherrill to add a card to their board

Ansible

0.8.0 is released - Changelog — Pulp Ansible 0.10.0 documentation

Pulp Container

2.2.2 released
import/export work in-progress
- https://github.com/pulp/pulp_container/pull/335

Pulp CLI

Pulp Certguard

What is the timeline for Issue #8783: CA certificate does not handle a CA bundle or chain file correctly - CertGuard - Pulp
- Talk to ehelms!

Katello

Filed: Story #8852: Delay publication file deletion to orphan cleanup to aid in online backups - Pulp
- Better enable online backups
- Online backup recommendations?
Bumped up priority of suse migration issue: Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file" - RPM Support - Pulp
- Can we talk to QE to get SUSE-repo access?
- Talk to Jake Callahan
More pulp2 code removal (actions) (continued)

QE

HTB customer support
next-release feature area testing and bz verification
zstream migration bz: Null in column “downloaded”: https://bugzilla.redhat.com/show_bug.cgi?id=1967131

ggainey · September 15, 2021, 5:12pm

June 9, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.10
- 4.1 branching ~May 2021
  - pulpcore 3.11
  - pulp_rpm 3.11
- 4.2 branching ~August 2021
- 4.3 branching ~Nov 2021

Pulp

We’re leaving freenode for matrix and libera.chat - come find us there!
- We moved to Matrix as official and libera as bridged officially
- Pulp Community is Leaving Freenode | software repository management

Pulpcore

‘Cleaning’ of on_demand repo is planned to be in pulpcore 3.15/3.16
- 1459231 – [RFE] Support 'cleaning' a repo of downloaded on_demand content
- Story #8459: As a user I want to reclaim disk space for a list of repositories - Pulp
- Consider integrating in Katello 4.3
3.15 will drop Python 3.6 and Django 2.2
- Planned for pulpcore-3.15 to give build-team time
- Will use Python 3.7 and Django 3.2
- Installer will no longer support EL7
- Task #8854: Indicate the installer no longer supports EL7 - Pulp
Orphan cleanup changes
- Story #8876: As a user, the `DELETE /pulp/api/v3/orphans/` is deprecated in favor of `POST /pulp/api/v3/orphans/cleanup/` - Pulp
- DELETE /pulp/api/v3/orphans/ being deprecated
  - Will remain in Pulp3 ‘forever’
- POST /pulp/api/v3/orphans/cleanup/ replacing it
Pulpcore 3.14 scheduled for Jun 29
- https://github.com/pulp/community/discussions/22
Planning eventual migration from pulp.plan.io to Github Issues
- Github Issues Migration Plan - HackMD
Desire to continue even if 403, 404, or other errors…
- Story #8873: pulp 3 stops when encounters 403 error - Pulp
- Feedback wanted
Tasking system with 3.14 getting good feedback
- 1500 task bursts (user workload) all completing successfully

RPM

Several bugfixes in progress
- https://github.com/pulp/pulp_rpm/pulls
- Notably, repository signing in pulp_rpm 3.11 is currently broken with pulpcore 3.11+
Several BZs will be investigated soon
- 1965936 – Huge memory usage when using CV filters in Sat6.9 with pulp-3
- 1965942 – pulp3: invalid depsolving warnings when publishing a CV with filters
Mirror-metadata feature is merged
- Not released yet
- Just need to specify ‘mirror’ - not a special kind-of repo

Migration

Fixed/merged
- Migration when one migrates one plugin at a time Issue #8686: Migrating one plugin removes publications/distributions of others. - Migration Plugin - Pulp
- On_demand kickstart migration issue Issue #8817: on_demand distribution trees with no images are not migrated - Migration Plugin - Pulp
- Very old pulp content (Satellite 6.1) issue Issue #8863: Very old pulp content that does not have `downloaded` flag fails during pre-migration - Migration Plugin - Pulp
Planned
- on_demand non-sha256 issue Issue #8725: Migrated repos are always published with sha256 for metadata if checksum type is not explicitly configured - Migration Plugin - Pulp, needed for 6.9.z
- SUSE repo migration issue Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file" - RPM Support - Pulp
Expect 0.11.z and 0.12.0 this week
- 0.12.0 is compatible with pulpcore 3.12+

Ansible

Some configuration-issues resolved
Token-support in-planning in pulpcore
- Will let ansible-tools pull from katello/satellite
- Pulpcore-3.15 current plan

Pulp Container

import/export in progress

Pulp CLI

No updates

ACS (Alternate Content Sources)

which plugins are needed by Katello?
- Rpm

Content-app performance investigation

Possibly not as bad as we thought last month
Gerrod has a caching-PR that has made great improvements
- Planned for 3.15?
- Gerrod investigating making it back-portable to 3.11

Katello

Sync download errors, lots of users reporting the inability to complete a successful repo sync
- Cdn.redhat.com (Issue #8867: downloads from cdn.redhat fail with 'Cannot connect to host cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:877)]' - Pulp)
  - Immediate-dl failures more prevalent
  - Ggainey has been investigating, see Issue #8867: downloads from cdn.redhat fail with 'Cannot connect to host cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:877)]' - Pulp
  - May need to re-evaluate “internal-to-sync retries”
  - Ipanova starting discussion w/ EXD on the problem
- Centos (download concurrency of 10 seems too high?)
  - Payload-is-not-complete errors
- Going to be a real issue downstream, already is an issue upstream. One user has reverted back to pulp2
- Filed: Issue #8865: Range requests for on demand content return the full file (Kickstarts fail for on_demand repos) - Pulp (range request issue)

QE

HTB
zstream bz verification
next-y bz verification and feature testing

ggainey · September 15, 2021, 5:12pm

June 16 2021

Overview

Katello Schedule

3.18
- pulpcore 3.7
- pulp_rpm 3.10
4.1 branching ~May 2021
- pulpcore 3.11
- pulp_rpm 3.11
4.2 branching ~August 2021
4.3 branching ~Nov 2021

Pulp

Pulpcore

Pulp-dev@ mailing list will be decommissioned
- Subscribe to https://github.com/pulp/community/discussions
Pulpcore 3.14 still scheduled for Jun 29
- https://github.com/pulp/community/discussions/22
Progress on old-tasking system issues
- https://github.com/pulp/pulpcore/pull/1416
Retry issue
- PR open, working through issues
- https://github.com/pulp/pulpcore/pull/1401
Content app in 3.14 will have substantially improved performance
- ~20% improvement without caching
- With caching (new feature), 2x improvement throughput, 3x improvement latency, + lower CPU usage
- Can this be backported? Discussion to be had
- “Pretty close” to Pulp2’s expected performance
New-style tasking system likely default in pulpcore 3.14
- Feedback to bmbouter
Upstream installer will likely continue to support EL7 even with pulpcore==3.15 when python 3.6 is dropped. The Python 3.8 SCL will be enabled.
Discussion: can katello-4.1 move to core-3.14+?

RPM

Huge reduction in memory use during sync:
- https://github.com/pulp/pulp_rpm/pull/2016
- Long-term fix requires work in createrepo_c
- createrepo_c Gang is interested
  - May be able to get the ‘real’ fix for 7.x?
- Discussion ongoing
Signing service currently broken
- Root cause of 1920511 – 3rd party repository sync fails with 'InvalidStringData: strings in documents must be valid UTF-8' discovered
- Should be easy to patch
Pulp2 issue

Migration

0.11.2 released
- Django no longer pinned
- Fixes for distribution tree migration
- Fix for checksum type migration
- Fix for those who upgraded from pre-historic pulp (Sat 6.1)
Tar.gz in repomd migration issue Issue #8275: ComplexRepoMigration fails with "file doesn't exists or not a regular file" - RPM Support - Pulp, reproduced, in progress
0.12 “soon” - ironing out release-process issues

Ansible

import/export discussions - finalizing use-cases
New integration around sync-lists
Token-auth support (coming in core, ansible will get a content-guard)
- Gubben is leading
- Aimed at 3.15
Content-deletion features incoming
- Reclaim disk space for a list of repositories: Story #8459: As a user I want to reclaim disk space for a list of repositories - Pulp
- Purge individual orphans: Story #8658: As a user I want to purge individual orphan or list of orphans - Pulp
- Limit number of repo versions: Story #8368: Limit number of old repository versions. - Pulp

Pulp Container

Pulp CLI

No updates

Community Survey is open

Katello

3.13 builds underway, hopefully we will get on it today
UL/cython issue (“cython not avail on py3/el7”)
- Build team may have a workaround
- Work w/dalley to get some help from pulp_rpm on this
Upcoming feature planning
- Ostree (jjeffers, talk w/ dkliban)
- Alternate Content-Sources (ACS) (iballou, talk w/ ppicka)

QE

HTB
next-y bz verification
next-z bz verification
Automation fixes

ggainey · September 15, 2021, 5:12pm

June 23, 2021

Overview

Katello Schedule
- 4.1 branching ~May 2021
  - pulpcore 3.11
- 4.2 branching ~August 2021
  - pulpcore 3.14?
- 4.3 branching ~Nov 2021

Pulp

Pulpcore

3.14 still scheduled for Jun 29
- https://github.com/pulp/community/discussions/22
- Max-retries
  - Story #8881: As a user, Pulp will retry downloads by default 3 times and I can configure this value on my Remote - Pulp
- New Tasking system will be on by default
  - Task #8948: Switch default tasking system and adjust documentation - Pulp
- old-tasking-system fixes have been merged, will resume investigation next Wednesday post 3.14 release
  - Issue #8912: [EPIC] Issues with the traditional tasking system - Pulp
- anaconda/range-header issue still in progress, will not be in 3.14
  - Issue #8865: Range requests for on demand content return the full file (Kickstarts fail for on_demand repos) - Pulp
- New tasking system performance analysis
  - A new Tale - Pulp gets queueless distributed task workers | software repository management
- New-tasking-system live interview Tues next
  - Ping bmbouter if need invite, recording will be taken

RPM

3.13.0 released
- It has a critical bug. 3.13.1 coming very soon (possibly today)
Caught by RHUI! Yay for collaboration

Migration

0.12 released, not interesting to katello
Deb fix to be backported to 0.11 Backport #8945: Backport 8928 to 0.11 - Migration Plugin - Pulp
Looking into potential errata issue now, HTB related

Ansible

3.15 features
- Token support
- Content removal
  - Reclaim disk space for a list of repositories: Story #8459: As a user I want to reclaim disk space for a list of repositories - Pulp
  - Purge individual orphans: Story #8658: As a user I want to purge individual orphan or list of orphans - Pulp
  - Limit number of repo versions: Story #8368: Limit number of old repository versions. - Pulp

Pulp Container

import/export in progress
2.6.0 compatibility release scheduled for 3.14
Work in progress in rbac area

Pulp CLI

Talking w/ build-team on packaging
If katello would like to see it, help us raise the priority!

Pulp OSTree

Started planning - Epic - Story #8924: As a user, I can manage OSTree repositories - OSTree Support - Pulp
dkliban/jjeffers will coordinate to start planning

Katello

Python creation merged!
Debugging migration issues from HTB
3.13 upgrade
- Filed: Issue #8952: modular rpms have is_modular set to 'false' - RPM Support - Pulp (thanks grant & daniel for the investigation and quick fix!)
- Filed: Issue #8950: rubygem bindings include python client? - Pulp
- Filed: Issue #8941: media_type list filter doesn't work with ruby bindings, expects array which api doesn't support - Container Support - Pulp
Iballou has ACS questions about RHUI - who should he talk to?
- mminar@ on RHUI, include ppicka in conversations
Discussion around pulp/bindings/downstream versions
- Makes prob-determination harder
- What about when you need to talk to multiple-versions-of-pulp from same katello?
- Cli doesn’t use/rely-on bindings due to “5 versions back” policy/goal
- How can pulp improve compatibility-testing for bindings?
  - Move all tests to use bindings, not straight-to-REST
  - Still need moar coverage

QE

QE test-automation is now branched to downstream-next
UI interface issues being resolved

ggainey · September 15, 2021, 5:12pm

June 30, 2021

Overview

Katello Schedule
- 4.1 branching ~May 2021
  - pulpcore 3.11
- 4.2 branching ~August 2021
  - pulpcore 3.14?
- 4.3 branching ~Nov 2021

Pulp

Pulpcore (skip for bmbouter)
- Bindings bug fix will change openapi/bindings names
  - https://github.com/pulp/pulpcore/pull/1438
    - https://github.com/pulp/pulpcore/pull/1438/files#diff-183a38cd633858d857b81af43f3756078fd53926010d3b417637c758a5e87e5e
    - Affects exporter-api
    - Will be in 3.14
- 3.14 will be releasing tomorrow, July 1
  - All compatible plugins should be available July 1 also
  - New tasking system is available by default
    - Overview video Pulp Tasking System Information Session - YouTube
  - Content serving caching speedup
    - Should we enable by default?
- Pulpcore 3.15 will switch to Django 3.2-LTS
  - Probably not difficult to backport
  - Katello likely should accept this back into early versions given that django 2.2 support drops in April 2022
  - Will prob be pulp/build-team work
    - Current thinking is no-code-changes-for-katello
  - Need to double-check on python-3.6 issue, pulp team on the hook for that
- Next Pulp-certguard release now supports bundled certs
  - Issue #8783: CA certificate does not handle a CA bundle or chain file correctly - CertGuard - Pulp
  - 1.4.0 has been released
- Question: should Pulp continue to try to “fix” the pre-3.14 tasking system
  - Prelim. answer from mmccune: no?
- Discussion around logging-bz
  - 1961508 – pulp-3 to log access logs to another file than /var/log/messages
  - Please add feedback to the BZ
RPM
- Fixes in-progress for 3.13.3 (or 3.14.0)
  - “Sync mirrorlist url fails” Issue #8981: Syncing mirrolist based remote fails - RPM Support - Pulp
- Pulp 2 error when syncing Nvidia repo
  - https://bugzilla.redhat.com/show_bug.cgi?id=1920511
Migration
- HTB investigations
Ansible (skip for bmbouter)
- No updates
Pulp Container
- 2.7.0 release compat with pulpcore 3.14
- Bindings issue solved
- Discussion around more backports for 2.5 (supporting core/3.11)
  - Decision: backport/release not needed
Pulp CLI
- 0.10.0 released, lots of fixes/additions
- https://github.com/pulp/pulp-cli/milestone/8?closed=1
Pulp OSTree
- Still planning - Story #892: Redesign of Uploads API - Pulp

Katello

Debugging HTB issues
- Capsule-sync 3-to-2 seems to miss a bunch of RPMs?
  - Maybe related to Issue #8622: CentOS 8 BaseOS .treeinfo's [general] section says AppStream instead of BaseOS - RPM Support - Pulp ??
Prepping for 3.14 upgrade
Started discussing ACS with Martin from RHUI
- ACS-for-Satellite might be getting pushed back
User-stories for ostree-in-katello Coming Soon
- jjeffers/dkliban to collab
Make sure iballou knows as soon as 3.14 is out

QE

HTB
next-Y bz verification
New test automation & fixes for next-Y

ggainey · September 15, 2021, 5:12pm

July 7, 2021

Overview

Katello Schedule
- 4.1 branching ~May 2021
  - pulpcore 3.14
  - pulp_rpm 3.13
- 4.2 branching ~August 2021
- 4.3 branching ~Nov 2021

Pulp

Pulpcore

core/3.14 released
file/1.8.1 released
Need backport requests filed for 3.14.1
- Bmbouter to work w/ Ian?
- 2 issues, already filed
- Backport requests:
  - Backport #9025: Backport fix for 9008 to 3.14.z -- PulpcoreClient::ExportersPulpExportsApi list results in uninitialized constant PulpcoreClient::AnyType error - Pulp
  - Backport #9015: Backport fix for 9003 to 3.14.z - unable to sync from repos on the filesystem - Pulp
- Made a milestone but it’s empty
  - 3.14.1 - Pulp
Waiting on feedback on the request to upgrade django and python in Katello
- Wanting to upgrade before Things go EOL (December-ish) timeframe
core/3.15 and CI and CentOS8 and python-3.7 vs python-3.8?
- Do we just need to move core/3.15 min-python to python/3.8?
- Needs discussion
- daviddavis to talk w/ build-team RE options (SCL, build-from-src, etc)
- Upstream-bandersnatch on 3.8+ only
After feedback, we are not continuing on investigating the tasking systems
- Issue #8912: [EPIC] Issues with the traditional tasking system - Pulp
Is this needed?
- Backport #8908: Backport 8708 to pulpcore 3.9 - Pulp
Range request fix is starting
- Issue #8865: Range requests for on demand content return the full file (Kickstarts fail for on_demand repos) - Pulp
- Should this go into 3.14.1 also?

RPM

Several bugfixes in progress, probably a 3.13.3 soon (along with 3.14.1)
- Katello backport request relating to Pulpcore 3.14 upgrade: Backport #9023: Backport fix for 9021 to pulp_rpm 3.13.z - errors when syncing from a local file:// source - RPM Support - Pulp
Would appreciate some guidance on the most important bugs to focus on
- Katello-tag and High, please

Migration

Ansible

Implementing a resolution for the deprecated metadata, finalizing import/export
- Story #8205: Add AnsibleCollectionDeprecated resource to export/import - Ansible Plugin - Pulp

Pulp Container

container/2.7.0 released

Pulp CLI

cli/0.10.1 released
Evgeni++ - RPMs exist!

Katello

QE

HTB
Next-Y bz verification
Next-Z bz verification
Pulp 3.14 discussion

ggainey · September 15, 2021, 5:10pm

July 14, 2021

Overview

Katello Schedule
- 4.1 branching ~May 2021
  - pulpcore 3.11
- 4.2 branching ~August 2021
  - pulpcore 3.14
- 4.3 branching ~Nov 2021

Pulp

Pulpcore

3.14.2 released
- 3.14.2 - Pulp
- Range-requests, proxy-auth
Is this needed for Satellite 6.9?
- Issue #8779: Task started on removed worker - Pulp
- Seems to be only post-migration work in HTB
- There exists two workarounds in 2to3 if needed
  - cancel the tasks, disable parallel publishes
  - Jsherrill wants to review 2to3 task-workflow
    - Cancel all tasks in 2to3 task-group
- ggainey: update issue w/ discussion ^^

RPM

3.13.3 (or maybe 3.14.0) coming this week with a few bugfixes
“relative path problem” under investigation that prevented syncing a couple of (usually poorly-constructed / deviant) repos
Meeting with EXD and RHEL about “signed repo metadata”
- Lots of pressure on this topic, impacting RHEL’s compliance
- Pulp3 (and Pulp2) already knows how to do this - not exposed at katello (yet)
- Mirroring signed-upstream-metadata will become a thing
- Managed-repos (like, say, CVs) will require katello to understand how to work with Pulp3’s signing workflow
- Requests for signature-verification as well

Migration

Ansible

No updates

Pulp Container

2.7.1 will have the proxy-auth fix - #coming-soon

Pulp CLI

OSTree

Planning to start implementing in the next 2 weeks. Will have a more concrete timeline after sprint planning tomorrow.
Sync content from cloud.redhat.com - maaaaaybe?
- Discussions in progress
Pulp-to-pulp will def be implemented (for capsules)

Katello

Filed: Issue #8750: Deadlock on rpm repository pulp2pulp sync - Pulp (deadlock during pulp-to-pulp sync, unknown severity yet, likely low/medium)
- Hit in CI, EL8, 3.14 upgraded from 3.11
Next snap (today or tomorrow) for QE will contain core/3.14
3.14 upgrade - bindings import/export name-changes
- Had to happen to fix a bug, sorry

QE

New test automation & fixes for next-Y

ggainey · September 15, 2021, 5:11pm

July 21, 2021

Overview

Katello Schedule
- 4.1 branching ~May 2021
  - pulpcore 3.11
- 4.2 branching ~August 2021
  - pulpcore 3.14?
- 4.3 branching ~Nov 2021

Pulp

Pulpcore

3.14.3 being released today/tomorrow
- 3.14.3 - Pulp
Backporting the tasks assigned to dead workers fix to 3.7
- Backport #9118: Backport #8779 "Task started on removed worker" fix to 3.7 - Pulp
3.15 tentative release Aug 2, starting weekly go/no-go reviews
- Alternate Content Source implementation still ongoing
Davis coordinating w/ ppicka out
3.15 dropping Python 3.6 support moving to 3.8 (already done on master)
- Not maintaining 3.6 CI anymore so backport releases will only receive 3.8 testing
3.15 moving to Django 3.2 (already done on master)
- Small code changes in Pulp needed
Revisit Katello upgrading Django 3.2 and Python 3.8 in a z-stream in Sept
- Apr-2022 is when Django-2.2LTS EOLs
- There is a more general issue around support-lifecycle for other (nonpulp) components

RPM

“relative path problem” under investigation that prevented syncing a couple of (usually poorly-constructed / deviant) repos 8133 7208
- Hopefully coming in time for 3.14
- #soon
3.14.0 coming this week (for real this time) with several major bugfixes (only bugfixes, no new features)
- Fixes a corrupt data problem that only exposed itself on pre-3.8 Python (!!)
- Fixes
  - metadata mirroring
  - Signing service
  - Proxy settings forwarding
  - Better behavior when copying package groups
  - SUSE repo syncing
- 3.14 will be replacing 3.13 as the pulp_rpm release stream for pulpcore 3.14 compatibility
- Test-issues under investigation, might delay us a day

Migration

No updates

Ansible

No updates
SAT-1562 - does this require token-auth in 6.10?
- Prob just unprotected syncs?
- When does token-auth-req come into katello? - PM discussion ongoing/needed

Pulp Container

2.7.1 released today
- Issue #8941: media_type list filter doesn't work with ruby bindings, expects array which api doesn't support - Container Support - Pulp - media_type list filter fix
- “Use proxy auth credentials during sync”
Drf token auth will be added to pulp-container - in the planning phase. Also needs more PM discussion

Pulp CLI

Doc work incoming, may ping katello/QE for feedback
ACS work for katello will be documented to use pulp-cli

OSTree

Started implementation of upload use case.

Katello

Updated to use mirrored metadata for library repo syncs & capsule syncs
Filed Issue #9111: can't sync rhel 8.4 baseos kickstart repo unless md5 (and sha1) is enabled - RPM Support - Pulp (already fixed!)
Filed Story #9099: task created_resources provides hrefs, but no guidance on how to determine the type from the href - Pulp
- bmbouter/jsherrill to have a quick discussion
Filed Issue #9098: Mirror-mode sync is not creating publications in the proper way for repos with distribution trees - RPM Support - Pulp (PR open)
We’re heavily focused on bugfixing atm
“CEE Enablement” - can we push it later? Who is/are the responsible group(s)?
- pulp/katello will collab closely on collateral/presentation

QE

Testing on pulp 3.14 has started
- Pulp 3.14 test scenarios are available for review
Next-Release bz verification
Next-Release feature area
Automation
Ltran out next week, swadely to report

ggainey · September 15, 2021, 5:11pm

July 28, 2021

Pulp

Pulpcore

Bmbouter out August 2-29, ttereshc taking over as lead
3.15 pushed back to August 10 (tentatively)
- 3.15.0 - Pulp
- Dkliban release lead
3.7.7 has been released
- pulpcore · PyPI
- Contains just the tasking system assignment to dead worker backport
3.14.3 is released
- 3.14.3 - Pulp
How to support python 3.6 for z-stream releases
- Dev boxes can use the older version of the installer, e.g. 3.14
- Installer needs a branch created to support this
- The CI can be configured to using Python 3.6 for older branches
Alternate Content Source implementation still ongoing
- Current plan to deliver in 3.16
Variety of small-but-important fixes for Django 3.2 compatibility
- Handling deprecations
- Some queryset evaluation differences
- Postgresql connection reuse within coroutines is different
- Will be in core/3.15
- django/3.2 means no deb-10 support, just deb-11
SELinux fix/investigation coordinating by @mikedep333
Pulp2-selinux vs pulp3-selinux collision issue
- https://github.com/pulp/pulpcore-selinux/pull/36

RPM

3.14.0 is out
- Changelog — Pulp RPM Support 3.14.2 documentation
“Relative path problem” has PRs open, should be resolved in the next releases of pulp_rpm + pulpcore
- https://github.com/pulp/pulp_rpm/pull/2050
- https://github.com/pulp/pulpcore/pull/1513
3.13.z incorrect metadata problems are fixed in 3.14.0, repair script is ready and tested by Katello, instructions provided to users
- Issue #9107: filelists and changelog metadata is not parsed properly - Pulp saves incorrect filelists and changelog metadata and generates incorrect metadata - RPM Support - Pulp
More bugs being fixed, will start moving on to bugzillas
- Kickstart metadata
- dep-resolution
Working on 6.10 bugs affecting Pulp3 to Pulp2 sync

Migration

0.11.z release planned (mostly deb-related backports)
Testing bugfixes for RPM for pulp3 to pulp2

Ansible

No updates

Pulp Container

Working on the pulpcore 3.15 compatibility in preparation for 2.8 release
import/export got merged and it enables import/export of synced repos

OSTree

Lubos is implementing the upload use case. Story #8918: As a user, I can create a new repository by uploading an archive with OSTree commits - OSTree Support - Pulp
Will start working on CI next week
Slated for katello-4.3; katello might be able to start looking at it in a month(ish)

Pulp CLI

[david] Will look into ACS support after 3.16

Katello

Semi urgent issue found as part of pulpcore 3.14.3: Issue #9146: sync of file:// repos tries to move files from /var/lib/pulp/sync_imports/ - Pulp (files removed from ‘import/file:// sync dir’)
Further python (& generic) support being added (upload, unit removal, cv publish)
Ian tested the filelist/changelog repair script, thanks ian and dalley!
Odd edge case when migrating-while-publishing-CV
- Talk to ttereshc

QE

Lots of test-automation-writing for new features
QE has a selinux-enforcing test
- https://github.com/SatelliteQE/robottelo/blob/fe01c9051f625c7e72504ced2b2c8fbfb63c04b8/tests/foreman/sys/test_pulp3_filesystem.py#L32

Build Team

3.14.3 rpms signed, re-running pipelines
- Will work w/ jsherrill on pipeline failure
- https://github.com/theforeman/pulpcore-packaging/pull/198

ggainey · September 15, 2021, 5:11pm

August 4, 2021

Pulp

Pulpcore

3.15 on track for August 10
- 3.15.0 - Pulp
- Requires Python 3.8+ and django 3.2
- Foreman installer will need to generate db encryption key
  - Feature: Story #8192: Add code to pulpcore that uses the db key to encrypt fields - Pulp
    - Davis to open foreman issue
      - Feature #33210: Generate a db encryption key when installing pulp - Installer - Foreman
- Reclaim disk space feature
  - Story #8459: As a user I want to reclaim disk space for a list of repositories - Pulp
  - Plugins need to opt-in, per-content-type
- Non-blocking orphan clean up as well
  - Task #8824: Convert orphan cleanup into a non-blocking task - Pulp
SELinux BZ for 6.10 is in progress
Issue #9146: sync of file:// repos tries to move files from /var/lib/pulp/sync_imports/ - Pulp import from /var/lib/pulp/ moving files
- Fixed and will be backported to 3.14.4
Relative path problem, should be backported to 3.14 and 3.7

RPM

New release this week (maybe today), fixes for:
- Downloading from Amazon Linux repositories
- RHEL Pulp 3 to Pulp 2 sync
- Capable of sync poorly-formed repos
- Distribution tree metadata fixes
  - Some issues remain: Issue #9208: Published .treeinfo metadata not matching expectations (remaining issues) - RPM Support - Pulp
  - Priority? (justin - normal since we aren’t aware of any implications for users)

Migration

0.11.3 released
- Modular-errata fix included - should get into next-z
Investigating issues

Ansible

Galaxy v3 Deprecation migration
- Only handle latest repo versions?
- Justin says “This is fine.”

Pulp Container

2.8.0 is out (core/3.14, 3.15 compatible)
- Reclaim-disk-space is enabled (under 3.15)

OSTree

Finishing up the upload story
- Tests are written
- Finishing up CI setup
Will work on sync story next

Pulp CLI

No updates

Katello

Pulp 2 metadata repo cleanup script release upstream, going into N/N-1 downstream, old-version-keeping will be turned off in N
Still focused on downstream bug fixing

QE

Next-Y testing
Feature testing
Automation failure fixes
Highlight upgrade blockers:
- CU DB: Blocked (cust DB is only GREEN)
  - 1962462 – pulp3: Sat6.9 with pulpcore tasking system assigns task to a removed worker - POST
    - backport old-tasking-fix
  - 1989282 – In satellite upgrade, content switchover failed with status 1 Triaged - NEW - Reported yesterday
    - poss test-db-corruption
Normal upgrade: Completely Blocked
- Satellite upgrade blocked in Next-Y Snap11 due to new reported issues:
  - 1989378 – Pulp migration failed with " Katello::Errors::Pulp3Error: out of memory" Triaged - NEW - Reported yesterday
    - OOM
Capsule Upgrade RED

Build Team

Python3/EL8 upstream prep

ggainey · August 11, 2021, 6:12pm

August 11, 2021

Pulp

Pulpcore
- 3.14.4 released yesterday
  - Relative path problem fix (needs pulp_rpm 3.14.1 release)
    - Issue #8133: Same package from different repos yields error on sync - Pulp
    - https://bugzilla.redhat.com/show_bug.cgi?id=1985958
  - No longer tries to move files from sync_imports/
    - Issue #9146: sync of file:// repos tries to move files from /var/lib/pulp/sync_imports/ - Pulp
    - 1986996 – sync of file:// repos tries to move files from /var/lib/pulp/sync_imports/
- 3.7.z is planned to address relative path problem at migration time
  - Will require matching pulp_rpm backport
- 3.15 is postponed due to Django 3.2 compatibility concern (too many db connections being open)
  - Fix/optimization is on the way, but changes won’t be backported upstream
  - 3.14 will still work with Django 3.2 but it’s no longer advised to do so
  - Downstream will be able to backport the fixes if they choose to upgrade to Django 3.2 (likely would require backports for content plugins)
- Shared task resources will be in 3.16 (Satellite-Next or a downstream backport)
  - Poss into core/3.15 but be disabled
    - Needs hardening before turning it on for everyone
  - Plugins need code changes to support the feature
  - Affected workflow - concurrent copy from the same source repos
  - Story #9148: As a plugin writer i want to be able to share task resources for reading - Pulp
  - 1986356 – pulp3: independent CV publishes with same source repos content copy not concurrent
    - Currently aligned w/ Next-Y - is it really a must-have?
    - Ttereshc will take ques to triage-group
- PulpCon planning
  - Happening in early November
  - Think about topics you’d like to see (or give!)
RPM
- 3.14.1 coming today (hopefully)
  - Fix for syncing from Amazon Linux repos
  - Fixes for .treeinfo metadata
  - Relative path problem fix
  - Pulp 3 → Pulp 2 sync issue fixed
- No feature work, only bugfixes
- Depsolving BZs
  - 1934545 – pulp3: Content View publish with filters and depsolve=true generates assert failures
  - https://bugzilla.redhat.com/show_bug.cgi?id=1965942
- Issue #9230: Syncing kickstart trees with mirror=true does not properly include images in publication - RPM Support - Pulp
- Issue #9224: Pulp 3.14 - can't sync a repository because of a checksum - RPM Support - Pulp
Migration
- No updates apart from helping QE to investigate issues (upgrade testing)
Ansible
- No updates
Pulp Container
- 1981225 – Unable to sync docker images
- Needs eyes from Katello - may not be Pulp?
- Jsherrill to triage/re-assign as needed
OSTree
- Merged upload use case
- Starting work on sync workflow
- First-release “soon” (beta)
- dkliban/mcorr to work on fedoramagazine article on this work
Pulp CLI
- 0.11.0 release (see changelog)
Pulp discourse
- Announcement: Moving to Pulp Community Discourse · Discussion #86 · pulp/community · GitHub
- https://discourse.pulpproject.org/

Katello

Katello 4.2.rc1 in progress (with pulpcore 3.14)
Continued downstream bugfixing
Katello-upstream users looking fwd to delayed core/3.14 EL8 RPMs

QE

Pulp 3.14 Testing:
- Test results: Mostly Green!
- Pending issues:
  - Kickstarter sync to external capsule: https://bugzilla.redhat.com/show_bug.cgi?id=1992329, 1985122 – pulp3: sync of a file repo to an external capsule fails
  - Not all files are uploaded to a repo: 1985303 – Not all files are uploaded to a repository, no error is shown
Next-Z bz verification
- Ensure migration plugin runs in FIPS mode
  - Failed: 1929347 – pulp3: Ensure migration plugin runs in FIPS mode and respects the ALLOWED_CONTENT_CHECKSUMS configuraiton
- Filed: publishing cv threw an error and marked successful with no error in /var/log/messages: 1992699 – Publishing a cv threw an error and marked successfull with no error in /var/log/messages on a FIPS enabled system
Next-Y verification
Automation failures

Build Team

No updates

ggainey · August 24, 2021, 6:15pm

[6 days post-mtg, noticed that I hadn’t posted the minutes - apologies all]

August 18, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.10
- 4.1 - May 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.2 branching ~August 2021
- 4.3 branching ~Nov 2021

Pulp

Pulpcore
- 3.15 release is postponed till next week
  - More work on db connections issue
- Issues with gunicorn
  - Premature cnx closing
  - Upload with concurrency 25+
  - Under investigation
- ACS progress
- FIPS fun : https://bugzilla.redhat.com/show_bug.cgi?id=1995175
RPM
- Focusing on
  - Issues with attached BZs (depsolving specifically at the moment)
  - Distribution tree issues
- Upstream discussions with DNF team
Migration
- ‘NoneType’ object has no attribute ‘url’ when migrating distribution tree content
  - Issue #8862: ‘NoneType’ object has no attribute ‘url’ when migrating distribution tree content - Migration Plugin - Pulp
- An issue with the incomplete publication from James
  - 1982542 – RefreshDistribution step fails with "publication":["Invalid hyperlink - Object does not exist."]
Ansible
- No updates
Pulp Container
- RBAC work in progress
- No other updates
OSTree
- Sync workflow continues - done #soon
- Needs testing w/ cloud.redhat.com
  - Will reach out to PM to arrange
Pulp CLI
- Working on ACS support

Katello

Continued bugfixing
- Remote update issue, now creating ‘test’ remotes
Alternate content source & Ostree planning continues
Ostree compatible with 3.15? - yes

QE

Pulp 3.14 Testing:
- Testing next-Y continues apace
- Pending issues:
  - Kickstarter sync to external capsule: 1992329 – Sync of a kickstart repo to the external capsule fails, 1985122 – pulp3: sync of a file repo to an external capsule fails
  - Not all files are uploaded to a repo: 1985303 – Not all files are uploaded to a repository, no error is shown
Dev investigating diff between snap 12 and snap 13 regarding export
next-Y bz verification
Automation failure fixes

Build Team

Prepping core/3.15 packaging work
Backports to core/3.7 rpm/3.11 may happen “soon” - Zach and Tanya to chat post-mtg

ggainey · August 25, 2021, 4:24pm

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.10
- 4.1 - May 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.2 branching ~August 2021
- 4.3 branching ~Nov 2021

Pulp

Pulpcore
- 3.7.8 is released
  - Relative path fix for the pulp-2to3-migration use case
- 3.14.5 is released
  - Content app fixes
  - Sync performance regression fix
  - On_demand to immediate switch and import/export
  - Changelog — Pulp Project 3.14.5 documentation
- 3.15.0 is planned for tomorrow
  - Django 3 db connections issue is resolved
- ACS is closed to being merged (pulp_file now, pulp_rpm #soon)
  - Katello prob won’t think about integrating for a few months
  - Maybe get some “light” katello-dev workflow testing, as kind of UAT?
  - Recheckin in 2 weeks
RPM
- Focusing on
  - Issues with attached BZs (depsolving specifically at the moment)
    - 1965942 – pulp3: Sat6.9 with pulp-3 raises invalid depsolving warnings when publishing a CV with filters - warnings RE missing dependencies (PR waiting merge)
    - 1965936 – Huge memory usage when using CV filters in Sat6.9 with pulp-3 - memory use bz (help/patch under test)
    - 1934545 – pulp3: Content View publish with filters and depsolve=true generates assert failures - assert during depsolving/logging
    - https://bugzilla.redhat.com/show_bug.cgi?id=1995232 - postgres-cpu/explain bz
- 3.11.2 is released
  - Fixes for the pulp-2to3-migration case
- 3.14.2 is released
  - Requires pulpcore 3.14.5
  - Fixes
    - for kickstart repo sync in mirror mode
    - Syncing identical content in parallel
- Katello-repos def need rpm/3.11-current
  - Zach will invoke build-team@
Migration
- On and off with the incomplete publication issue, can’t reproduce
Ansible
- 0.9.1 is released
  - Issue #9250: improper validation when passing json as requirements yaml - Ansible Plugin - Pulp
  - Issue #9220: content_ansible_collection_versions_create endpoint requires a lot of fields from inside uploaded artifact - Ansible Plugin - Pulp
- Bindings version - Update pulp-ansible by fao89 · Pull Request #9577 · Katello/katello · GitHub
  - VCR tests will tell us if there’s an issue w/ bindings
  - Needs work - katello/fao89 will collab on getting it addressed
Pulp Container
- No update
OSTree
- No update
Pulp CLI
- ACS support in progress
  - Will be ready the same time as pulpcore/file/rpm (in 1-2 weeks)
Pulp Python
- 3.4.1 Released
  - Bugfixes and sha256 content filter
  - Changelog — Pulp python Support 3.4.1 documentation

Katello

Filed Issue #9292: race condition syncing repositories with the same tags results in bad data within the database - Container Support - Pulp (container sync race condition, bz linked)

QE

Final push to review bugs for beta, still working on test automation for new features.
From katello: 2 test-databases have been broken/corrupted in similar ways
- Is there backup/restore issue?

Build Team

Thanks for beta requirements, going into snap today
Prepping for 3.15 as soon as available to package

ggainey · September 1, 2021, 7:09pm

September 01, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1 - May 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.2 branching ~August 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 branching ~Nov 2021

Pulp

Pulpcore
- Connection already closed issue in content app, aligned to 3.16
  - Issue #9276: Content app can have unusable/closed db connections - Pulp
  - If you run into is, please share a box with a reproducer
- Handling Content-Encoding better, in progress
  - Issue #9213: DNF gets wrong rpm download size, until package downloaded with curl, then works - Pulp
  - Maybe katello hack to download kickstart tree images with on_demand sync will no longer be needed
- Https proxy support, in progress
  - Story #9304: As a user, I can configure https proxy using https:// - Pulp
- 3.16 tentative release date - Sep-28
  - ACS
- https://github.com/pulp/community/ is archived, discussions are down
  - Actively trying out https://discourse.pulpproject.org/
RPM
- 1965936 – Huge memory usage when using CV filters in Sat6.9 with pulp-3 - memory use bz (help/patch under test)
- 1934545 – pulp3: Content View publish with filters and depsolve=true generates assert failures - assert during depsolving/logging
- https://bugzilla.redhat.com/show_bug.cgi?id=1995232 - postgres-cpu/explain bz
- Katello 4.1.2.1 - 404 error through content proxy due to incorrect location_href - Support - TheForeman
- 3.14.3 is released 31-AUG
Migration
- Incomplete publication fixed
- 0.11.4 released
- ‘NoneType’ object has no attribute ‘url’ when migrating distribution tree content
  - Issue #8862: ‘NoneType’ object has no attribute ‘url’ when migrating distribution tree content - Migration Plugin - Pulp
  - In progress
Ansible
- 0.10.0 is released (compatible with pulpcore 3.15)
  - Collection Deprecation is a content now
Pulp Container
- Issue #9292: race condition syncing repositories with the same tags results in bad data within the database - Container Support - Pulp - fixed and will be backported to 2.7.z
OSTree
- Upload and Sync are done. Will release a beta this week.
- Working on the repository version delete next.
- Trying to find out what Satellite users will need to do to sync from c.rh.c
Pulp CLI
- Options (e.g. --remote) will accept pulp href or name
  - Instead of having --remote and --remote-href
- 0.12 in release-planning

Katello

Still heavy bugfixing
Starting ostree soon - yes please
- Target sync/upload early
- Wait on delete - “coming soon”

QE

beta blocker testing
Automation failure fixes
Only one blocker for 3.14 testing left

Build Team

Ggainey to ask build-team to package newer libsolv for el7

goosemania · September 8, 2021, 7:26pm

September 8, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1 - May 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.2 branching ~August 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 branching ~Nov 2021

Pulp

Pulpcore
- https proxy support
  - Story #9304: As a user, I can configure https proxy using https:// - Pulp
  - 1993917 – Syncing repositories with https proxy set ends with warning Katello::Errors::Pulp3 Error Only http proxies are supported
- ACS progress
- 3.14.6 is out
  - Content-Encoding headers issue Issue #9213: DNF gets wrong rpm download size, until package downloaded with curl, then works - Pulp
  - FIPS issue which prevented workers to even start Issue #9249: ValueError: _blake2 is not available in FIPS mode - Pulp
- To upgrade to 3.15 you’ll need this resolved
  - Task #9355: Create django 3.2 patch for Katello that allows md5 to be used even on FIPS systems - Pulp
    - Planned to deliver patch by Sept 17th to @Zhenech
RPM
- Solution for the depsolving issues (both time, memory, and errors) likely in hand, needs some more evaluation and testing first, but there is cause for optimism
- Syncing from mirrorlists seems broken (or perhaps just Amazon Linux repos), investigating: Issue #9328: repository metadata not getting published for some repos synced from a mirror list - RPM Support - Pulp
- Additional memory consumption improvements will be coming soon, waiting on createrepo_c
Migration
- SELinux issues
- ‘NoneType’ object has no attribute ‘url’ when migrating distribution tree content
  - Issue #8862: ‘NoneType’ object has no attribute ‘url’ when migrating distribution tree content - Migration Plugin - Pulp
  - Patch tested by katello user
Ansible
- Pulp_ansible planning to sync repo from git repos
  - Story #9344: [EPIC] As a user, I can sync content from git - Ansible Plugin - Pulp
- Pulp_ansible planning to add signing (tickets not yet written)
- Pulp_ansible planning to have CLI start auto-wrapping Role content as Collections (tickets not yet written)
Pulp Container
- 2.8.1 is out with the tags sync race condition fix
OSTree
- Working on the beta release today
- Compatible with 3.15+
Pulp CLI
- No updates

Katello

Pulp 3.15 needed for ostree development, rpms not available for another 1.5 weeks
- Investigated deploying a pulp-installer installation of pulp3 alongside a katello vm, documented instructions here: Pulp 3 Integration - Katello - Foreman
Switch away from multi-copy api for content view publishing unless dep solving

QE

Still have a blocker for the upcoming release:
- 1998291 – Change Puma defaults to match recommendations from Puma documentation - not Pulp
next Z-release bz verification
next Y-release bz verification
Automation failures

Build Team

No updates

ggainey · September 15, 2021, 4:55pm

September 15, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1 - May 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.2 branching ~August 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 branching ~Nov 2021

Pulp

Pulpcore
- Working on a resolution to https://bugzilla.redhat.com/buglist.cgi?quicksearch=2003264, PR opened
- 3.16 planned for 28-SEP
  - On schedule
  - Will include ACS support
- Secure proxy work is ongoing
  - Coordination w/ aiohttp group ongoing, pretty successful so far
- Deadlock investigation getting much attention
RPM
- 3.14.4 releasing soon, maybe tomorrow. Fixes for:
  - Poor depsolving-enabled copy performance and memory use
    - 2003764 – CV-promotion with-filters causes pulp code to do repoclosure once per-rpm-copied, which is Very Nonperformant.
  - Fix a regression in mirrorlist support
  - Throw a sync error for rare repos that use an out-of-repo location_href if mirroring is enabled
- Temporarily shifting some attention back to Pulpcore as the issue count and severity has decreased
- Jsherrill to schedule a mtg to discuss mirrored-metadata-wrkflow “discoveries”
Migration
- 0.11.4 released
  - Backport #9372: Backport "‘NoneType’ object has no attribute ‘url’ when migrating distribution tree content" to 0.11.z - Migration Plugin - Pulp distribution tree migration fix
Ansible
- Proxy issue (TCP_DENIED/407) backported
  - 0.5.10 released
  - 0.7.4 to be released
  - Fully fixed on 0.10, partially fixed on 0.8 and 0.9
Pulp Container
- Investigation of concurrent-push issue continues
  - May affect any concurrent-content-upload?
OSTree
- Released alpha last week
  - 3.15-compatible
  - Katello will start looking into this #soon (jjeffers)
Pulp CLI
- ACS work ongoing
PulpCon!
- Incoming Nov-8 - would love to have presentations from katello/qe/build
- What would people like to see presentations on?
- PulpCon 2021 Call for Proposals

Katello

Probably need to discuss other aspects of 2003264 – clients connected to on_demand capsules will start getting 404s if the Satellite regenerates repository metadata in more detail with dralley, also related to signed metadata (mentioned above, will schedule a mtg)
James is going to start looking into ostree integration, discussions with pm are ongoing
- Workflow needs “a little bit of work”[™]

QE

Memory leak in puma that may impact beta:
- 1998259 – potential memory leak in puma
3.14 - new bz: 2004397 – Sync of a Kickstart repo to the N-1 capsule fails
- Pulp2 capsule syncing from pulp3 satellite
Finishing up Next-Z bz verification
- Verification takes A While due to nature of the bug
- https://bugzilla.redhat.com/show_bug.cgi?id=1962462
Test rcm-compose

Build Team

3.15 RPM progress
- Patch in-progress to support FIPS on Django-3.2
  - Task #9355: Create django 3.2 patch for Katello that allows md5 to be used even on FIPS systems - Pulp
- Need new version of openssl to relax pulpcore-3.15-req on python-crypto
- Build-team has some of its own issues to resolve
  - ‘Fun’ With SCL

ggainey · September 22, 2021, 4:17pm

September 22, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1 - May 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.2 branching ~August 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 branching ~Nov 2021

Pulp

Pulpcore
- Task group responses Story #9380: As a plugin writer, I have machinery that can return a task_group from viewsets - Pulp (3.16)
  - The ACS refresh endpoints will return task groups instead of the parent task (3.16)
  - Also updated the import create endpoint in 3.16
    - Task #9382: Update import code to return a task group - Pulp
- PR submitted for deadlock BZ/issue
  - 1997623 – Capsule sync fails with duplicate key and deadlock
  - Issue #8750: Deadlock on rpm repository pulp2pulp sync - Pulp
  - Arranging for stress-testing of 3.14-backport on a RHUI-4 setup
- Pulpcore 3.16 released moved back to Oct 5, at the latest we will release on Oct 12
- Discussing backporting to all releases in-between, e.g. everything on 3.14.z should go into 3.15.z. Is this important?
  - “Could reduce future problems” - katello
- FIPS patches provided to build team. Still waiting for openssl resolution upstream.
- Secure Proxy is going well
  - Finalizing aiohttp/pulp work #soon (by end of month?) into aiohttp-3.7.last
    - If needed earlier - ping bmbouter for patch
  - Aiohttp-3.8 dropped py-3.6 support
RPM
- One particular microsoft RPM repo for RHEL 8 is known to cause very severe issues with Pulp
  - Basically unsyncable for two different reasons, one known, one unknown, both related to memory
  - May involve patching Pulpcore
  - Issue #9399: sync error: invalid memory alloc request size - RPM Support - Pulp
  - Issue #9406: Trivial OOM on sync for a particular Microsoft repo - RPM Support - Pulp
- 3.14.4 Release should (finally) go out today or tomorrow, got delayed last week due to emergent issues
Migration
- 1993951 – RPM1008: Checksum type "sha256" is not available for all units in the repository
- Under investigation, might be RPM not Migration
Ansible
- No updates
- How far back does Pulp need to support ansible clients?
  - There needs to be some discussion around support-expectations downstream, soon-ish
Pulp Container
- Working to 3.16 compat release
- Gunicorn-issue-miniteam looking to get reproducer
OSTree
- No updates
Pulp CLI
- 0.12.0 releasing in the next week or two
  - With ACS support

Katello

Ostree integration ongoing
- Hit issue with sync (Issue #9431: Sync tasks fail with the latest pulpcore version - OSTree Support - Pulp)
Better support for mirrored metadata (preventing shooting of feet)

QE

No updates

Build Team

3.15 continues apace

ggainey · September 29, 2021, 4:18pm

September 29, 2021

Overview

Katello Schedule
- 3.18
  - pulpcore 3.7
  - pulp_rpm 3.11
- 4.1 - May 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.2 branching ~August 2021
  - pulpcore 3.14
  - pulp_rpm 3.14
- 4.3 branching ~Nov 2021

Pulp

Pulpcore
- 3.16 delayed to 12-OCT? – NO
  - Downstream dev-freeze coming in 1st week of November
  - Need to coordinate closely with build-team to go w/ this
  - Re-evaluation in progress - might go back to 5-OCT, stay tuned
- 3.14.7 incoming (deadlocks, remote-url-update, and other fixes)
- 3.15 behind on backports
  - Pulp/katello using 3.15, so might like updates?
  - AH wants 3.15 - so, make it happen
  - AI: someone in pulp needs to own backporting Things to 3.15 [dkliban]
- Secure proxy/aiohttp work
  - Bmbouter opening patch against downstream today
  - Prob evgeni to review
- Auth in URL bz 2007655 – Authorization repository causing invalid upstream url
  - We plan on not allowing credentials in the Remote URL.
  - Migration plugin will migrate URLs with auth as regular URLs and separate username and password fields on the remote
- Total counter increases during task execution 1994391 – Total steps for tasks increases
  - Closing wontfix
RPM
- 3.14.5 incoming
- One outstanding blocker issue (Auth in URL): Issue #9464: Regression: Syncing username+password authenticated remotes fails - RPM Support - Pulp
- Investigating reports that Kickstart doesn’t work - but it seems like these users are using pulp_rpm 3.10, and that it might work fine on 3.14. Would appreciate confirmation of this
  - Unable to kickstart Rocky Linux 8 due to .treeinfo - #4 by llandin - Support - TheForeman
  - Foreman failed to download metadata for repo 'AppStream': cannot download repomd.xml - Support - TheForeman
  - AI: Who can help dralley with testing? Alternately - ask users to upgrade-and-retest [jsherrill]
Migration
- 1993951 – RPM1008: Checksum type "sha256" is not available for all units in the repository capsule sync failure (pulp_rpm 3.14.5)
- https://bugzilla.redhat.com/show_bug.cgi?id=2006307 investigation of migration failure due to oom
- https://bugzilla.redhat.com/show_bug.cgi?id=2003888 documenttoolarge during migration (looking for reproducer for potential fix)
Ansible
- Git-sync in development (future work)
- Mdellweg joining the miniteam
Pulp Container
- Container-signing ongoing (future work)
OSTree
- Fixed a katello bug, patch tested
- Needs a release
Pulp CLI
- Wrapping up pulp_rpm ACS commands for 0.12.0
- 0.12 coming Real Soon Now
Thanks David Davis!

Katello

Ostree work continues
Bringing this up for discussion: Backport #9446: Backport #8133 "Same package from different repos yields error on sync" to 3.9.z - Pulp
- Was blocking 4.0/1/2
- Katello fwd-ported Things from 3.7 to 3.9
- Comes back to “should pulp backports go to All Releases”

QE

Next-Y bzs
Waiting on Next-Z build
Automation failures & fixes
3.14 bug verification happening/ed
- 2004397 – Sync of a Kickstart repo to the N-1 capsule fails

Build Team

Next-Y Stuff is Crazy-Making
Next-Z - not till next week
3.15 and SCL - Evgeni working on it, some dep-work still in flight