Pulpcore Meeting Minutes

ipanova · January 26, 2022, 9:32am

January 25th, 2022

3.18 release - Feb 8 (tentative)
- we have 2 features merged
- any blockers/must-haves?
  - redirect content guard
    - https://github.com/pulp/pulpcore/issues/2151
- plan to schedule go/no-go on Feb 1
  - scheduled Feb 2 based on availability
Need to provide the ability to reroot
- https://github.com/pulp/pulpcore/issues/2148
- https://github.com/pulp/pulpcore/blob/main/pulpcore/constants.py#L47
High-priority bugs that need fixing ASAP:
- touch() deadlock
  - 2021406 – syncing tens of repos to capsule can cause deadlock: while updating tuple (...) in relation "core_content"
- import/export unique-constraint failure
- taskreservedresource migration
  - 2031154 – After upgrading to Satellite 6.10, Repository sync randomly fails if a ReservedResource exists in core_taskreservedresource table of pulpcore DB.
Should / Can we provide (general) manage command to help admins to migrate existing permission sets to existing roles?
- Idea: manage command to report “unmigrated” permissions and let the admin assign the roles via api
- Create the command in pulp_container codebase
Downloader Refactor
- started out obvious
- grew out of scope
- we will slow role and think about separation of concerns again
- nothing urgent; please don’t hold any other work for it

x9c4 · February 8, 2022, 3:13pm

February 1, 2022

Discuss memory use proposal Reducing Memory Usage during Sync
Github issues process adjustment suggestions
- Github Issues: Process and Procedures - HackMD
Changelog cherry-picking problem
- No longer using backport issues. Changelog commit removes files from the CHANGES directory. These files need to be present on ‘main’ branch until the changes are released in the latest release.
- We need to figure out Something - because this is the last, larget, manual step blocking releases
- Maybe each previous-branch-changelog is a separate file?
  - How?!?

February 8, 2022

periodic tasks request from galaxy_ng for them to run a galaxy_ng task periodically
- https://github.com/pulp/pulpcore/issues/2204
- [bmbouter] my proposal is to move the PeriodicTasks work to its own PR.
- proposal: plugin writer only feature, not user-facing
- AI: all discuss on issue and review next week
plugin removal test
- as a separate job which runs similar to the upgrade test but only nightly?
- current PR which struggles to make it work as typical test
  - https://github.com/pulp/pulpcore/pull/1434/files#r800633658
Touch() bug fix - when can we get it tested?
- https://github.com/pulp/pulpcore/pull/2159/files
3.18 schedule discourse thread
- Pulpcore 3.18.0 release schedule

x9c4 · February 15, 2022, 3:42pm

February 15, 2022

rerooting update
- pulp-cli and pulpcore is working well
- pulp_installer to get some updates to its reverse proxy configs
- plan is to test in one of the jobs
review periodic tasks discussion (aka TaskSchedule)
- Request to post periodic metrics
- https://github.com/pulp/pulpcore/pull/2208
plugin removal
- blocked by CI weirdness around service names in ci container
- We fix the ci base container to forget all workarounds for the old tasking system

ipanova · March 1, 2022, 2:56pm

Pending AIs

[dkliban] work on changelog cherry-picking automation
[bmbouter] PoC of memory reduction in Stages API
[matthias] write a post on discourse to start download refactor SIG

March 1st, 2022

Mirrorlist / Metalink support in downloaders
- Old issue, still a good idea: https://github.com/pulp/pulpcore/issues/1815
- https://github.com/pulp/pulp_rpm/issues/2386
- Consider implementing it in the 3.20 / 3.21 timeframe along with downloader refactor?
- (mirrorlists) are a current source of a lot of hacks in the RPM plugin w/ lack of native support
Can we prioritize https://github.com/pulp/pulpcore/issues/2101 ?
- yes - ggainey has a branch already
- dkliban to keep ggainey on track
https://github.com/pulp/pulpcore/pull/2261 needs review, merge and backports
z-stream Releases for 3.17 && 3.18 maybe 3.16 needed.
bmbouter switching unit test runner to use pytest and the django-pytest plugin
- will involve rewriting or removing 7 or so unit tests on pulpcore
- going into plugin_template as a non-optional change. will advertise at open floor
An effort is underway to have a validating openAPI spec
- https://github.com/ansible/galaxy_ng/pull/1136

ipanova · March 8, 2022, 2:39pm

Pending AIs

[bmbouter] PoC of memory reduction in Stages API

March 8th

memory PoC in-progress
- one feature tries to log the relative memory usage of declarative versus everything else
- adds a new setting TASK_MEMORY_LIMIT
  - Attempts to reduce batching and put() calls at first stage
pending openapi schema changes to improve usability
- https://github.com/pulp/pulpcore/pull/2292
update on pytest unittest issue
- Unittest Problems - HackMD
we will need to release 3.15.5 once QE test the patches
- [daniel] will take care

ipanova · March 22, 2022, 2:09pm

Pending AIs

[bmbouter] open proposal issue to remove queues
[fabricio/dennis] ask katello if they want ruby bindings docs
[brian/matthias] open django prefetch bug

March 22

memory PoC continued!
- 27% reduction in memory by reducing queue count from 1000 to 1
  - should we remove the queues entirely?
- concern: deadlocking due first-stage waiting on the resolution of a future, or some even the pipeline is supposed to produce
  - more of a concern now that the pipeline is even less filled
proposal: removing django admin UI views
- Roles in pulpcore - #7 by bmbouter
- https://github.com/pulp/pulpcore/blob/901499a3566a940516664ae6ca042455efee0115/docs/configuration/settings.rst#L365
- https://github.com/pulp/pulpcore/issues/2374
- no concerns, it is in Tech preview also
bindings docs
- should we publish for both: ruby and python?
- https://github.com/pulp/plugin_template/pull/599
- let’s ask katello
mkdocs switch?
- should we?
- fabricio will show his PoC
weird “db connection already closed” CI failure https://github.com/pulp/pulpcore/pull/2357#issuecomment-1072333934
- maybe issue with a test, dkliban will take a look
absolutely weird RemoteArtifactSaver failure in pulp_container
- pulp_container sync duplicate key investigation - HackMD
- Disable Prefetch calls in RAS Stage by bmbouter · Pull Request #2375 · pulp/pulpcore · GitHub
Goal: Deduplicating tests between pulpcore and pulp_file

March 15th

memory PoC concluded
- adjusting batch size is effective at reducing memory
  - see the results
- dynamically adjusting it doesn’t keep it within a “target amount”
- it runs extremely slow 10x plus
- memory is never returned to the OS, the pages mostly just stay allocated and later
- new memory charting debugging feature added:
  - https://github.com/pulp/pulpcore/pull/2331
- proposal: to expose the batch size to users, the batch size is likely the single best adjustment to reduce memory
- we need each plugin to also look at their own first-stage code to see if it can be memory optimized
- proposal: https://github.com/pulp/pulpcore/issues/2069#issuecomment-1066853598
maintainability policy
- how far should we keep ensuring old branches are working?
- single container currently does latest 3 y-stream
- pulp-cli tests with 5 y-streams

dralley · March 31, 2022, 2:45pm

Pending AIs

[bmbouter] Go through possible close query Issues · pulp/pulpcore · GitHub

March 29

filed issue to remove the queues in 3.20
- https://github.com/pulp/pulpcore/issues/2377
Closing old/outdated/finished issues: Issues · pulp/pulpcore · GitHub
- Went through and marked issues I thought could be closed
- 86 issues!
- Options:
  - Have a meeting to go through some of the issues and discuss if they should be closed
  - Have everyone look through some of the issues on their own and close them if they feel confident
Unittest update
- Added ability to run additional tests in plugin_template
  - https://github.com/pulp/plugin_template/pull/602
- Using that to run the pulp_file tests
  - Run pulp_file functional tests too by bmbouter · Pull Request #2432 · pulp/pulpcore · GitHub
  - Fail in CI due to pulp_file tests not being a clean environment after pulpcore tests
- Goals:
  - Test independance, run single tests, or in any order
  - Cleanup all objects except orphans, a test that needs its orphans removed needs to run orphan cleanup at the beginning
  - move the pulpcore.tests.functional.api.using_plugin tests to pulp_file.tests.functional.api
- to run the “pulpcore” tests you have to run pulp_file.tests.functional and pulpcore.tests.functional to be sure
  - We thought better this than a formal import dependency on pulp_file
- switching unittest runner to pytest is also continuing
Do we want to do this eventually https://github.com/pulp/pulpcore/issues/1984? -YES
- used in pulp-ansible https://github.com/pulp/pulp_ansible/blob/main/pulp_ansible/app/tasks/collections.py#L398
- and similarly in content app https://github.com/pulp/pulpcore/blob/main/pulpcore/content/handler.py#L774

ipanova · April 5, 2022, 2:08pm

Pending AIs

[someone] open issue to audit and establish an upper bound for our requirements.txt deps

April 5th

How many releases do we want to keep supporting?
- Currently 5, in a few weeks, it will be 6
  - https://github.com/pulp/pulpcore/blob/main/template_config.yml#L35
  - switch to lower bound which is 3.14 [fao]
- We can’t drop 3.14
- Should we come up with some kind of LTS policy or continue with as-needed unofficial LTS?
- we should also audit and establish an upper bound for our requirements.txt deps
FYI here’s a proposal to auto-check that tests are always independant
- Making tests totally independent
- used to make these changes: Run pulp_file tests in pulpcore CI by bmbouter · Pull Request #2432 · pulp/pulpcore · GitHub
3.19 - last call, are there any PRs which need to merge today
- let’s push back a week
FYI - do not use gettext in log statements https://github.com/pulp/pulpcore/issues/2477
- [all] read and provide feedback
- [tanya] bring this to PMs attention

ipanova · April 13, 2022, 1:38pm

Pending AIs

[fabricio] audit and establish an upper bound for our requirements.txt deps

April 12th

3.19 release day
- No objections? Will wait for task lock PR and import/export PR to merge
functional test improvements
- accomplished
  - got pulp_file tests running along with the pulpcore tests
    - https://github.com/pulp/pulpcore/pull/2432
  - improved pulp_file tests to be independant
    - https://github.com/pulp/pulp_file/pull/674
    - https://github.com/pulp/pulp_file/pull/678
  - the --pulp-no-leftover-objs feature is ready for pulp-smash, still off by default
    - https://github.com/pulp/pulp-smash/pull/1285
- next steps
  - merge or revise --pulp-no-leftover-objs
    - https://github.com/pulp/pulp-smash/pull/1285
  - move all pulpcore.tests.functional.api.using_plugin tests to pulp_file
  - porting tests:
    - run in parallel
    - native pytest
    - use local fixtures (not fixtures.pulpproj.org)
  - adding list endpoints for remote and alternate content sources
dkliban is switching unittest runner to pytest
- https://github.com/pulp/pulpcore/pull/2498/ ← main branch
- https://github.com/dkliban/pulpcore/pull/63/ ← 3.14 branch
- https://github.com/pulp/plugin_template/pull/612
- Goal: to not have unittests require pulp-smash to be configured
  - Solution: need to move conftest.py into the functional folder “down a level”
  - Will have to do similarly for any plugin that already adopted conftest.py
Should we stop i8ln log messages?
- https://github.com/pulp/pulpcore/issues/2477
Mark issues as “good first issue”
- Would make it much easier to hand off issues to interns / people who want to get started with development

goosemania · April 19, 2022, 1:57pm

Pending AIs

[bmbouter] talk about plugin compatibility policy for 3.20+ on discourse

April 19th

test improvements update
- unittest switch to pytest is done on master and all branches back to 3.14
  - plugins switching in-progress (thanks @fao!)
- moving the using_plugin tests to pulp_file
  - https://github.com/pulp/pulpcore/pull/2535
  - https://github.com/pulp/pulp_file/pull/685
- next step: make pulpcore a pytest plugin so its fixtures can be versioned with it
  - involves moving all pulpcore pytest fixtures from pulp-smash to pulpcore
3.20 plugin API breaking changes
- 3.20.0 blockers Milestone · GitHub
- https://github.com/pulp/pulpcore/blob/f57bb9c8fc41831b5e5c392334c863035627b8c1/pulpcore/app/models/base.py#L74

ipanova · April 26, 2022, 1:58pm

Pending AIs

[matthias] find a slot for downloaders SIG kick off meeting (ipanova, dalley, bmbouter)
[grant] will open ticket with legals re:logo licensing

April 26th

Discussion around list APIs for Exporters and Remotes
- https://github.com/pulp/pulpcore/issues/2529
- https://github.com/pulp/pulpcore/issues/2530
- ok to proceed
Downloader SIG: need to schedule kick-off meeting
Release rotation - time to rotate?
- Brian will take care of 3.20 release
- what to do with the blogposts on the pulpproject.org? Probably we will drop those and have only discourse release announcements
- need to fix deps requirements( upper bound) on older releases and main branch before 3.20 release
decision on compatibility release?
- if 3.25 then it wil be 9-10 months roughly until next breaking change. 3.25 it is
Is the Pulp logo available under Creative Commons, CC0, CC-BY, or CC-BY-SA licenses?
- our logo is unlicensed
- we need to open a ticket with legals
Streaming support needs a little more work around content ranges and caching; test will probably a new ci scenario with sftp storage
triage security alerts?
- yes, add them next to the PR review ( from the next meeting)

dralley · May 6, 2022, 5:04am

May 3rd

Removing the “test keys” from the repo, recently flagged by security scanners and bmbouter was notified
- pulp_file/pulp_file/tests/functional/api/from_pulpcore/artifacts/x509 at main · pulp/pulp_file · GitHub
3.20 removal of MissingResource exception
- https://github.com/pulp/pulpcore/issues/1812#issuecomment-1112403081
- Removes `MissingResource` from the plugin API by bmbouter · Pull Request #2633 · pulp/pulpcore · GitHub
3.20 tentative date (or not ready yet?)
- (dalley) not ready yet IMO
Worth doing new Z-releases today?
- pulpcore/CHANGES at 3.18 · pulp/pulpcore · GitHub
- (no)

dralley · May 10, 2022, 2:29pm

May 10th

Look through remaining maybe-finished issues
- Issues · pulp/pulpcore · GitHub
z-releases
3.20 tentative release date? first go/no-go?
status of django 2.2 in pulpcore 3.7 / 3.14
- updating core/3.7 to last 2.2LTS, will require a change in (at least) matching pulp_container version
- need to evaluate other plugins (Required PR: ?)
  - file/rpm/container/ansible
  - https://github.com/pulp/pulpcore/pull/2651
- need a plan for who/when/testing
bmbouter and dkliban are updating pulp_file tests to use pytest, merging them with pulpcore

bmbouter · May 24, 2022, 5:20pm

May 24th

[ipanova] need to release 3.19.1 https://github.com/pulp/pulpcore/pull/2567
continued LTS discussion
- What is will be the next LTS?
- How aggressively should we be backporting to the LTS?
- How many LTS branches are supported concurrently?
telemetry PoC request for comment
- https://github.com/bmbouter/analytics.pulpproject.org
- https://github.com/pulp/pulpcore/pull/2118

bmbouter · June 7, 2022, 2:00pm

May 31st

[mdellweg] Serving artifact urls from the pulp_api for external verification (e.g. Virus scanning)
telemetry PoC request for review
- https://github.com/pulp/pulpcore/pull/2118
changing the tests run by default
- https://github.com/pulp/pulp-smash/pull/1293
LTS continued?
- wait on Dennis’ results

bmbouter · June 7, 2022, 2:00pm

Pending AIs

[dkliban] release pulpcore 3.19.1
[bmbouter] to propose majority use of nightly tests
[bmbouter] to highlight UI component creation on discourse

June 7th

telemetry pr in-review
- https://github.com/pulp/pulpcore/pull/2118
CI Update
- Functional tests to be run inside the container instead of the host
  - Installation is a nightmere today
  - More similar to dev-environment
Reducing CI runtime:
- moving most tests to nightly
Collaboration with david N on using galaxy NG generic React + Patternfly components for a future Pulp UI

ipanova · June 14, 2022, 1:37pm

Pending AIs

[dkliban] release pulpcore 3.19.1
[bmbouter] to propose majority use of nightly tests
[bmbouter] to highlight UI component creation on discourse
[matthias] figure out where sort is coming from

June 14th

[lmjachky] What to do next with https://github.com/pulp/pulpcore/pull/2713?
- exporting/importing repositories, distributions, and other types
- converting to different types of repositories (https://github.com/pulp/pulp_container/pull/826#issuecomment-1153680951)
telemetry update:
- dev systems will now report to the “dev analytics site” which gets summarized every 24 hours
  - once merged https://dev-analytics-pulpproject-org.pulpproject.workers.dev/
- going to make new chart for x.y components instead of x.y.z
- going to configure analytics.pulpproject.org to the workers with a “coming soon” page
ordering filter vs sort filter
- https://github.com/pulp/pulpcore/issues/2703#issuecomment-1154133290

bmbouter · July 12, 2022, 2:21pm

Pending AIs

[x9c4] Post on discourse about uninstalling pep8speaks
[x9c4] Investigate and post on Discourse about squashing migrations

July 12, 2022

proposal: remove pep8speaks. It’s redundant give flake8 is in the linter. currently enabled on pulpcore, pulp_file, plugin_template, and pulp_ansible
- https://github.com/organizations/pulp/settings/installations/39250
Time to squash migrations?
- it is almost 100 now (pulpcore)
- about 40 in ansible, container, rpm
- Migrations | Django documentation | Django
  - Can this be done without A) special requirements about the upgrade process and B) requiring plugins to squash at the same time?
Call for feedback: an experiment: marking 50% of tests as nightly only
- Marking majority of tests to only run nightly - #4 by bmbouter
- How are the tests chosen?
- What would be the revert policy?

bmbouter · July 19, 2022, 1:51pm

July 19, 2022

proposal to sync flake8 configs in the project
- https://github.com/pulp/plugin_template/pull/558
- no concerns heard at the meeting
Remove daviddavis from pypi projects
- Removal from PyPI projects
Change test assumption to assume all tests will be run on the same system pulp is installed on
- Functional tests will now run inside the container running Pulp - #5 by bmbouter
  - would include retiring pulp-smash
    - Yank pulp-smash from pypi?

bmbouter · August 9, 2022, 2:36pm

August 9th, 2022

[lmjachky - PTO] Please, take a look at the current implementation of the remotes removal. We decided to implement an option where a user needs to specify the “force=True” header to delete the remote if it is being used by existing content artifacts. Otherwise, an HTTP 406 error is returned by Pulp:
When to allow removal of remotes used for on-demand syncing? - #16 by lubosmj [ipanova] will comment
- we can’t use force due to it being a DELETE call
- let’s not create a force now because we’d have to make an entirely new endpoint
- let’s have a great error message returned
- we need tests
- [AI] @ipanova to represent ^ changes on the discourse and the PR reviews
PoC pulp-operator in Golang https://docs.google.com/document/d/1j6eanLgVz4HboEslzhjhOGKK4HF-jSSuC2wSC34eVpA/edit#heading=h.mdfz93aqubxv (pros and cons in the doc)