Pulp Installers Meeting Minutes

Dec 10 Agenda

  • 3.15 RPMs support landed
    • I did code fixes / workarounds for all the upgrades issue.
  • 3.16 RPMs support
    • Running into another upgrade issue from 3.3
    • Decided to actually drop support for upgrading from pre-3.8 this time
      • Need to announce on discourse.
  • Cluster support
    • Clarify what changes are still needed?
      • Identical token auth keys
        • galaxy_ng users probably need this, not just regular pulp_container users.
        • Even if their QA hasn’t caught it.
      • Molecule test for multiple containers
      • Fixing any bugs along the way
      • Prototyping with a load balancer or something
      • Documentation updates.
    • Work with someone from Ansible QE to test it out. With their HA environment.
  • Focus before cluster support or usability will be molecule performance
1 Like

Jan 12 Agenda

  • CI issue with pulp_rpm migrations not being idempotent
  • CI issue with locale_gen module missing from ansible-core 2.12 from geerlingguy role
    • ppicka looking into this
  • Will drop support for EL7 pip mode while doing performance improvements over next week

Jan 19 Agenda

  • Fixed 3 CI issues / actual bugs in pulp_installer, now encountering 2 more
    • Force merged since it makes the CI overall less buggy, and ppicka’s PR will be better fixed with it.
  • Noticed in Ansible docs that ansible-galaxy collection install can operate on a source directory. Builds and installs in 1 step.
  • CI issue with pulp_rpm migrations not being idempotent
    • This was fixed by pulp_rpm.
  • “pulp_repos” role improvement / bugfixes:
    • Eliminates need for workaround in example-use playbook.
    • Fixes actually adding EPEL on RHEL7 for the pulp_database/pulp_redis roles
    • Assigned to ppicka.
  • postgresql role issue

Jan 26 Agenda

  • Is waiting for a long time to merge now https://github.com/pulp/pulp_installer/pull/382
    • [mikedep333] I will re-review
  • Finished triaging Issue #9211: Vagrant devel installs have SELinux errors - Pulp by linking to it in 2 GH issues
    • Is this the appropriate way to migrate to GH issues?
  • Is tox actually overall beneficial for us?
    • It causes silly/laborious integration between CI and tox
    • How often do we test locally with all the python versions?
    • How often do we test locally with any different python version that then venv’s?
    • It is preventing me from calling molecule steps individually in the CI (which helps identify performance regressions.)
    • agreed: Remove tox
  • Drop idempotence from the longest running test, source-dynamic?
    • +1 for PRs, keep on nightly [fao]
      • +1 [mikedep333]
  • Design of https://github.com/pulp/pulp_installer/pull/830
    • Agreed: Use the cache_enabled variable to determine whether or not the pulp_redis role actually does anything. Wrap the tasks in a when condition. No more need for a separate role.

Feb 4 Agenda

  • What a week full of CI and installer breakages!
    • CentOS 8 discontinued was kind of foreseeable and avoidable
    • pip-tools being broken by pip 22 is a routine breakage
    • pulp_devel idempotency breakage is a routine brekage
    • release CI breakage (and nightly CI breakage) - not sure if we can test for this better. I just run yamllint.
  • Rocky Linux 8 support?
    • Ask on discourse
    • There’s also alma - can be supported via generic check for “RedHat” family and major version 8.
  • Release announcement for 3.17.2?
    • agreed: Do an announcement
  • GitHub generated release notes?
  • Reviewed https://github.com/pulp/pulp_installer/pull/870

Feb 9 Agenda

Feb 16 Agenda

  • We really do not need the “source-dynamic” test.
    • Nobody uses this exact combination. (devel role + dynamic include)
    • I cannot see why someone would need to use it.
    • It’s realistically tested by release-dynamic and source-static
    • Already done a day ago: Moved to nightly CI only
  • “packages-dynamic” or “release-dynamic” should be moved to nightly CI
    • release-dynamic always tests the latest version of pulpcore
    • packages-dynamic is quicker
    • Already moved a day ago
  • Status of CI refactoring (& performance improvements)
    • Made the 2 above changes, only on nightly CI now
      • PR CI tests dropped from 11 to 9
    • tox removed, python versions and everything now determined by GHA logic
    • ansible-core tests merged into our other test matrix and GHA logic
      • PR CI tests dropped from 9 to 7
    • Added more recent versions of ansible to the test matrix
    • We can see how long each molecule phase takes
    • Performance still not satisfactory.
      • Will look into dropping idempotency from the longest running test or 2.
      • Will look into more general performance improvements
      • Noticed that packages tests take like 10 fewer minutes than release tests
        • I’m trying to use (with molecule) an ansible plugin that reports the longest running tasks.
        • fao89: Try comparing only uses CentOS 7 and CentOS 8 in the release tests
      • packages-static takes 8 more minutes than packages-dynamic?
        • Dynamic should take longer, it runs pulp_common repeatedly.
        • maybe ansible interpreter version? Python version?
          • yes, 2.10 vs latest, 3.7 vs 3.9
        • fao89: See variables
          • differences are unix sockets and different webserver.

Feb 24 Agenda

  • Remaining CI performance improvement: reduce the time to run source-static
    • Proposal: Only run the devel role in nightly CI or for PRS when the devel role gets modified. For most PRs, just install from source.
      • Sub-proposal: Don’t create a new test, just run sed to not include pulp_devel for the regular PR CI test
      • Note: This still tests installing the latest, master branch, of pulp.
  • pulp_repos & redis role: https://github.com/pulp/pulp_installer/pull/901
    • Where does redis come from on EL7? EPEL7, for both centos and RHEL
    • Wrong dependency on pulp_common

Mar 3 Agenda

  • Galaxy Signing Service
    • Looks like it can be done in a few days to a week
  • HA
    • Need to refactor the dynamic molecule tests into multi-node tests
    • Need some adjustments on keys (encryption, token auth, webserver, …)

Mar 10 Agenda

  • onboarding new team members
    • read pulp 3 docs, particularly architecture
    • familiarity with task tracking
  • Status of galaxy signing service
    • going fairly well
    • wrapping around gpg commands is awkward but doable
    • this might help for some gpg commands: Ansible Galaxy
  • PR for repos role usage
  • merging of certs fix

Mar 16 Agenda

  • Is it possible to have 2 dynaconf settings files, one generated by installer, and a user-override?
  • Status of AH signing service
    • 1 remaining bug affecting release installs on EL7 only

Mar 23 Agenda

  • Most of the way done on making db encryption keys the same
    • Use of run_once / register / debug module to desgignate the primary host
    • How to handle non-identical keys already on the cluster?
    • Worker nodes need the key too, right?
  • Working partial implementation of a cluster for release-dynamic
  • Resolved multiple accidental dependencies of pulp_common on pulp_database
  • Need to do a release still for the AH signing service
    • SELinux updates just merged
  • galaxy-importer support in SELinux
    • Need test steps
    • agreed: reach out to AH
  • settings.local.py
    • agreed: Put header at the top of settings.py saying to modify settings.local.py instead
  • Settings pulp_user_home should set the entirety of /var/lib/pulp
    • There is a mismatch between certain variables and certain sub-variables of pulp_settings
    • We should look for ways to merge these variables into 1.
    • ppicka will address if he has time

Mar 30 Agenda

  • Inconsistency in default config for postgres/redis
    • Currently:
      • postgres binds to
      • postgres only permits connections from
      • redis binds to
    • What should we default to?
      • Accept connections / bind to
    • Can we configure postgres to allow the other hosts by IP?
      • we cannot guess correctly enough which is the correct IP address
  • Refactoring pulp_webserver to use the __pulp_database_config_real_sole_host instead of installing pulp-common
  • Current status of fixing non-identical database fields key
    • Will not do anything about non-identical existing keys but to throw a proper error message in the installer
    • Lots of effort involved in picking the correct host to run pulp_database_config.

Apr 6 Agenda

  • Welcome Humberto!
  • Figured out how to set the most global of variables
    • access once set with hostvars[‘localhost’][‘var_name’]
      • as opposed to the normal way “var_name” which can be also done as: hostvars[inventory_hostname][‘var_name’]
    • set with “set_fact:” and “delegate_facts: True” “delegate_to: localhost” “run_once: True”

Apr 13 Agenda

Apr 20 Agenda

  • Status of database fields key PR
    • Satoe messaged me privately with an error, which leads to a design question.
  • Should we continue to wait for the database fields key PR to release pulp_installer 3.19.0?
    • agreed: Release it beforehand if someone complains.
  • 3.18 RPMs

Apr 27 Agenda

  • Updated the db fields encryption keys PR to support replacing other hosts’ keys with 1 host’s key

May 9 Agenda

  • Status of 2 big cluster support PRs:
    • Cluster CI
    • pulp_webserver independence
    • dependent on the epel7 PR
  • Status of el7 support in packages?
    • el7 packages will still be built for pulpcore 3.18 RPMs.
  • docs examples not showing up properly Customizing Your Pulp Deployment - Pulp Installer
  • Suggestions on renaming / moving: Customizing Your Pulp Deployment - Pulp Installer
    • How about move to a new page called “cluster examples”?
      • Not technically accurate because 1 example is an external postgres/redis, but they could be postgres/redis clusters.
  • Suggestions on renaming / moving: Customizing Your Pulp Deployment - Pulp Installer
    • How about “specifying plugin versions” or “Installing specific plugin versions”
  • Running into an issue with my easy-approach-to-settings
    • Desire: settings like content_origin get set to “the 1 host that will run pulp-webserver”
    • Problem example: content_origin needs to be set for the pulp-api host, but pulp webserver gets deployed afterwards. I cannot determine “the 1 host that will run pulp-webserver”, only “the 1 host that has already run pulp-webserver”.
    • Possible solution: Special group names like pulp_webservers? A host could be in multiple groups. Users would still need to apply the correct roles list to each host.
    • agreed: follow up with pavel

May 11 Agenda

  • Remaining CentOS 9 work
    • vagrant box
    • upgrade images
  • Additional complexity in implementing webserver support for multiple api/content hosts
    • This is basically load balancing
    • Load balancing parameters per-host
    • Global load balancing parameters
    • Proposed design
      • pulp_webserver_api_balancing_params:
        foo: bar
        foo2: bar2
      • pulp_webserver_api_servers:
        • url: pulp-api-1:24817
          foo: bar
          foo2: bar2
        • url: pulp-api-2:24817
          foo: bar
          foo2: bar2
      • pulp_webserver_content_balancing_params:
        foo: bar
        foo2: bar2
      • pulp_webserver_content_servers:
        • url: pulp-content-1:24816
          foo: bar
          foo2: bar2
        • url: pulp-content-2:24816
          foo: bar
          foo2: bar2
  • Lets triage open issues

May 17 Agenda

  • Mike’s desire to make vagrant installs no longer build & install the collection
    • https://github.com/pulp/pulp_installer/pull/1099
    • This conflicts with molecule, which does not build and install the collection. But the installed collection takes precedence over the local repo.
      • I have repeatedly run molecule commands, 2 only to have to repeat them a 3rd time after deleting the collection.
    • Making this change would require all vagrant users to run rm -rf ~/.ansible/collections/ansible_collections/pulp/
    • agreed: Make this change, and communicate it well. Devs often use vagrant envs for months.