Pulp Installers Meeting Minutes

Minutes
#61

April 4, 2023

#62

April 11, 2013

  • Just completed the clowder prototype (thanks hyagi!)
    • Now to estimate the remaining work to get to staging environment
  • [ttereshc] tracking planned work and work in progress
    • let’s find a way not to assign issues which are not actively being worked on, any suggestions are welcome
      • We will be more cautious in which issues get assigned in a weekly meeting.
      • We will unassign issues when they cannot be worked on soon.
  • Does spredzy still need to be in our github subteam?
    • Last PR to pulp_installer was Jun 2022
    • No PRs to pulp-oci-images
    • No PRS to pulp-operator
    • He’s now a manager
    • Just remove him, we can re-add later
  • Add Decko to Pulp Deployments team
    • Decko talked about this with Tanya earlier
    • Only needs for pulp-oci-images repo now
    • Added: He gets write access to repos. Protected branches exist for all repos except the brand new pulp-clowder-deployments.
  • clowder database version in pulp-clowder-deployments
    • I set it to 15 because it’s the latest
    • Setting to 13 would enable hypothetical data migration to pulp-operator or compose or multi-process container
    • pulp-operator CI tests are done using 13
    • hstore as a trusted extension was done in 13
    • Agreed: Let’s just set it to 13.
  • [hyagi] Is everyone aware of the process of releasing a new version of the operator?
    • if not, in case AAP team agree with merging https://github.com/pulp/pulp-operator/pull/877 while I’m on vacantions, just ask Fabricio about the processs (I also don’t know how to do it and I was waiting for #877 to be merged to learn the process with him).
  • mikedep333 to create a proper repo for https://github.com/mikedep333/pulp-ocp-template as written up as https://github.com/pulp/pulp-operator/issues/618
#63

April 18, 2023

  • Implementing the integrated ephemeral environment now
    • Estimated to be done by end of Friday
#64

April 25, 2023

  • Implementing the configuration from a container for the integrated ephemeral environment now
    • remainder of integrated ephemeral environment done
  • Having instabilities for oci_env docker tests
  • hyagi mentoring decko on openshift on a very particular pace
1 Like
#65

Apologies for not posting these for a while, it was an oversight:

Jun 20, 2023

  • Haven’t done the quick fix / release for AAP-11911 yet
  • Hosted Content: Deploying to staging is making slow but steady progress
  • Re-review please: https://github.com/pulp/pulp-oci-images/pull/508
  • We need to publish a pulp-web image to be used on consoledot.
    • Agreed: Create another dockerfile in pulp-clowder-deployments repo. Build in same script.

Jun 13, 2023

  • Was asked to fix AAP-11911
  • Hosted Content: Deploying to staging is making steady progress
  • 3.26 fixed release:
    • Not released yet, we just merged the PR to update the list of branches to build
    • Manually triggered a 3.26 CI run (pulp-oci-images CI) to build and push the 3.26 image

Jun 6, 2023

  • Hosted services: Mike working on deployment method
    • Plan to merge PR to pulp-clowder-deployments, and submit PR to app-interface today
  • Did CI fixes so that 3.25 image is built and pushed

May 30, 2023

  • Hosted services: Mike working on deployment method
    • Working on the dummy script, trying things out
  • A couple of open PRs / issues
    • Mike providing reviews, answering questions
  • Mike helped Decko with UBI image limitations
    • Some packages exist in repos like “RHEL base” but not “UBI base”
    • Adding CentOS Stream Repos for now
    • If there’s breakage, we can try the rockylinux / almalinux repos. Not using them now because their 8.9 release will lag behind RHEL 8.9 release by a few weeks.

May 23, 2023

  • AAP-11911 is actually on hold (bmbouter relayed this on 5/17)
  • Work with the consulting team (Matt Dorn)?
    • No issues have come up
    • I think we wanted their review on this: https://github.com/pulp/pulp-operator/pull/898
      • Agreed: This is such a big review, we will focus as a team (with or without their help) on reviewing it when hyagi gets back, and establishes who needs this, and when.

May 17, 2023

(No meeting on May 9 or May 10)

May 2, 2023

1 Like
#66

Jun 27, 2023

1 Like
#67

Jul 12, 2023

  • We fixed pulp-oci-images CI last week
  • Did the release for fixed images for prior pulp versions
  • oci_env CI has the Ubuntu podman-compose fix now (from pulp-oci-images)
#68

Jul 19, 2023

  • CI dashboard will be helpful
  • for hosted-content: CI smoke tests with CJI IQE (requires writing plugin) or a custom bash test script that calls pytest?
    • Agreed: Write custom bash test script for now
#69

Jul 25, 2023

  • mikedep333 to continue following up with user support requests / bug reports for pulp_installer
#70

Aug 1, 2023

  • Enabling multiple pulp operators to run in a namespace
    • Needed because content-sources and image-builder will each have a pulp operator running in a single namespace
    • Except for a few singletons, almost all k8s objects will have names like “content-sources-pulp-api-abcdef” and “image-builder-pulp-api-abcdef”
1 Like
#71

Aug 8, 2023

  • Idea (not proposal) for pulpcon: demo HA pulp in k8s
    • Services team will be doing HA soon (part of production plan)
    • Let’s do a joint presentation: mikedep333 & hyagi
    • [mikedep333] Please re-explain the pulp-operator deployment modified function check
  • Any changes with decko on leave?
    • dkliban and mikedep333 to maintain pulp-oci-images
  • [mikedep333] [services] pulp-smash config issue
#72

Aug 15, 2023

#73

Aug 22, 2023

  • Compose data loss bug
    • Fixing the migration of config files from multi-process container to compose
  • Helping jhutar with performance testing cluster
  • pulpcore-selinux 2.0.0 released for katello / satellite
#74

Aug 29, 2023

  • pulp_container CI is failing with latest pulp-ci-centos image
    • There was a regression in the capabilities of newuidmap / newgidmap - easy to fix
    • I noticed that we never set VOLUMES for /var/lib/containers & /var/lib/pulp/.local/share/containers
      • Ask the pulp_container team about this.
    • Lots of other prescribed changes too that we never did
    • Agreed: Do not focus on all the other prescribed changes
    • Agreed: File an issue
1 Like
#75

Sept 5, 2023

  • backup of /var/lib/pulp/tmp content
    • can we ignore this folder during backup? does it have anything that is important?
#76

September 12, 2023

  • Need another pulp-operator release with the new entrypoint for pulpcore-api.

    • How can we avoid carrying a separate entry point in the operator?

  • Looks like we still have a single migration job running when we have two deployments of Pulp in one namespace.

#77

October 10, 2023

  • Resuming work on nested latest podman (for pulp_container CI)
    • 2 issues fixed, at least 1 remaining
  • Going to finally merge the data loss bug
    • 1st need to rebase
    • was delayed due to CI being red
  • pulp-operator testing
    • Modify an existing test to deploy 2 instances of the operator side-by-side, for image-builder + content-sources
    • We are considering making our operator global, watching all namespaces
    • (Another project) image is registered as a catalog source, as a bundle image. So they push these in CI, and their cluster-wide OLM operator knows to deploy a new version.
    • Releasing to the catalog from the main branch?
1 Like
#78

October 17, 2023

  • Status of nested containers

    • Resumed investigating, the 3rd issue is with mounting /proc in the nested container

  • pulp-operator roadmap

    • We should have one.
    • Next year or 2 timeframe.
    • Promote shared understanding to the team.

  • Suggested items

    • [dkliban][HMS] Add testing of dual pulp deployments in one namespace
    • [hyagi]bring some samples that we will use in pulpcon presentation up to date
    • [dkliban][HMS] fix how we release the operator bundle (wrong tag)
    • [decko for now] release process automation
    • [hyagi] CVE http/2
    • [collaborative effort?][HMS] Singleton operator - can watch all the namespaces
    • [good for newcomers] review and update the operator spec fields description/appearance in ocp
    • refactor https://docs.google.com/document/d/1a9j7pVi7fv_Fym6TBSxDNEvPyZVqexUNns3jdcjhIrs/edit#heading=h.8vjxjmi9nsre (see ‘code improvements’)
    • metadata signing needs to be fixed (dkliban to check if HMS is affected)
1 Like
#79

October 25, 2023

  • Status of nested containers
    • Explored alternatives thoroughly, but approach is require outer containers to support UIDs up to 75535 (65535 + 10000)
      • This avoids need for migration of existing containers
      • This avoids security issues and potential compatibility of supporting UIDs inside the nested container of upto 65534.
      • OpenShift is compatible for now because they do not run containers in user namespaces and therefore have 4G UIDs available
  • [tanya/hyagi] need clarification on impact on community because of “[HMS] Singleton operator - can watch all the namespaces”
    • what is the use case
    • at some point we moved from cluster-scoped to namespace-scoped operator, any concern in watching all namespaces now?
    • will we provide one option or all the combinaitons possible?
      • all
  • [hyagi] CVE for operator framework
  • [hyagi] Issue #1121
    • Does it affect pulp?
    • Should we warn users about “galaxy-operator” migration?
    • What is the correct approach to fix the error?
  • [hyagi] In a galaxy thread on slack the following question arose:
  • “Isn’t it required to stop or scale down services before running migrations to prevent locks or conflicts?”*
    • [dkliban] Not required
1 Like
#80

October 31, 2023

  • Status of nested containers
    • Fixed an issue whereby the nested container needed 64K + 1 UIDs from the outer container (because UID 0 is already mapped in Podman but not Docker)
    • Hopefully the CI passes, we won’t need to run “podman system migrate” in the inner container, and I can move onto the pulp-container PR.
  • weird CI breakage for pulp-ansible / pulpcore version
  • pulp-operator dual Pulp deployment check
  • Update on testing 2 instances of pulp-operator with kind (K8s on container instead of VM)
    • we are maxing the limits of GHA CI
    • We got the rudimentary check done
1 Like