Pulp Installers Meeting Minutes

Minutes
#41

Nov 16 Agenda

#42

Nov 23 Agenda

  • need to file an issue for upgrade docs on going from pulp_installer to container based install
  • need to write a post on discourse announcing the stop of development of pulp_installer post 3.22
  • pulp-operator triage at future meetings
#43

you should rename the topic or starting a new one

1 Like
#44

Nov 29 Agenda

  • Unimplemented features/docs of single container vs pulp_installer
    • docs for signing service scripts (issue filed)
    • docs for keys/certs (issue filed)
    • variables like gunicorn workers (issue filed)
    • agreed: do these as soon as migration docs are done
  • Continue providing Apache snippets now that pulp_installer is EOL?
    • None of our deployers use Apache.
    • Foreman uses Apache. Do they use our snippets?
  • Because of containers, every user is going to have several plugins in their database schema, and we are stuck maintaining those plugins forever
    • agreed: Make a post about this on discourse
  • Separate image names for the complete list of plugins?
    • x9c4 had some thoughts on this.
    • agreed: Let’s get more feedback
    • We should decrease our image sprawl by implementing https as a variable (rather than a tag) 1st
#45

Dec 6 Agenda

  • PR up with migration instructions (many different scenarios)
  • Need someone to work on pulp-oci-images while Mike works on the ephemeral environment for crhc
    • Humberto will, availability permitting
#46

Dec 13 Agenda

  • Plan for HCaaS ephemeral environment:
    • jsherrill will be given cluster admin rights, then deploy the pulp-operator CRD
    • Each dev will run bonfire to create an ephemeral namespace
    • Each dev will run oc apply to deploy each individual pulp-operator deployment
    • mikedep333 to make any needed changes to pulp-operator for it to be applied successfully
      • Persistent volumes will likely need additional rights. Mike to test them after CRD is deployed.
        • NOTE: We cannot use object storage until production, and an ephemeral “emptyDir” is incompatible with pulp-operator’s current design of having multiple pods.
#47

January 10, 2023

#48

January 17, 2023

#49

January 24, 2023

  • [mikedep333] Finished fixing several issues in pulp_installer, 3.22.0 release PR is ready
  • 3.22 images are released
  • [dkliban] will follow up with x9c4 about branch protection rules on the pulp-oci-images repository
#50

January 31, 2023

  • pulp_installer 3.22.0 released
  • katello CI will use pulp_installer 3.22 for older pulpcore releases, no need to maintain 3.18 to 3.21 branches
  • [mikedep333] working on pulp_installer token auth issue
  • more tests and adjustments for the operator migration/upgrade
  • investigation and fix of operator CI issues
1 Like
#51

February 7, 2023

  • released pulp_installer 3.21.5 and 3.22.1 with the token auth key fix
  • [mikedep333] Proposed agenda for pulp_installer is:
    • pin ansible-lint on 3.21 & 3.22
    • fix & re-enable EL9 CI
    • Help galaxy fix their dependency issue that is breaking release-galaxy
  • [mikedep333] Figured out how to make docker-compose forward ports to scaled containers
    • What ranges to use on the host? 24816 and 24817 are adjacent to eachother.
      • pulp-content on 24716-24816 and pulp-api on 24817-24917
        • The 1st pulp-content container will be 24716 rather than 24816, but good enough
  • [mikedep333] “CEE enablement notes” was on my to-do list from last week
    • follow-up with Tanya (after configmgmtcamp)
  • [bmbouter] want to try pulp_ostree in ephemeral environment
#52

February 14, 2023

  • Want to discuss the design of solving this
  • Pulp Operator scope of responsibilities
    • openshift-specific routes needed to fix issues / improve performance by eliminating the webserver
    • AH requires multiple openshift-specific features
    • It makes sense for downstream to make the operator cluster-scoped rather than namespace-scoped
    • HCaaS will require a pod auto-scaler, this is openshift-specific
  • nightly building/publishing multi-process images is failing with a curl SSL error during tests, but those tests succeed for PR building
    • Should I just fix this during the CI files merge?
      • Yes
#53

February 21, 2023

  • I do not have notes from the meeting that included discussing the staging approach (clowder vs extending-our-operator)
    • clowder does use deployment templates though
    • I believe Pete Savage was the main person I talked to
    • auto-scaling will be a required feature very early, probably before we go into production
      • Partially because on the days of product releases, Red Hat traffic will increase greatly
  • Let’s add documentation about https://www.redhat.com/sysadmin/quadlet-podman
#54

February 28, 2023

  • pulp-oci-images CI code deduplication is merged
    • minor and micro tags for all images now
  • periodic-ci-pulp-pulp-operator-main-deploy-pulp-on-openshift failures in our slack channel
    • have not been updated since moving to golang operator
  • CI is not building old branches (properly)
  • working on updating multi-process container section of website
    • How much should I mention about other container solutions?
  • Researching more into getting HCaaS to staging
    • multiple stakeholders desire to start testing soon
  • Auto-scaling info sharing
  • opentelemetry-instrument command?
    • bmbouter advised me on the correct pip packages to install. That fixed the command error.
#55

March 7, 2023

  • pulp-operator support
    • k8s? openshift?
    • what do we claim and what we want to claim
    • Current CI we have?
      • Pre-merge: regualr 8ks
      • nightly schedule: openshift org CI (for pre-merge, use crc on your laptop)
    • What openshift-specific features do we have?
      • routes and the elimination of the webserver - solves some big problems
      • scc (openshift 4.12) - you cannot run a pod in openshift without an scc, we do not want to run containers as root. regular k8s has a “pod secuirty context”, which is like a limited version of SCC.
      • OLM - catalog to install the operator. OLM can be installed on regular k8s, but does not come by default.
      • In the future, openshift monitoring / logging
    • Agreed: In the medium-term, try to CI test against ephemeral / staging
      • platform team will provide jenkins
    • Agreed: In the short-term, we manually deploy against ephemeral / staging
  • 3.22 images getting pushed by both 3.22 branch and latest branch
    • PR will stop building 3.22 branch
    • We should rebase the 3.22 branch after 3.23 release
  • Decko mentioned that he needs to customize the s6 service files
    • Perhaps I should look into refactoring the s6 service files to make this less burdensome?
  • Clowder deployment
1 Like
#56

@mikedep333 Can we give another look/try on using systemd as supervisor to pulp all-in-one container?

#57

@decko That would make my life a lot easier, but be unacceptable for some users.

oci-systemd-hook enables podman to run systemd as the supervisor without any onerous requirements.

However, docker doesn’t have anything like oci-systemd-hook. It requires a systemd container to be run as a privileged container, which is unacceptable for many users.

1 Like
#58

March 14, 2023

  • Status of getting pulp on clowder
    • Determined we cannot use multi-process container
    • Did a sample “hello” app
      • going to verify a route
    • Haven’t decided whether to pursue the operator or “mutliple containers without operator” yet
    • whether we can create something like routes to replace webserver container is tbd
  • Migration to golang operator
    • backup/restore PR is open, awaits AAP review
  • support AAP CEE
  • need to figure out priority for https://issues.redhat.com/browse/AAP-5094
#59

March 21, 2023

  • Still working on wrapping operator in clowder
  • old versions aren’t publishing, looking into it
#60

March 30, 2023

  • HoloNet coffee hour
    • We had a big lego millenial falcon to build at one point.
  • [mikedep333] Thanks to hyagi, I now understand what it means to have a clowder “template”. The clowder objects can exist alongside other objects, even the misc operator objects. Clowder shouldn’t wrap the operator, it provides services/resources like redis to it.
    • Will try to use the OLM subscription again, can fall back to not using it.