Offline install pulp with helm in k8s cluster

Problem:
Installing pulp using helm fails, several pods don’t start.
Expected outcome:
Pulp install completes with all pods running
Pulpcore version:
N/A
Pulp plugins installed and their versions:
N/A
Operating system - distribution and version:
N/A
Other relevant data:
Hello,
I am trying to deploy pulp in an air-gapped kubernetes environment using helm.

As a first step I tried to install pulp in an internet connected envrionment first so I could get a feel for how pulp gets installled. The directions from here worked perfectly with no issues. Install Pulp Operator with Helm

Next, I tried to install pulp in our disconnected environment and found that the operator pod is running with no issues but other pods such as pulp-api, pulp-worker, pulp-content are failing to start (image pull backoff). I believe this is due to missing secrets for pulp in my cluster. I modified the helm chart as follows to pull images from my private repo.

# Default values for chart.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# Overrides for generated resource names to the previous value
fullnameOverride: pulp-operator

# Might want to be moved to under `.operator`, but left for backwards compatibility.
image: gsil-docker1.idm.gsil.org:5001/pulp-operator:v1.0.0-beta.4
namespace: pulp

podSecurityContext:
  runAsNonRoot: true

imagePullSecrets:
- name: redhat-operators-pull-secret

operator:
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
  resources:
    limits:
      cpu: 500m
      memory: 128Mi
    requests:
      cpu: 10m
      memory: 64Mi
  relatedImages:
    pulp: gsil-docker1.idm.gsil.org:5001/pulp-minimal:stable
    pulpWeb: gsil-docker1.idm.gsil.org:5001/pulp-web:3.68.1
    redis: gsil-docker1.idm.gsil.org:5001/redis:latest
    postgres: gsil-docker1.idm.gsil.org:5001/postgres:13

kubeProxy:
  image: gsil-docker1.idm.gsil.org:5001/kube-rbac-proxy:v0.13.0
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
  resources:
    limits:
      cpu: 500m
      memory: 128Mi
    requests:
      cpu: 10m
      memory: 64Mi

I presume I will need to manually create all my secrets for the cluster. Is there a best/easiest install method for this?

Hi @Jeremy_Tourville

I presume I will need to manually create all my secrets for the cluster. Is there a best/easiest install method for this?

Running tests and documenting disconnected installations is something that is in our radar for a long time: [DOC] Add a section explaining how to configure operator in disconnected environments. · Issue #684 · pulp/pulp-operator · GitHub
Off the top of my head, I guess you will need to create a Secret to pull from your private registry and update Pulp CR with this new Secret and the images:

  • create the image pull secret:
kubectl create secret docker-registry <new_pull_secret_name> \
    --docker-server=<registry_server> \
    --docker-username=<user_name> \
    --docker-password=<password> \
    --docker-email=<email>

## OR if you already have a .dockercfg file for the secured registry: 
kubectl create secret generic <new_pull_secret_name> \
    --from-file=.dockercfg=<path/to/.dockercfg> \
    --type=kubernetes.io/dockercfg

for example:

kubectl create secret docker-registry my-private-registry \
    --docker-server=gsil-docker1.idm.gsil.org:5001 \
    --docker-username=admin \
    --docker-password=password \
    --docker-email=not-needed
  • update Pulp CR with the new secret and the images:
kubectl patch pulp pulp --type merge -p '{"spec": {"image": "gsil-docker1.idm.gsil.org:5001/pulp-minimal", "image_web": "gsil-docker1.idm.gsil.org:5001/pulp-web", "database": {"postgres_image": "gsil-docker1.idm.gsil.org:5001/postgres:13"}, "cache": {"redis_image": "gsil-docker1.idm.gsil.org:5001/redis:latest"}, "image_pull_secrets": ["my-private-registry"]}}'

but I cannot think of an easier way to do disconnected installations.

1 Like

OK, I was going down the wrong rabbit hole based on memory when I wrote this posting. I am sorry to mislead on this.
I have found my true issue on why I am getting Init:ImagePullBackoff status. Even though I though I had modified the charts properly they are somehow getting an overrride. I’m not sure where the override is happening. If I run kubectl describe deploy/kube-api -n pulp for example, I can see the image line says:
image: quay.io/image/pulp-minimal:stable
So it’s trying to pull from the wrong location and not my private registry.

So your kubectl patch command basically modifies the pulp CR like this:

   apiVersion: repo-manager.pulpproject.org/v1beta2
    kind: Pulp
    metadata:
      name: pulp
      namespace: pulp
    spec:
      api:
        replicas: 1
      content:
        replicas: 1
      worker:
        replicas: 1
      image:
        gsil-docker1.idm.gsil.org:5001/pulp-minimal
      image_web:
        gsil-docker1.idm.gsil.org:5001/pulp-web
      database:
        postgres_image:
          gsil-docker1.idm.gsil.org:5001/postgres:13
      cache:
        redis_image:
          gsil-docker1.idm.gsil.org:5001/redis:latest  

So when I apply this I see pods getting pulled from the correct location when adding those parameters to the CR.

That is progress! Now I have to dig further on why I am getting a status of ERROR or CrashLoopBackoff for some of the pods.

UPDATE:
upon reviewing logs for the pods that are failing I am seeing a common theme. They all say something like this: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE

My servers in the cluster are configured with fips mode turned on. I am required to run this way due to security policies. Where do I go next?

1 Like

I haven’t encountered this error before. As a starting point, what do you think about following the “Diagnostic Steps” from FIPS: custom python 3.9 application loading pymssql package crashes with "OpenSSL internal error, assertion failed: FATAL FIPS SELFTEST FAILURE" - Red Hat Customer Portal for pulpcore packages (https://github.com/pulp/pulp-oci-images/blob/02ebce228d9ef7332d39f3a5be6418cc2106d193/images/pulp-minimal/stable/Containerfile.core#L19-L28) to try to find the culprit of this FIPS error?

Here are my findings.

[root@gsil-kube04 jtourville.sa]# pip3 list
Package      Version
------------ -------
ansible-core 2.16.3
cffi         1.16.0
cryptography 41.0.7
pip          23.2.1
ply          3.11
pycparser    2.20
PyYAML       6.0.1
setuptools   68.2.2
[root@gsil-kube04 jtourville.sa]# python3
Python 3.12.6 (main, Nov  5 2024, 07:59:45) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ansible-core
  File "<stdin>", line 1
    import ansible-core
                  ^
SyntaxError: invalid syntax
>>> import cffi
>>> import cryptography
>>> import pip
>>> import ply
>>> import pycparser
>>> import PyYAML
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'PyYAML'
>>> import setuptools
>>> exit()

hm… can you please try again, but this time from inside the pulpcore-content pod:

# kubectl exec -it deployment/pulp-content  -- bash
bash-5.1$ pip3 list

OR
# kubectl exec  deployment/pulp-content  -- python3 -m pip list

@hyagi
I can’t exec into the pod because the pod is never in a ready state.

[root@gsil-kube04 jtourville.sa]# kubectl get po -n pulp
NAME                                              READY   STATUS             RESTARTS          AGE
pulp-api-7cbfc6c4f6-x5lb7                         1/1     Running            178 (8m35s ago)   17h
pulp-content-c54778dcf-n4tpq                      0/1     CrashLoopBackOff   205 (4m13s ago)   17h
pulp-database-0                                   1/1     Running            0                 17h
pulp-operator-controller-manager-b4bd6f77-gjqsg   2/2     Running            0                 17h
pulp-redis-df74c6bf5-4dgdf                        1/1     Running            0                 17h
pulp-worker-bb7789cc5-wc95s                       0/1     CrashLoopBackOff   206 (49s ago)     17h

[root@gsil-kube04 jtourville.sa]# kubectl exec -it deployment/pulp-content -- bash
error: unable to upgrade connection: container not found ("content")

If I try this same procedure in my internet connected cluster where I have FIPS off, I can exec into the pod. The list of modules is quite long. I started to test a few and here is what I get.

[root@kube01 ~]# kubectl exec -it deploy/pulp-content -- bash
bash-5.1$ pip3 list
Package                                  Version
---------------------------------------- ------------
aiodns                                   3.2.0
aiofiles                                 24.1.0
aiohappyeyeballs                         2.4.3
aiohttp                                  3.10.10
aiohttp_socks                            0.9.1
aiohttp-xmlrpc                           1.5.0
aiosignal                                1.3.1
ansible-builder                          3.1.0
ansible-compat                           24.10.0
ansible-core                             2.14.18
ansible-lint                             6.22.2
asgiref                                  3.8.1
async-lru                                2.0.4
async-timeout                            4.0.3
asyncio-throttle                         1.0.2
attrs                                    22.2.0
azure-core                               1.32.0
azure-storage-blob                       12.24.0
backoff                                  2.2.1
bandersnatch                             6.1.0
beautifulsoup4                           4.12.3
bindep                                   2.11.0
black                                    24.10.0
boto3                                    1.35.66
botocore                                 1.35.66
bracex                                   2.5.post1
cachetools                               5.5.0
certifi                                  2024.8.30
cffi                                     1.14.5
chardet                                  5.2.0
charset-normalizer                       3.4.0
click                                    8.1.7
createrepo_c                             1.1.4
cryptography                             43.0.3
dbus-python                              1.2.18
defusedxml                               0.7.1
Deprecated                               1.2.15
diff-match-patch                         20241021
distro                                   1.9.0
Django                                   4.2.16
django-auth-ldap                         5.1.0
django-filter                            24.3
django-guid                              3.5.0
django-import-export                     3.3.9
django-lifecycle                         1.2.4
django-readonly-field                    1.1.2
django-storages                          1.14.3
djangorestframework                      3.15.2
djangorestframework-queryfields          1.1.0
drf-access-policy                        1.5.0
drf-nested-routers                       0.94.1
drf-spectacular                          0.27.2
dynaconf                                 3.2.6
et_xmlfile                               2.0.0
filelock                                 3.16.1
flake8                                   6.1.0
frozenlist                               1.5.0
galaxy_importer                          0.4.26
gitdb                                    4.0.11
GitPython                                3.1.43
google-api-core                          2.23.0
google-auth                              2.36.0
google-cloud-core                        2.4.1
google-cloud-storage                     2.18.2
google-crc32c                            1.6.0
google-resumable-media                   2.7.2
googleapis-common-protos                 1.66.0
gpg                                      1.15.1
gunicorn                                 23.0.0
humanfriendly                            10.0
idna                                     3.10
importlib-metadata                       6.0.1
importlib_resources                      6.4.5
inflection                               0.5.1
iniparse                                 0.5
isodate                                  0.7.2
Jinja2                                   3.1.4
jmespath                                 1.0.1
jq                                       1.8.0
json-stream                              2.3.2
json-stream-rs-tokenizer                 0.4.26
jsonschema                               4.23.0
jsonschema-specifications                2024.10.1
libcomps                                 0.1.21.post1
lxml                                     5.3.0
mailbits                                 0.2.1
Markdown                                 3.7
markdown-it-py                           3.0.0
MarkupPy                                 1.14
MarkupSafe                               3.0.2
mccabe                                   0.7.0
mdurl                                    0.1.2
multidict                                6.1.0
mypy-extensions                          1.0.0
nh3                                      0.2.18
odfpy                                    1.4.1
openpyxl                                 3.1.5
opentelemetry-api                        1.27.0
opentelemetry-exporter-otlp-proto-common 1.27.0
opentelemetry-exporter-otlp-proto-http   1.27.0
opentelemetry-proto                      1.27.0
opentelemetry-sdk                        1.27.0
opentelemetry-semantic-conventions       0.48b0
packaging                                24.2
Parsley                                  1.3
pathspec                                 0.12.1
pbr                                      6.1.0
pillow                                   11.0.0
pip                                      21.3.1
pkginfo                                  1.11.2
platformdirs                             4.3.6
ply                                      3.11
productmd                                1.33
propcache                                0.2.0
proto-plus                               1.25.0
protobuf                                 4.25.5
psycopg                                  3.2.3
psycopg-binary                           3.2.3
psycopg2                                 2.8.6
pulp-ansible                             0.23.0
pulp-certguard                           1.8.0
pulp-container                           2.22.0
pulp-deb                                 3.4.0
pulp-file                                1.16.0
pulp-glue                                0.29.2
pulp-maven                               0.8.1
pulp-ostree                              2.4.4
pulp-python                              3.12.5
pulp-rpm                                 3.27.2
pulpcore                                 3.68.0
pyasn1                                   0.6.1
pyasn1_modules                           0.4.1
pycares                                  4.4.0
pycodestyle                              2.11.1
pycparser                                2.20
pyflakes                                 3.1.0
Pygments                                 2.18.0
PyGObject                                3.40.1
pygtrie                                  2.5.0
PyJWT                                    2.9.0
pyOpenSSL                                24.2.1
pyparsing                                3.2.0
pypi-simple                              0.10.0
python-dateutil                          2.8.1
python-debian                            0.1.49
python-gnupg                             0.5.3
python-ldap                              3.4.4
python-nginx                             1.5.7
python-socks                             2.5.3
PyYAML                                   6.0.2
redis                                    5.2.0
referencing                              0.35.1
requests                                 2.32.3
resolvelib                               0.8.1
rhsm                                     1.19.2
rich                                     13.9.4
rpds-py                                  0.21.0
rpm                                      4.16.1.3
rsa                                      4.9
ruamel.yaml                              0.18.6
ruamel.yaml.clib                         0.2.12
s3transfer                               0.10.4
semantic-version                         2.10.0
setuptools                               53.0.0
six                                      1.15.0
smmap                                    5.0.1
solv                                     0.7.29
soupsieve                                2.6
sqlparse                                 0.5.2
subprocess-tee                           0.4.2
systemd-python                           234
tablib                                   3.5.0
tomli                                    2.1.0
typing_extensions                        4.12.2
uritemplate                              4.1.1
url-normalize                            1.4.3
urllib3                                  1.26.20
uuid6                                    2024.7.10
wcmatch                                  10.0
whitenoise                               6.8.2
wrapt                                    1.16.0
xlrd                                     2.0.1
xlwt                                     1.3.0
yamllint                                 1.35.1
yarl                                     1.17.1
zipp                                     3.21.0


bash-5.1$ import aiodns
bash: import: command not found
bash-5.1$ python3
Python 3.9.20 (main, Sep  9 2024, 00:00:00)
[GCC 11.5.0 20240719 (Red Hat 11.5.0-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import aiodns
>>> import aiofiles
>>> import aiohappyeyeballs
>>> import aiohttp
>>> import aiohttp_socks
>>> import aiohttp-xmlrpc
  File "<stdin>", line 1
    import aiohttp-xmlrpc
                  ^
SyntaxError: invalid syntax
>>> import aiosignal
>>> import ansible-builder
  File "<stdin>", line 1
    import ansible-builder
                  ^
SyntaxError: invalid syntax
>>> import ansible-compat
  File "<stdin>", line 1
    import ansible-compat
                  ^
SyntaxError: invalid syntax
>>> import ansible-core
  File "<stdin>", line 1
    import ansible-core
                  ^
SyntaxError: invalid syntax
>>> import ansible-lint
  File "<stdin>", line 1
    import ansible-lint
                  ^
SyntaxError: invalid syntax
>>> import asgiref
>>> import async-lru
  File "<stdin>", line 1
    import async-lru
           ^
SyntaxError: invalid syntax
>>> import async-timeout
  File "<stdin>", line 1
    import async-timeout
           ^
SyntaxError: invalid syntax
>>> import async-throttle
  File "<stdin>", line 1
    import async-throttle
           ^
SyntaxError: invalid syntax
>>> import import attrs
  File "<stdin>", line 1
    import import attrs
           ^
SyntaxError: invalid syntax
>>> import azure-core
  File "<stdin>", line 1
    import azure-core
                ^
SyntaxError: invalid syntax
>>> import azure-storage-blob
  File "<stdin>", line 1
    import azure-storage-blob
                ^
SyntaxError: invalid syntax
>>> import backoff
>>> import bandersnatch
>>> import beautifulsoup
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'beautifulsoup'
>>> import bindep
>>> import black
>>> import boto3
>>> import botocore
>>> import bracex
>>> import cachetools
>>> import certfi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'certfi'
>>> import certifi
>>> import cffi
>>> import chardet
>>> import charset-normalizer
  File "<stdin>", line 1
    import charset-normalizer
                  ^
SyntaxError: invalid syntax
>>> import click
>>> import createrepo_c
>>> import cryptography
>>> import dbus-python
  File "<stdin>", line 1
    import dbus-python
               ^
SyntaxError: invalid syntax
>>> import defusedxml
>>> import Deprecated
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'Deprecated'
>>> import diff-match-patch
  File "<stdin>", line 1
    import diff-match-patch
               ^
SyntaxError: invalid syntax
>>> import distro
>>> import Django
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'Django'
>>> import djano-auth-ldap
  File "<stdin>", line 1
    import djano-auth-ldap
                ^
SyntaxError: invalid syntax
>>> import django-filter
  File "<stdin>", line 1
    import django-filter
                 ^
SyntaxError: invalid syntax
>>> import django-auth-ldap
  File "<stdin>", line 1
    import django-auth-ldap
                 ^
SyntaxError: invalid syntax
>>> import django-guid
  File "<stdin>", line 1
    import django-guid
                 ^
SyntaxError: invalid syntax
>>> import django-import-export
  File "<stdin>", line 1
    import django-import-export
                 ^
SyntaxError: invalid syntax
>>> import django-lifecycle
  File "<stdin>", line 1
    import django-lifecycle
                 ^
SyntaxError: invalid syntax
>>> import django-lifecycle
  File "<stdin>", line 1
    import django-lifecycle
                 ^
SyntaxError: invalid syntax

Should I keep going? I haven’t found any that throw a SSL error yet… FIPS is definitely keeping the pods from starting for whatever reason. If I turn on FIPS in my connected cluster the pods end up in the same state as my disconnected environment, so that variable is consistent.

That was a very good idea to try the same in the connected cluster, it shows that we are probably heading in the right direction.

I can’t exec into the pod because the pod is never in a ready state.

hmm… in this case, we can try some of these options:
option 1- it seems like the api pod is in a running state (even though it is also restarting a lot):

NAME                                              READY   STATUS             RESTARTS          AGE
pulp-api-7cbfc6c4f6-x5lb7                         1/1     Running            178 (8m35s ago)   17h

Considering that the pulpcore-api and pulpcore-content pods run the same image, we could also verify the FIPS issue in it:

kubectl exec -it  deployment/pulp-api  -- bash

option 2- if this is an openshift cluster, you could use the oc debug to get into the pod and run the commands. For example:

oc debug deployment/pulp-content

option 3- run the kubectl debug:

kubectl debug -it pod/<name of the pod in crashinglooback state> --image <pulp container image>

for example:
kubectl debug -it pod/pulp-content-c54778dcf-n4tpq --image gsil-docker1.idm.gsil.org:5001/pulp-minimal:stable
1 Like

Thanks,

I tested the entire list of modules (which took quite a while) and never got any fips.c(145): OpenSSL internal error, assertion failed: FATAL FIPS SELFTEST FAILURE

Many modules imported with no issues.
Everything that had a problem fell into two categories:

File "<stdin>", line 1
import <module_name>
SyntaxError: invalid syntax

OR

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named '<module_name>'

I don’t understand why those particular errors are happening, especially since I copy pasted the names right from the list when importing.

I tried debugging the pods as you suggested. (BTW, this is a pure Kubernetes cluster, not Openshift) I am still pretty new to kubernetes. I am not sure what you intended for me to do at this point. I can get to the CLI of the pod but what do I do next? This seems the same to me as exec(ing) into the pod. I am feeling pretty lost at this point. I am not sure what else I can try. Help! :slight_smile:

I can get to the CLI of the pod but what do I do next?

The idea of the debug pod was to be able to “exec into the crashing pod” (not exactly the crashing pod, but a similar pod with the same image) and run the Diagnostic Steps.

I tested the entire list of modules […]

Did you do that from inside the debug pod/container? If not, please, repeat the test from “inside” a pulpcore-content pod so we can check which python package(s) installed in the container image is(are) the possible culprit of the FIPS error.

1 Like

yes

The pulp-api pod does stay up long enough to exec in. I tried testing the modules from within the pod.