Fail to download python packages when ingress enabled

Support
#1

Problem:
I have deployed pulp project and defined local pypi repository along with remote repository pointing to pypi.org. When I used port-forwarding for the web and api services and tried to install packages using pip3 it worked (the caching mechanism didn’t work but we will leave it for different discussion).

I wanted to expose the pulp project using ingress out of my k8s cluster, the web UI is working, but when I try to install the packages I receive the following errors:

ERROR: Could not install packages due to an OSError: (“Connection broken: InvalidChunkLength(got length b’’, 0 bytes read)”, InvalidChunkLength(got length b’’, 0 bytes read))

AND

pip._vendor.urllib3.exceptions.ProtocolError: (“Connection broken: BrokenPipeError(32, ‘Broken pipe’)”, BrokenPipeError(32, ‘Broken pipe’))

Here is my operator deployment:

_helpers.tpl

{{- define "secret" -}}

{{- $secret := (lookup "v1" "Secret" .namespace .name).data }}

{{- if and $secret (index $secret .key) }}

{{- index $secret .key | b64dec }}

{{- else }}

{{- .value }}

{{- end }}

{{- end }}

# if the secret exists, return it otherwise generate random password (16 characters)

{{- define "pulp.adminPassword" -}}

{{ include "secret" (dict "namespace" .Release.Namespace "name" .Values.secrets.adminSecret "key" "password" "value" (randAlphaNum 16)) }}

{{- end}}

pulp-operator.yaml

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: operatorgroup
  namespace: {{ .Release.Namespace }}
spec:
  targetNamespaces:
  - {{ .Release.Namespace }}
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: pulp-operator
  namespace: {{ .Release.Namespace }}
spec:
  channel: beta
  name: pulp-operator
  source: operatorhubio-catalog
  sourceNamespace: olm

pulp.yaml

apiVersion: repo-manager.pulpproject.org/v1beta2
kind: Pulp
metadata:
  name: {{ .Chart.Name }}
  namespace: {{ .Release.Namespace }}
spec:
  admin_password_secret: {{ .Values.secrets.adminSecret }}
  api:
    replicas: {{ .Values.global.replicaCount }}
  content:
    replicas: {{ .Values.global.replicaCount }}
  worker:
    replicas: {{ .Values.global.replicaCount }}
  web:
    replicas: {{ .Values.global.replicaCount }}
    
  # file storage persistence
  file_storage_storage_class: {{ .Values.global.storageClass }}
  file_storage_size: {{ .Values.file.size }}
  file_storage_access_mode: {{ .Values.file.accessMode }}

  database:
    postgres_storage_class: {{ .Values.global.storageClass }}

  cache:
    enabled: true # Always enable caching mechanism
    redis_storage_class: {{ .Values.global.storageClass }}
  
  pulp_settings:
    content_origin: "https://{{ .Release.Namespace }}.{{ .Values.global.domain }}"
    content_path_prefix: "/pulp/content/"

secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.secrets.adminSecret }}
  namespace: {{ .Release.Namespace }}
  annotations:
    helm.sh/resource-policy: keep
stringData:
  password: {{ include "pulp.adminPassword" . }}
---
# tls certificate secret
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Release.Name }}-tls-secret
  namespace: {{ .Release.Namespace }}
type: kubernetes.io/tls
data:
  tls.crt: |-
    {{ .Files.Get "files/pulp.crt" | b64enc | indent 4 }}
  tls.key: |-
    {{ .Files.Get "files/pulp.key" | b64enc | indent 4 }}

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-ingress
  namespace: {{ .Release.Namespace }}
spec:
  ingressClassName: nginx-apps
  rules:
  - host: "{{ .Release.Namespace }}.{{ .Values.global.domain }}"
    http:
      paths:
      - backend:
          service:
            name: pulp-project-web-svc 
            port:
              number: 24880
        path: /
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              number: 24817
        path: /pulp/api/v3/
        pathType: Prefix
  tls:
  - hosts: 
    - "{{ .Release.Namespace }}.{{ .Values.global.domain }}"
    secretName: {{ .Release.Name }}-tls-secret

Logs:

kubectl logs pulp-project-worker-677df88f9-h5gdc -n pulp-prod

Waiting on postgresql to start...

Postgres started.

Checking for database migrations

Database migrated!

pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker

pulp [None]: pulpcore.tasking.worker:INFO: New worker '1@pulp-project-worker-677df88f9-h5gdc' discovered
Blockquote

kubectl logs pulp-project-web-67bf57796b-74cqz -n pulp-prod

2025/03/05 11:53:36 [notice] 7#7: using the "epoll" event method

2025/03/05 11:53:36 [notice] 7#7: nginx/1.27.3

2025/03/05 11:53:36 [notice] 7#7: built by gcc 12.2.0 (Debian 12.2.0-14)

2025/03/05 11:53:36 [notice] 7#7: OS: Linux 5.14.21-150400.24.119-default

2025/03/05 11:53:36 [notice] 7#7: getrlimit(RLIMIT_NOFILE): 1048576:1048576

2025/03/05 11:53:36 [notice] 7#7: start worker processes

2025/03/05 11:53:36 [notice] 7#7: start worker process 8

172.20.66.33 - - [05/Mar/2025:11:53:43 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 5425 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:53:52 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4854 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:54:02 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4854 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:54:12 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4854 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:54:22 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4854 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:54:32 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4854 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:54:42 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4854 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:54:52 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4854 "-" "kube-probe/1.27"

2025/03/05 11:55:02 [error] 8#8: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.66.33, server: pulp-project-web-67bf57796b-74cqz, request: "GET /pulp/api/v3/status/ HTTP/1.1", upstream: "http://10.53.129.121:24817/pulp/api/v3/status/", host: "10.52.1.128:8080"

172.20.66.33 - - [05/Mar/2025:11:55:02 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 502 157 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:55:12 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:55:22 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:55:32 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:55:42 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:55:52 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:56:02 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:56:12 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:56:22 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

10.52.0.136 - - [05/Mar/2025:11:56:24 +0000] "GET /pypi/local-pypi/simple/keras/ HTTP/1.1" 200 59045 "-" "pip/25.0.1 {\x22ci\x22:null,\x22cpu\x22:\x22arm64\x22,\x22distro\x22:{\x22name\x22:\x22macOS\x22,\x22version\x22:\x2214.7.4\x22},\x22implementation\x22:{\x22name\x22:\x22CPython\x22,\x22version\x22:\x223.12.9\x22},\x22installer\x22:{\x22name\x22:\x22pip\x22,\x22version\x22:\x2225.0.1\x22},\x22openssl_version\x22:\x22OpenSSL 3.4.1 11 Feb 2025\x22,\x22python\x22:\x223.12.9\x22,\x22rustc_version\x22:\x221.85.0\x22,\x22setuptools_version\x22:\x2275.8.0\x22,\x22system\x22:{\x22name\x22:\x22Darwin\x22,\x22release\x22:\x2223.6.0\x22}}"

172.20.66.33 - - [05/Mar/2025:11:56:32 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

2025/03/05 11:56:39 [info] 8#8: *39 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too (104: Connection reset by peer) while reading upstream, client: 10.52.0.136, server: pulp-project-web-67bf57796b-74cqz, request: "GET /pypi/local-pypi/simple/tensorflow/ HTTP/1.1", upstream: "http://10.53.129.121:24817/pypi/local-pypi/simple/tensorflow/", host: "pulp-prod.k8s.com"

10.52.0.136 - - [05/Mar/2025:11:56:39 +0000] "GET /pypi/local-pypi/simple/tensorflow/ HTTP/1.1" 200 332017 "-" "pip/25.0.1 {\x22ci\x22:null,\x22cpu\x22:\x22arm64\x22,\x22distro\x22:{\x22name\x22:\x22macOS\x22,\x22version\x22:\x2214.7.4\x22},\x22implementation\x22:{\x22name\x22:\x22CPython\x22,\x22version\x22:\x223.12.9\x22},\x22installer\x22:{\x22name\x22:\x22pip\x22,\x22version\x22:\x2225.0.1\x22},\x22openssl_version\x22:\x22OpenSSL 3.4.1 11 Feb 2025\x22,\x22python\x22:\x223.12.9\x22,\x22rustc_version\x22:\x221.85.0\x22,\x22setuptools_version\x22:\x2275.8.0\x22,\x22system\x22:{\x22name\x22:\x22Darwin\x22,\x22release\x22:\x2223.6.0\x22}}"

172.20.66.33 - - [05/Mar/2025:11:56:42 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:56:52 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:57:02 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:57:12 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:57:22 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

10.52.0.136 - - [05/Mar/2025:11:57:29 +0000] "GET /pypi/local-pypi/simple/tensorflow/ HTTP/1.1" 200 741428 "-" "pip/25.0.1 {\x22ci\x22:null,\x22cpu\x22:\x22arm64\x22,\x22distro\x22:{\x22name\x22:\x22macOS\x22,\x22version\x22:\x2214.7.4\x22},\x22implementation\x22:{\x22name\x22:\x22CPython\x22,\x22version\x22:\x223.12.9\x22},\x22installer\x22:{\x22name\x22:\x22pip\x22,\x22version\x22:\x2225.0.1\x22},\x22openssl_version\x22:\x22OpenSSL 3.4.1 11 Feb 2025\x22,\x22python\x22:\x223.12.9\x22,\x22rustc_version\x22:\x221.85.0\x22,\x22setuptools_version\x22:\x2275.8.0\x22,\x22system\x22:{\x22name\x22:\x22Darwin\x22,\x22release\x22:\x2223.6.0\x22}}"

2025/03/05 11:57:30 [warn] 8#8: *51 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/1/00/0000000001 while reading upstream, client: 10.52.0.136, server: pulp-project-web-67bf57796b-74cqz, request: "GET /pulp/content/local-pypi/tensorflow-2.18.0-cp312-cp312-macosx_12_0_arm64.whl?redirect=https://files.pythonhosted.org/packages/dc/bf/4cc283db323fd723f630e2454b2857054d2c81ff5012c1857659e72470f1/tensorflow-2.18.0-cp312-cp312-macosx_12_0_arm64.whl HTTP/1.1", upstream: "http://10.53.224.206:24816/pulp/content/local-pypi/tensorflow-2.18.0-cp312-cp312-macosx_12_0_arm64.whl?redirect=https://files.pythonhosted.org/packages/dc/bf/4cc283db323fd723f630e2454b2857054d2c81ff5012c1857659e72470f1/tensorflow-2.18.0-cp312-cp312-macosx_12_0_arm64.whl", host: "pulp-prod.com"

172.20.66.33 - - [05/Mar/2025:11:57:32 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:57:42 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

2025/03/05 11:57:45 [info] 8#8: *51 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too (104: Connection reset by peer) while reading upstream, client: 10.52.0.136, server: pulp-project-web-67bf57796b-74cqz, request: "GET /pulp/content/local-pypi/tensorflow-2.18.0-cp312-cp312-macosx_12_0_arm64.whl?redirect=https://files.pythonhosted.org/packages/dc/bf/4cc283db323fd723f630e2454b2857054d2c81ff5012c1857659e72470f1/tensorflow-2.18.0-cp312-cp312-macosx_12_0_arm64.whl HTTP/1.1", upstream: "http://10.53.224.206:24816/pulp/content/local-pypi/tensorflow-2.18.0-cp312-cp312-macosx_12_0_arm64.whl?redirect=https://files.pythonhosted.org/packages/dc/bf/4cc283db323fd723f630e2454b2857054d2c81ff5012c1857659e72470f1/tensorflow-2.18.0-cp312-cp312-macosx_12_0_arm64.whl", host: "pulp-prod.com"

10.52.0.136 - - [05/Mar/2025:11:57:45 +0000] "GET /pulp/content/local-pypi/tensorflow-2.18.0-cp312-cp312-macosx_12_0_arm64.whl?redirect=https://files.pythonhosted.org/packages/dc/bf/4cc283db323fd723f630e2454b2857054d2c81ff5012c1857659e72470f1/tensorflow-2.18.0-cp312-cp312-macosx_12_0_arm64.whl HTTP/1.1" 200 13336843 "-" "pip/25.0.1 {\x22ci\x22:null,\x22cpu\x22:\x22arm64\x22,\x22distro\x22:{\x22name\x22:\x22macOS\x22,\x22version\x22:\x2214.7.4\x22},\x22implementation\x22:{\x22name\x22:\x22CPython\x22,\x22version\x22:\x223.12.9\x22},\x22installer\x22:{\x22name\x22:\x22pip\x22,\x22version\x22:\x2225.0.1\x22},\x22openssl_version\x22:\x22OpenSSL 3.4.1 11 Feb 2025\x22,\x22python\x22:\x223.12.9\x22,\x22rustc_version\x22:\x221.85.0\x22,\x22setuptools_version\x22:\x2275.8.0\x22,\x22system\x22:{\x22name\x22:\x22Darwin\x22,\x22release\x22:\x2223.6.0\x22}}"

172.20.66.33 - - [05/Mar/2025:11:57:52 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:58:02 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:58:12 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:58:22 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

10.52.0.136 - - [05/Mar/2025:11:58:24 +0000] "GET /pypi/local-pypi/simple/keras/ HTTP/1.1" 200 59045 "-" "pip/24.3.1 {\x22ci\x22:null,\x22cpu\x22:\x22x86_64\x22,\x22distro\x22:{\x22name\x22:\x22macOS\x22,\x22version\x22:\x2215.3.1\x22},\x22implementation\x22:{\x22name\x22:\x22CPython\x22,\x22version\x22:\x223.13.1\x22},\x22installer\x22:{\x22name\x22:\x22pip\x22,\x22version\x22:\x2224.3.1\x22},\x22openssl_version\x22:\x22OpenSSL 3.4.0 22 Oct 2024\x22,\x22python\x22:\x223.13.1\x22,\x22system\x22:{\x22name\x22:\x22Darwin\x22,\x22release\x22:\x2224.3.0\x22}}"

172.20.66.33 - - [05/Mar/2025:11:58:32 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

172.20.66.33 - - [05/Mar/2025:11:58:42 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"

10.52.0.136 - - [05/Mar/2025:11:58:47 +0000] "GET /pypi/local-pypi/simple/keras/ HTTP/1.1" 200 59045 "-" "pip/22.0.2 {\x22ci\x22:null,\x22cpu\x22:\x22x86_64\x22,\x22distro\x22:{\x22id\x22:\x22jammy\x22,\x22libc\x22:{\x22lib\x22:\x22glibc\x22,\x22version\x22:\x222.35\x22},\x22name\x22:\x22Ubuntu\x22,\x22version\x22:\x2222.04\x22},\x22implementation\x22:{\x22name\x22:\x22CPython\x22,\x22version\x22:\x223.10.12\x22},\x22installer\x22:{\x22name\x22:\x22pip\x22,\x22version\x22:\x2222.0.2\x22},\x22openssl_version\x22:\x22OpenSSL 3.0.2 15 Mar 2022\x22,\x22python\x22:\x223.10.12\x22,\x22setuptools_version\x22:\x2259.6.0\x22,\x22system\x22:{\x22name\x22:\x22Linux\x22,\x22release\x22:\x225.15.0-133-generic\x22}}"

2025/03/05 11:58:47 [warn] 8#8: *74 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/2/00/0000000002 while reading upstream, client: 10.52.0.136, server: pulp-project-web-67bf57796b-74cqz, request: "GET /pulp/content/local-pypi/keras-3.9.0-py3-none-any.whl?redirect=https://files.pythonhosted.org/packages/2b/98/e81c6b2cb522f0eadcc8e16f3cabaccd5462bff6cf52194acfed4a031d3f/keras-3.9.0-py3-none-any.whl HTTP/1.1", upstream: "http://10.53.224.206:24816/pulp/content/local-pypi/keras-3.9.0-py3-none-any.whl?redirect=https://files.pythonhosted.org/packages/2b/98/e81c6b2cb522f0eadcc8e16f3cabaccd5462bff6cf52194acfed4a031d3f/keras-3.9.0-py3-none-any.whl", host: "pulp-prod.k8s.com"

10.52.0.136 - - [05/Mar/2025:11:58:47 +0000] "GET /pulp/content/local-pypi/keras-3.9.0-py3-none-any.whl?redirect=https://files.pythonhosted.org/packages/2b/98/e81c6b2cb522f0eadcc8e16f3cabaccd5462bff6cf52194acfed4a031d3f/keras-3.9.0-py3-none-any.whl HTTP/1.1" 200 1341511 "-" "pip/22.0.2 {\x22ci\x22:null,\x22cpu\x22:\x22x86_64\x22,\x22distro\x22:{\x22id\x22:\x22jammy\x22,\x22libc\x22:{\x22lib\x22:\x22glibc\x22,\x22version\x22:\x222.35\x22},\x22name\x22:\x22Ubuntu\x22,\x22version\x22:\x2222.04\x22},\x22implementation\x22:{\x22name\x22:\x22CPython\x22,\x22version\x22:\x223.10.12\x22},\x22installer\x22:{\x22name\x22:\x22pip\x22,\x22version\x22:\x2222.0.2\x22},\x22openssl_version\x22:\x22OpenSSL 3.0.2 15 Mar 2022\x22,\x22python\x22:\x223.10.12\x22,\x22setuptools_version\x22:\x2259.6.0\x22,\x22system\x22:{\x22name\x22:\x22Linux\x22,\x22release\x22:\x225.15.0-133-generic\x22}}"

10.52.0.136 - - [05/Mar/2025:11:58:48 +0000] "GET /pypi/local-pypi/simple/optree/ HTTP/1.1" 200 483834 "-" "pip/22.0.2 {\x22ci\x22:null,\x22cpu\x22:\x22x86_64\x22,\x22distro\x22:{\x22id\x22:\x22jammy\x22,\x22libc\x22:{\x22lib\x22:\x22glibc\x22,\x22version\x22:\x222.35\x22},\x22name\x22:\x22Ubuntu\x22,\x22version\x22:\x2222.04\x22},\x22implementation\x22:{\x22name\x22:\x22CPython\x22,\x22version\x22:\x223.10.12\x22},\x22installer\x22:{\x22name\x22:\x22pip\x22,\x22version\x22:\x2222.0.2\x22},\x22openssl_version\x22:\x22OpenSSL 3.0.2 15 Mar 2022\x22,\x22python\x22:\x223.10.12\x22,\x22setuptools_version\x22:\x2259.6.0\x22,\x22system\x22:{\x22name\x22:\x22Linux\x22,\x22release\x22:\x225.15.0-133-generic\x22}}"

kubectl logs pulp-project-content-7c4d596bd9-4znpl -n pulp-prod

Waiting on postgresql to start...
Postgres started.
Checking for database migrations
Database migrated!
/usr/local/bin/pulpcore-content
[2025-03-05 11:53:42 +0000] [1] [INFO] Starting gunicorn 23.0.0
[2025-03-05 11:53:42 +0000] [1] [INFO] Listening at: http://[::]:24816 (1)
[2025-03-05 11:53:42 +0000] [1] [INFO] Using worker: aiohttp.GunicornWebWorker
[2025-03-05 11:53:42 +0000] [28] [INFO] Booting worker with pid: 28
[2025-03-05 11:53:42 +0000] [30] [INFO] Booting worker with pid: 30
[2025-03-05 11:57:45 +0000] [30] [ERROR] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_protocol.py", line 480, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_app.py", line 569, in _handle
    return await handler(request)
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py", line 48, in authenticate
    return await handler(request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 354, in cached_function
    response = await self.make_entry(
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 400, in make_entry
    response = await handler(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 268, in stream_content
    return await self._match_and_stream(path, request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 778, in _match_and_stream
    return await self._stream_remote_artifact(
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 1146, in _stream_remote_artifact
    download_result = await downloader.run(
  File "/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py", line 274, in run
    return await download_wrapper()
  File "/usr/local/lib/python3.9/site-packages/backoff/_async.py", line 151, in retry
    ret = await target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py", line 259, in download_wrapper
    return await self._run(extra_data=extra_data)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py", line 296, in _run
    to_return = await self._handle_response(response)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py", line 207, in _handle_response
    await self.handle_data(chunk)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 1129, in handle_data
    await response.write(data)
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_response.py", line 550, in write
    await self._payload_writer.write(data)
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/http_writer.py", line 133, in write
    self._write(chunk)
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/http_writer.py", line 81, in _write
    raise ClientConnectionResetError("Cannot write to closing transport")
aiohttp.client_exceptions.ClientConnectionResetError: Cannot write to closing transport
::ffff:10.52.1.128 [05/Mar/2025:11:58:47 +0000] "GET /pulp/content/local-pypi/keras-3.9.0-py3-none-any.whl?redirect=https://files.pythonhosted.org/packages/2b/98/e81c6b2cb522f0eadcc8e16f3cabaccd5462bff6cf52194acfed4a031d3f/keras-3.9.0-py3-none-any.whl HTTP/1.0" 200 309 "-" "pip/22.0.2 {"ci":null,"cpu":"x86_64","distro":{"id":"jammy","libc":{"lib":"glibc","version":"2.35"},"name":"Ubuntu","version":"22.04"},"implementation":{"name":"CPython","version":"3.10.12"},"installer":{"name":"pip","version":"22.0.2"},"openssl_version":"OpenSSL 3.0.2 15 Mar 2022","python":"3.10.12","setuptools_version":"59.6.0","system":{"name":"Linux","release":"5.15.0-133-generic"}}"

kubectl logs pulp-project-api-56b8d69556-xx8cp -n pulp-prod

Waiting on postgresql to start...

Postgres started.

Checking for database migrations

Database migrated!

/usr/local/bin/pulpcore-api

[2025-03-05 11:53:08 +0000] [1] [INFO] Starting gunicorn 23.0.0

[2025-03-05 11:53:08 +0000] [1] [INFO] Listening at: http://[::]:24817 (1)

[2025-03-05 11:53:08 +0000] [1] [INFO] Using worker: pulpcore.app.entrypoint.PulpApiWorker

[2025-03-05 11:53:08 +0000] [14] [INFO] Booting worker with pid: 14

[2025-03-05 11:53:08 +0000] [15] [INFO] Booting worker with pid: 15

('pulp [3104cdcef0184c8cbca28bf717832144]: ::ffff:172.20.66.33 - - [05/Mar/2025:11:53:12 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 5418 "-" "kube-probe/1.27"',)

('pulp [bfe7b9c56c5142ec899e9f186ca30232]: ::ffff:10.52.1.122 - - [05/Mar/2025:11:53:15 +0000] "GET /pulp/api/v3/status/ HTTP/1.0" 200 4856 "-" "kube-probe/1.27"',)

/usr/local/lib/python3.9/site-packages/pypi_simple/util.py:25: UnexpectedRepoVersionWarning: Repository's version (1.3) has greater minor component than supported version (1.0)

warnings.warn(

('pulp [996f5ce65e354e358a242049d238243a]: ::ffff:10.52.1.122 - - [05/Mar/2025:11:53:16 +0000] "GET /pypi/local-pypi/simple/keras/ HTTP/1.0" 200 58922 "-" "pip/25.0.1 {\"ci\":null,\"cpu\":\"arm64\",\"distro\":{\"name\":\"macOS\",\"version\":\"14.7.4\"},\"implementation\":{\"name\":\"CPython\",\"version\":\"3.12.9\"},\"installer\":{\"name\":\"pip\",\"version\":\"25.0.1\"},\"openssl_version\":\"OpenSSL 3.4.1 11 Feb 2025\",\"python\":\"3.12.9\",\"rustc_version\":\"1.85.0\",\"setuptools_version\":\"75.8.0\",\"system\":{\"name\":\"Darwin\",\"release\":\"23.6.0\"}}"',)

('pulp [4fce26411ff54308bfb19c9610624eec]: ::ffff:10.52.1.122 - - [05/Mar/2025:11:53:19 +0000] "GET /pypi/local-pypi/simple/keras/ HTTP/1.0" 200 58922 "-" "pip/25.0.1 {\"ci\":null,\"cpu\":\"arm64\",\"distro\":{\"name\":\"macOS\",\"version\":\"14.7.4\"},\"implementation\":{\"name\":\"CPython\",\"version\":\"3.12.9\"},\"installer\":{\"name\":\"pip\",\"version\":\"25.0.1\"},\"openssl_version\":\"OpenSSL 3.4.1 11 Feb 2025\",\"python\":\"3.12.9\",\"rustc_version\":\"1.85.0\",\"setuptools_version\":\"75.8.0\",\"system\":{\"name\":\"Darwin\",\"release\":\"23.6.0\"}}"',)

('pulp [58bd1d72f5a04ff4880b4361abae4c55]: ::ffff:172.20.66.33 - - [05/Mar/2025:11:53:31 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4855 "-" "kube-probe/1.27"',)

/usr/local/lib/python3.9/site-packages/pypi_simple/util.py:25: UnexpectedRepoVersionWarning: Repository's version (1.3) has greater minor component than supported version (1.0)

warnings.warn(

('pulp [c04f5d19065d4b088264af2a42f5e29a]: ::ffff:10.52.1.128 - - [05/Mar/2025:11:56:24 +0000] "GET /pypi/local-pypi/simple/keras/ HTTP/1.0" 200 58922 "-" "pip/25.0.1 {\"ci\":null,\"cpu\":\"arm64\",\"distro\":{\"name\":\"macOS\",\"version\":\"14.7.4\"},\"implementation\":{\"name\":\"CPython\",\"version\":\"3.12.9\"},\"installer\":{\"name\":\"pip\",\"version\":\"25.0.1\"},\"openssl_version\":\"OpenSSL 3.4.1 11 Feb 2025\",\"python\":\"3.12.9\",\"rustc_version\":\"1.85.0\",\"setuptools_version\":\"75.8.0\",\"system\":{\"name\":\"Darwin\",\"release\":\"23.6.0\"}}"',)

('pulp [adb16a9d03cc4757a82d75ae38d5cf6c]: ::ffff:172.20.66.33 - - [05/Mar/2025:11:56:31 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"',)

('pulp [0581e864b64a47a8ab822697b37d5871]: ::ffff:10.52.1.128 - - [05/Mar/2025:11:56:32 +0000] "GET /pulp/api/v3/status/ HTTP/1.0" 200 3191 "-" "kube-probe/1.27"',)

('pulp [1041211601f84b73a4b7e835261d46a1]: ::ffff:10.52.1.128 - - [05/Mar/2025:11:56:39 +0000] "GET /pypi/local-pypi/simple/tensorflow/ HTTP/1.0" 200 331966 "-" "pip/25.0.1 {\"ci\":null,\"cpu\":\"arm64\",\"distro\":{\"name\":\"macOS\",\"version\":\"14.7.4\"},\"implementation\":{\"name\":\"CPython\",\"version\":\"3.12.9\"},\"installer\":{\"name\":\"pip\",\"version\":\"25.0.1\"},\"openssl_version\":\"OpenSSL 3.4.1 11 Feb 2025\",\"python\":\"3.12.9\",\"rustc_version\":\"1.85.0\",\"setuptools_version\":\"75.8.0\",\"system\":{\"name\":\"Darwin\",\"release\":\"23.6.0\"}}"',)

('pulp [13daf4c41ee0414fa6f174217dcc6c39]: ::ffff:10.52.1.128 - - [05/Mar/2025:11:57:22 +0000] "GET /pulp/api/v3/status/ HTTP/1.0" 200 3191 "-" "kube-probe/1.27"',)

('pulp [67fee173c8094a2fa503eb68bf3ee579]: ::ffff:10.52.1.128 - - [05/Mar/2025:11:57:29 +0000] "GET /pypi/local-pypi/simple/tensorflow/ HTTP/1.0" 200 739977 "-" "pip/25.0.1 {\"ci\":null,\"cpu\":\"arm64\",\"distro\":{\"name\":\"macOS\",\"version\":\"14.7.4\"},\"implementation\":{\"name\":\"CPython\",\"version\":\"3.12.9\"},\"installer\":{\"name\":\"pip\",\"version\":\"25.0.1\"},\"openssl_version\":\"OpenSSL 3.4.1 11 Feb 2025\",\"python\":\"3.12.9\",\"rustc_version\":\"1.85.0\",\"setuptools_version\":\"75.8.0\",\"system\":{\"name\":\"Darwin\",\"release\":\"23.6.0\"}}"',)

('pulp [4f64d363109d4d9698edf4c845db836a]: ::ffff:10.52.1.128 - - [05/Mar/2025:11:58:24 +0000] "GET /pypi/local-pypi/simple/keras/ HTTP/1.0" 200 58922 "-" "pip/24.3.1 {\"ci\":null,\"cpu\":\"x86_64\",\"distro\":{\"name\":\"macOS\",\"version\":\"15.3.1\"},\"implementation\":{\"name\":\"CPython\",\"version\":\"3.13.1\"},\"installer\":{\"name\":\"pip\",\"version\":\"24.3.1\"},\"openssl_version\":\"OpenSSL 3.4.0 22 Oct 2024\",\"python\":\"3.13.1\",\"system\":{\"name\":\"Darwin\",\"release\":\"24.3.0\"}}"',)

('pulp [d72b344585634c7dbf91f2673c293be4]: ::ffff:172.20.66.33 - - [05/Mar/2025:11:58:31 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3191 "-" "kube-probe/1.27"',)

('pulp [02b5150b8b22428f9d1dd61ef6f6127b]: ::ffff:10.52.1.128 - - [05/Mar/2025:11:58:32 +0000] "GET /pulp/api/v3/status/ HTTP/1.0" 200 3191 "-" "kube-probe/1.27"',)

('pulp [9332316d68614ac08b2a50e6753b8cc5]: ::ffff:10.52.1.128 - - [05/Mar/2025:11:58:42 +0000] "GET /pulp/api/v3/status/ HTTP/1.0" 200 3191 "-" "kube-probe/1.27"',)

('pulp [c8dd774ecda647e7aafb6e25b9f7a548]: ::ffff:10.52.1.128 - - [05/Mar/2025:11:58:47 +0000] "GET /pypi/local-pypi/simple/keras/ HTTP/1.0" 200 58922 "-" "pip/22.0.2 {\"ci\":null,\"cpu\":\"x86_64\",\"distro\":{\"id\":\"jammy\",\"libc\":{\"lib\":\"glibc\",\"version\":\"2.35\"},\"name\":\"Ubuntu\",\"version\":\"22.04\"},\"implementation\":{\"name\":\"CPython\",\"version\":\"3.10.12\"},\"installer\":{\"name\":\"pip\",\"version\":\"22.0.2\"},\"openssl_version\":\"OpenSSL 3.0.2 15 Mar 2022\",\"python\":\"3.10.12\",\"setuptools_version\":\"59.6.0\",\"system\":{\"name\":\"Linux\",\"release\":\"5.15.0-133-generic\"}}"',)

('pulp [fa990e4b17984955a491495c412626bd]: ::ffff:10.52.1.128 - - [05/Mar/2025:11:58:48 +0000] "GET /pypi/local-pypi/simple/optree/ HTTP/1.0" 200 482887 "-" "pip/22.0.2 {\"ci\":null,\"cpu\":\"x86_64\",\"distro\":{\"id\":\"jammy\",\"libc\":{\"lib\":\"glibc\",\"version\":\"2.35\"},\"name\":\"Ubuntu\",\"version\":\"22.04\"},\"implementation\":{\"name\":\"CPython\",\"version\":\"3.10.12\"},\"installer\":{\"name\":\"pip\",\"version\":\"22.0.2\"},\"openssl_version\":\"OpenSSL 3.0.2 15 Mar 2022\",\"python\":\"3.10.12\",\"setuptools_version\":\"59.6.0\",\"system\":{\"name\":\"Linux\",\"release\":\"5.15.0-133-generic\"}}"',)

Pypi configuration script:

set -x
# Start by creating a new repository named "local-pypi":
pulp python repository create --name local-pypi

# Create a remote that syncs some versions of shelf-reader into your repository.
pulp python remote create \
  --name remote-pypi \
  --url https://pypi.org/ \
  --policy on_demand

# Link the remote to the repository
pulp python repository update \
  --name local-pypi \
  --remote remote-pypi

# Using the Remote we just created, we kick off a sync task
pulp python repository sync \
  --name local-pypi \
  --remote remote-pypi

# Create a new publication specifying the repository_version.
pulp python publication create --repository local-pypi

# Publications can only be referenced through their pulp_href
PUBLICATION_HREF=$(pulp python publication list | jq -r .[0].pulp_href)

# Distributions are created asynchronously. Create one, and specify the publication that will
# be served at the base path specified.
pulp python distribution create --name local-pypi-distro --base-path local-pypi --publication "$PUBLICATION_HREF"

# This configures the repository to produce new publications when a new version is created
pulp python repository update \
  --name local-pypi \
  --autopublish

# Link the remote to the distribution for pull-through caching
pulp python distribution update \
  --name local-pypi-distro \
  --remote remote-pypi

I guess that my ingress configuration is not quite correct which causes the download to fail.

Thanks in advance :slight_smile:

#2

I have also tried to follow the following ingress instructions:
https://pulpproject.org/pulp-operator/docs/admin/guides/configurations/networking/exposing/

new Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-ingress
  namespace: {{ .Release.Namespace }}
spec:
  ingressClassName: nginx-apps
  rules:
  - host: "{{ .Release.Namespace }}.{{ .Values.global.domain }}"
    http:
      paths:
      - backend:
          service:
            name: pulp-project-content-svc
            port:
              number: 24816
        path: /pulp/content/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              number: 24817
        path: /pulp/api/v3/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              number: 24817
        path: /auth/login/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              number: 24817
        path: /
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              number: 24817
        path: /pulp_ansible/galaxy/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              number: 24817
        path: /v2/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              number: 24817
        path: /extensions/v2/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-content-svc
            port:
              number: 24816
        path: /pulp/container/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              number: 24817
        path: /token/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              number: 24817
        path: /pypi/
        pathType: Prefix
    #   - backend:
    #       service:
    #         name: pulp-project-web-svc 
    #         port:
    #           number: 24880
    #     path: /
    #     pathType: Prefix
    #   - backend:
    #       service:
    #         name: pulp-project-api-svc
    #         port:
    #           number: 24817
    #     path: /pulp/api/v3/
    #     pathType: Prefix
  tls:
  - hosts: 
    - "{{ .Release.Namespace }}.{{ .Values.global.domain }}"
    secretName: {{ .Release.Name }}-tls-secret

I have also added to the Pulp kind:

spec:
  ingress_type: ingress

and I have received the following error upon pip3 install:

pip._vendor.urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(1 bytes read, 1 more expected)', IncompleteRead(1 bytes read, 1 more expected))

AND

pip._vendor.urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
#3

Hi @Eliko

To make sure the issue is really in the ingress config, let’s try to first disable cache and see if the error changes (from a quick look at the stack trace, it seems like it failed during a cache operation):

spec:
...
  cache:
    enabled: false
...

after modifying the CR, the operator should trigger a reconciliation and we can check if cache is now disabled through:

$ pulp status

now, try to install the python package again and let’s see if the exception in pulp-content logs is the same.

note: based on the ingressClassName, I guess this cluster has a nginx ingress controller, right? If so, have you also tried to set is_nginx_ingress: true (we are lacking doc for this config), for example:

spec:
...
  ingress_type: ingress
  ingress_class_name: nginx-apps
  ingress_host: ...
  is_nginx_ingress: true
...
#4

@hyagi

I have disabled the cache:

“database_connection”: {
“connected”: true
},
“redis_connection”: {
“connected”: false
},
“storage”: {
“total”: 1099511627776,
“used”: 90177536,
“free”: 1099421450240
},
“content_settings”: {
“content_origin”: “https://pulp-prod.k8s.com”,
“content_path_prefix”: “/pulp/content/”
},
“domain_enabled”: false
}

I receive the same error upon installation:

ERROR: Could not install packages due to an OSError: (“Connection broken: InvalidChunkLength(got length b’’, 0 bytes read)”, InvalidChunkLength(got length b’’, 0 bytes read))

kubectl logs pulp-project-content-87b6f7694-mf4gt -n pulp-prod

Waiting on postgresql to start…
Postgres started.
Checking for database migrations
Database migrated!
/usr/local/bin/pulpcore-content
[2025-03-06 09:08:59 +0000] [1] [INFO] Starting gunicorn 23.0.0
[2025-03-06 09:08:59 +0000] [1] [INFO] Listening at: http://[::]:24816 (1)
[2025-03-06 09:08:59 +0000] [1] [INFO] Using worker: aiohttp.GunicornWebWorker
[2025-03-06 09:08:59 +0000] [22] [INFO] Booting worker with pid: 22
[2025-03-06 09:09:00 +0000] [24] [INFO] Booting worker with pid: 24
[2025-03-06 09:09:09 +0000] [22] [ERROR] Error handling request
Traceback (most recent call last):
File “/usr/lib64/python3.9/asyncio/selector_events.py”, line 854, in _read_ready__data_received
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_protocol.py”, line 480, in _handle_request
resp = await request_handler(request)
File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_app.py”, line 569, in _handle
return await handler(request)
File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_middlewares.py”, line 117, in impl
return await handler(request)
File “/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py”, line 48, in authenticate
return await handler(request)
File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 268, in stream_content
return await self._match_and_stream(path, request)
File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 778, in _match_and_stream
return await self._stream_remote_artifact(
File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 1146, in _stream_remote_artifact
download_result = await downloader.run(
File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 274, in run
return await download_wrapper()
File “/usr/local/lib/python3.9/site-packages/backoff/_async.py”, line 151, in retry
ret = await target(*args, **kwargs)
File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 259, in download_wrapper
return await self._run(extra_data=extra_data)
File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 296, in _run
to_return = await self._handle_response(response)
File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 207, in _handle_response
await self.handle_data(chunk)
File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 1129, in handle_data
await response.write(data)
File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_response.py”, line 550, in write
await self._payload_writer.write(data)
File “/usr/local/lib64/python3.9/site-packages/aiohttp/http_writer.py”, line 137, in write
await self.drain()
File “/usr/local/lib64/python3.9/site-packages/aiohttp/http_writer.py”, line 203, in drain
await protocol._drain_helper()
File “/usr/local/lib64/python3.9/site-packages/aiohttp/base_protocol.py”, line 100, in _drain_helper
await asyncio.shield(waiter)
ConnectionError: Connection lost

and I have added the following to my Pulp:

ingress_type: ingress
ingress_class_name: nginx-apps
ingress_host: “{{ .Release.Namespace }}.{{ .Values.global.domain }}”
is_nginx_ingress: true

#5

Hi @Eliko

hmm… so with and without cache we are seeing the “client” (or something in the middle) closing/resetting the connection:

  • with cache
aiohttp.client_exceptions.ClientConnectionResetError: Cannot write to closing transport
  • without cache
ConnectionResetError: [Errno 104] Connection reset by peer
...
ConnectionError: Connection lost

which, as you suspected, can be caused by the ingress controller.
Now that you updated Pulp CR with the ingress configurations, could you please verify if the following annotations were added to it?

$ oc get ingress <ingress name> -oyaml
...
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: ...
    nginx.org/client-max-body-size: ...
    nginx.ingress.kubernetes.io/proxy-read-timeout: ...
    nginx.ingress.kubernetes.io/proxy-connect-timeout: ...
    nginx.ingress.kubernetes.io/proxy-send-timeout: ...
...

and also could you check the nginx logs (not from pulp-web, but the ingress controller) to see if we can get more clues?

2 Likes
#6

@hyagi
Alright, so here are the nginx ingress controller logs (related to pulp, other logs are ignored):

W0309 08:27:34.021016 7 controller.go:331] ignoring ingress pulp-project in pulp-prod based on annotation : no object matching key “nginx-apps” in local store
I0309 08:27:34.021048 7 main.go:107] “successfully validated configuration, accepting” ingress=“pulp-prod/pulp-project”

I have noticed that pulp doesn’t recognize the nginx type so I have switched to “nginx”, once I have applied the changes it created me a new “ingress” object (where I have my own ingress object in my helm chart).

Here is the new ingress that was created by pulp:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: 120s
    nginx.ingress.kubernetes.io/proxy-read-timeout: 120s
    nginx.ingress.kubernetes.io/proxy-send-timeout: 120s
    nginx.org/client-max-body-size: 10m
    web: "false"
  creationTimestamp: "2025-03-09T08:39:02Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: pulp-operator
    app.kubernetes.io/part-of: pulp
    pulp_cr: pulp-project
  name: pulp-project
  namespace: pulp-prod
  ownerReferences:
  - apiVersion: repo-manager.pulpproject.org/v1beta2
    blockOwnerDeletion: true
    controller: true
    kind: Pulp
    name: pulp-project
    uid: 199ab529-d974-4b87-a43f-3d11a8510221
  resourceVersion: "252443184"
  uid: 0053a309-e7da-4083-8dd9-8cca0789c884
spec:
  ingressClassName: nginx
  rules:
  - host: pulp-prod.k8s.com
    http:
      paths:
      - backend:
          service:
            name: pulp-project-content-svc
            port:
              name: content-24816
        path: /pulp/content/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              name: api-24817
        path: /pulp/api/v3/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              name: api-24817
        path: /auth/login/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              name: api-24817
        path: /
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              name: api-24817
        path: /pulp_ansible/galaxy/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              name: api-24817
        path: /v2/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              name: api-24817
        path: /extensions/v2/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-content-svc
            port:
              name: content-24816
        path: /pulp/container/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              name: api-24817
        path: /token/
        pathType: Prefix
      - backend:
          service:
            name: pulp-project-api-svc
            port:
              name: api-24817
        path: /pypi/
        pathType: Prefix

Update: After few minutes of sync it works now with Ingress! Is there an option to add annotation for that pulp to use secret for certificates?

I have tried to edit the ingress and add the use of certificate manually but I get 404 when I try to access the /api/status/v3 endpoint and I can’t access the web ui without certificate. Also when I am looking at the created ingress I see an annotation: web: “false”

1 Like
#7

Update: I am using custom nginx controller, I have removed the ingress I have created in the helm chart, added the nginx annotations and it created the ingress resource. I am able to access the UI but when I try to install packages, sometimes it works and fails upon Connection reset (54) sometimes it still says that that’s no matching version for the given distribution when I am trying to install generic python package (tensorflow, keras, pandas) without specific version.

#8

Update: After few minutes of sync it works now with Ingress! Is there an option to add annotation for that pulp to use secret for certificates?

Glad to know it worked with Ingress!
Yes, you can pass the ingress_annotations to Pulp CR, for example:

spec:
...
  ingress_type: ingress
  ingress_class_name: nginx-apps
  ingress_host: ...
  is_nginx_ingress: true
  ingress_annotations:
    my-custom-annotation-1: my-custom-value-1
    my-custom-annotation-2: my-custom-value-2
    <string>: <string>
    ...
...

Also when I am looking at the created ingress I see an annotation: web: “false”

If you are seeing this, it means you configured your Pulp CR with is_nginx_ingress: true, pulp-operator recognized that, and did not deploy pulp-web pods (expected behavior) to avoid a “double layer” proxy (ingress + pulp-web).

Update: I am using custom nginx controller, I have removed the ingress I have created in the helm chart, added the nginx annotations and it created the ingress resource. I am able to access the UI but when I try to install packages, sometimes it works and fails upon Connection reset (54) sometimes it still says that that’s no matching version for the given distribution when I am trying to install generic python package (tensorflow, keras, pandas) without specific version.

I am not sure if I understand how things are.
1- you are not using the ingress deployed by pulp-operator anymore and created one by yourself (using helm charts), right?
2- when you tried to sync using the ingress deployed by pulp-operator it worked, but you had to add some annotations and, since you could not do it, you decided to create another ingress manually and with this other ingress it is failing to pull the packages from pulp, correct?