Pulp status giving 502 error

Support
#1

Pulp status was working fine until I reboot the host. after the reboot I am getting below 502 error and below services that are now failing. Could someone please help on this.

please let me know if you need any more info.

Failing services after reboot:
pulpcore-api.service
pulpcore-worker@1.service
pulpcore-worker@2.service

(pulp) [root@prodjp-yumrepo-s1-z2-0 ext_a_reghatte_sap_com]# pulp status
status_read : get https://prodjp-yumrepo-s1-z2-0.jp1.gcpint.ariba.com/pulp/api/v3/status/
Response: 502
Error:

502 Bad Gateway

502 Bad Gateway

nginx

Failing services:

● pulpcore-api.service - Pulp WSGI Server
Loaded: loaded (/usr/lib/systemd/system/pulpcore-api.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2024-11-11 07:52:40 PST; 99ms ago
Process: 16211 ExecStart=/usr/local/lib/pulp/bin/gunicorn pulpcore.app.wsgi:application --name pulp-api --bind 127.0.0.1:24817 --workers 1 --timeout 90 (code=exited, status=>
Main PID: 16211 (code=exited, status=3)
Status: “Gunicorn arbiter booted”
CPU: 176ms

● pulpcore-worker@1.service - Pulp Worker
Loaded: loaded (/usr/lib/systemd/system/pulpcore-worker@.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2024-11-11 07:52:39 PST; 820ms ago
Process: 16209 ExecStart=/usr/local/lib/pulp/bin/pulpcore-worker (code=exited, status=1/FAILURE)
Main PID: 16209 (code=exited, status=1/FAILURE)
CPU: 38ms

● pulpcore-worker@2.service - Pulp Worker
Loaded: loaded (/usr/lib/systemd/system/pulpcore-worker@.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2024-11-11 07:52:39 PST; 827ms ago
Process: 16210 ExecStart=/usr/local/lib/pulp/bin/pulpcore-worker (code=exited, status=1/FAILURE)
Main PID: 16210 (code=exited, status=1/FAILURE)
CPU: 38ms

(pulp) [root@prodjp-yumrepo-s1-z2-0 ext_a_reghatte_sap_com]# /usr/local/lib/pulp/bin/pulpcore-worker
Traceback (most recent call last):
File “/usr/local/lib/pulp/bin/pulpcore-worker”, line 5, in
from pulpcore.tasking.entrypoint import worker
File “/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/entrypoint.py”, line 9, in
django.setup()
File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/init.py”, line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/apps/registry.py”, line 124, in populate
app_config.ready()
File “/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/app/apps.py”, line 126, in ready
self.import_viewsets()
File “/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/app/apps.py”, line 172, in import_viewsets
self.viewsets_module = import_module(viewsets_module_name)
File “/usr/lib64/python3.9/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_container/app/viewsets.py”, line 19, in
from pulpcore.plugin.actions import raise_for_unknown_content_units
ImportError: cannot import name ‘raise_for_unknown_content_units’ from ‘pulpcore.plugin.actions’ (/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/actions.py)
(pulp) [root@prodjp-yumrepo-s1-z2-0 ext_a_reghatte_sap_com]# pip show pulpcore pulp-container
Name: pulpcore
Version: 3.66.0
Summary: Pulp Django Application and Related Modules
Home-page: https://pulpproject.org
Author: Pulp Team
Author-email: pulp-list@redhat.com
License: GPLv2+
Location: /usr/local/lib/pulp/lib/python3.9/site-packages
Requires: aiodns, aiofiles, aiohttp, async-timeout, asyncio-throttle, backoff, click, cryptography, Django, django-filter, django-guid, django-import-export, django-lifecycle, djangorestframework, djangorestframework-queryfields, drf-access-policy, drf-nested-routers, drf-spectacular, dynaconf, gunicorn, importlib-metadata, jinja2, jq, json-stream, opentelemetry-distro, opentelemetry-exporter-otlp-proto-http, opentelemetry-instrumentation-aiohttp-server, opentelemetry-instrumentation-wsgi, protobuf, psycopg, pulp-glue, pygtrie, PyOpenSSL, pyparsing, python-gnupg, PyYAML, redis, tablib, url-normalize, uuid6, whitenoise, yarl
Required-by: pulp-container, pulp-file, pulp-rpm

Name: pulp-container
Version: 2.14.16
Summary: Container plugin for the Pulp Project
Home-page: https://pulpproject.org/
Author: Pulp Team
Author-email: pulp-list@redhat.com
License: GPLv2+
Location: /usr/local/lib/pulp/lib/python3.9/site-packages
Requires: jsonschema, pulpcore, pyjwkest, pyjwt, tablib
Required-by:

#2

It looks like you are using a very old version of pulp_container. (One that is definitely not safe to use with the latest version of pulpcore.)
Can you elaborate a bit on the “after reboot” part? Specifically, how do you upgrade the software on reboot?

1 Like
#3

before reboot pulp status was fine but we were getting below error when we tried to create a repo, we thought of rebooting host to fix the issue but after the reboot pulp status also getting same 502 error. pulp was working until we installed pulp file, after installing it we are seeing 502 error issue.

Please let me know if you need any more details.

Issue before reboot: (pulp) [root@prodjp-yumrepo-s1-z2-0 ~]# pulp rpm repository create --name test
repositories_rpm_rpm_create : post https://prodjp-yumrepo-s1-z2-0.jp1.gcpint.ariba.com/pulp/api/v3/repositories/rpm/rpm/
Response: 502
Error:

502 Bad Gateway

502 Bad Gateway

nginx

(pulp) [root@prodjp-yumrepo-s1-z2-0 ext_a_reghatte_sap_com]# pip list |grep -i pulp
pulp-cli 0.29.2
pulp-container 2.14.16
pulp-file 1.16.0
pulp-glue 0.29.2
pulp-rpm 3.19.12
pulpcore 3.66.0

#4

The pulp-file plugin is built-in pulpcore since pulpcore==3.40.0 and installing this older standalone version on top can lead to unpredictable behavior (like this maybe? No idea).

edit: actually pulp-file 1.16 is a dummy package and contains nothing (to avoid this very kind of conflict). Still:

Since this is what caused the problem, can you try removing the pulp-file package?

Do you have a particular reason for choosing those versions? You really should ensure the plugins are compatible, because that can break things badly.

#5

Well still, only upgrading pulp_container to something recent will solve your problem at this time.
I’d suggest to not install pulp-cli in the server environment. (You can install it elsewhere or in a separate virtual environment. It’s supposed to interact via the api and so works remote…) Further i suggest to pip install --upgrade pulpcore pulp_container pulp_rpm in a single call (every other way, pip can leave the sytem in an undefined state.).

2 Likes
#6

after removing it all services are failing, There is no reason we are just trying to setup pulp3 newly and while installation it automatically installed those versions

(pulp) [root@prodjp-yumrepo-s1-z2-0 ext_a_reghatte_sap_com]# pip uninstall pulp-file
Found existing installation: pulp-file 1.16.0
Uninstalling pulp-file-1.16.0:
Would remove:
/usr/local/lib/pulp/lib/python3.9/site-packages/pulp_file-1.16.0.dist-info/*
Proceed (Y/n)? y
Successfully uninstalled pulp-file-1.16.0

(pulp) [root@prodjp-yumrepo-s1-z2-0 ext_a_reghatte_sap_com]# systemctl status pulpcore-api.service pulpcore-content.service pulpcore-worker@1.service pulpcore-worker@2.service nginx.service
● pulpcore-api.service - Pulp WSGI Server
Loaded: loaded (/usr/lib/systemd/system/pulpcore-api.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2024-11-12 12:07:28 PST; 2s ago
Process: 1892761 ExecStart=/usr/local/lib/pulp/bin/gunicorn pulpcore.app.wsgi:application --name pulp-api --bind 127.0.0.1:24817 --workers 1 --timeout 90 (code=exited, status=3)
Main PID: 1892761 (code=exited, status=3)
Status: “Gunicorn arbiter booted”
CPU: 177ms

● pulpcore-content.service - Pulp Content App
Loaded: loaded (/usr/lib/systemd/system/pulpcore-content.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2024-11-12 12:07:30 PST; 1s ago
Process: 1892864 ExecStart=/usr/local/lib/pulp/bin/gunicorn pulpcore.content:server --name pulp-content --bind 127.0.0.1:24816 --worker-class aiohttp.GunicornWebWorker --workers 8 --timeout 90 --access-logfile - (code=exited, status=3)
Main PID: 1892864 (code=exited, status=3)
Status: “Gunicorn arbiter booted”
CPU: 356ms

● pulpcore-worker@1.service - Pulp Worker
Loaded: loaded (/usr/lib/systemd/system/pulpcore-worker@.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2024-11-12 12:07:28 PST; 2s ago
Process: 1892762 ExecStart=/usr/local/lib/pulp/bin/pulpcore-worker (code=exited, status=1/FAILURE)
Main PID: 1892762 (code=exited, status=1/FAILURE)
CPU: 39ms

● pulpcore-worker@2.service - Pulp Worker
Loaded: loaded (/usr/lib/systemd/system/pulpcore-worker@.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2024-11-12 12:07:28 PST; 2s ago
Process: 1892763 ExecStart=/usr/local/lib/pulp/bin/pulpcore-worker (code=exited, status=1/FAILURE)
Main PID: 1892763 (code=exited, status=1/FAILURE)
CPU: 39ms

● nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: disabled)
Active: active (running) since Tue 2024-11-12 12:07:12 PST; 18s ago
Process: 1892598 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 1892601 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
Process: 1892602 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
Main PID: 1892604 (nginx)
Tasks: 2 (limit: 383816)
Memory: 3.2M
CPU: 42ms
CGroup: /system.slice/nginx.service
├─1892604 “nginx: master process /usr/sbin/nginx”
└─1892605 “nginx: worker process”

#7

I tried uninstalling pulp-file and pulp-cli , now all the services are failing except nginx & pulpcore.service

(pulp) [root@prodjp-yumrepo-s1-z2-0 ext_a_reghatte_sap_com]# pip uninstall pulp-file
Found existing installation: pulp-file 1.16.0
Uninstalling pulp-file-1.16.0:
Would remove:
/usr/local/lib/pulp/lib/python3.9/site-packages/pulp_file-1.16.0.dist-info/*
Proceed (Y/n)? y
Successfully uninstalled pulp-file-1.16.0

(pulp) [root@prodjp-yumrepo-s1-z2-0 ext_a_reghatte_sap_com]# pip uninstall pulp-cli
Found existing installation: pulp-cli 0.29.2
Uninstalling pulp-cli-0.29.2:
Would remove:
/usr/local/lib/pulp/bin/pulp
/usr/local/lib/pulp/lib/python3.9/site-packages/pulp_cli-0.29.2.dist-info/*
/usr/local/lib/pulp/lib/python3.9/site-packages/pulp_cli/*
/usr/local/lib/pulp/lib/python3.9/site-packages/pulpcore/cli/ansible/*
/usr/local/lib/pulp/lib/python3.9/site-packages/pulpcore/cli/common/*
/usr/local/lib/pulp/lib/python3.9/site-packages/pulpcore/cli/container/*
/usr/local/lib/pulp/lib/python3.9/site-packages/pulpcore/cli/core/*
/usr/local/lib/pulp/lib/python3.9/site-packages/pulpcore/cli/file/*
/usr/local/lib/pulp/lib/python3.9/site-packages/pulpcore/cli/migration/*
/usr/local/lib/pulp/lib/python3.9/site-packages/pulpcore/cli/python/*
/usr/local/lib/pulp/lib/python3.9/site-packages/pulpcore/cli/rpm/*
/usr/local/lib/pulp/lib/python3.9/site-packages/pytest_pulp_cli/*
Proceed (Y/n)? y
Successfully uninstalled pulp-cli-0.29.2

(pulp) [root@prodjp-yumrepo-s1-z2-0 ext_a_reghatte_sap_com]# pip list |grep -i pulp
pulp-container 2.22.0
pulp-glue 0.29.2
pulp-rpm 3.27.2
pulpcore 3.68.0

(pulp) [root@prodjp-yumrepo-s1-z2-0 ext_a_reghatte_sap_com]# systemctl status pulpcore.service pulpcore-api.service pulpcore-content.service pulpcore-worker@1.service pulpcore-worker@2.service nginx.service
● pulpcore.service - Pulpcore Application
Loaded: loaded (/usr/lib/systemd/system/pulpcore.service; enabled; preset: disabled)
Active: active (exited) since Tue 2024-11-12 12:42:19 PST; 27s ago
Process: 859 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 859 (code=exited, status=0/SUCCESS)
CPU: 1ms

● pulpcore-api.service - Pulp WSGI Server
Loaded: loaded (/usr/lib/systemd/system/pulpcore-api.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2024-11-12 12:42:46 PST; 925ms ago
Process: 5521 ExecStart=/usr/local/lib/pulp/bin/gunicorn pulpcore.app.wsgi:application --name pulp-api --bind 127.0.0.1:24817 --workers 1 --timeout 90 (code=exited, status=3)
Main PID: 5521 (code=exited, status=3)
Status: “Gunicorn arbiter booted”
CPU: 174ms

● pulpcore-content.service - Pulp Content App
Loaded: loaded (/usr/lib/systemd/system/pulpcore-content.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2024-11-12 12:42:46 PST; 860ms ago
Process: 5522 ExecStart=/usr/local/lib/pulp/bin/gunicorn pulpcore.content:server --name pulp-content --bind 127.0.0.1:24816 --worker-class aiohttp.GunicornWebWorker --workers 8 --timeout 90 --access-logfile - (code=exited, status=3)
Main PID: 5522 (code=exited, status=3)
Status: “Gunicorn arbiter booted”
CPU: 351ms

● pulpcore-worker@1.service - Pulp Worker
Loaded: loaded (/usr/lib/systemd/system/pulpcore-worker@.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2024-11-12 12:42:46 PST; 1s ago
Process: 5519 ExecStart=/usr/local/lib/pulp/bin/pulpcore-worker (code=exited, status=1/FAILURE)
Main PID: 5519 (code=exited, status=1/FAILURE)
CPU: 36ms

● pulpcore-worker@2.service - Pulp Worker
Loaded: loaded (/usr/lib/systemd/system/pulpcore-worker@.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2024-11-12 12:42:46 PST; 1s ago
Process: 5520 ExecStart=/usr/local/lib/pulp/bin/pulpcore-worker (code=exited, status=1/FAILURE)
Main PID: 5520 (code=exited, status=1/FAILURE)
CPU: 38ms

Nov 12 12:42:46 prodjp-yumrepo-s1-z2-0 systemd[1]: pulpcore-worker@2.service: Failed with result ‘exit-code’.

● nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: disabled)
Active: active (running) since Tue 2024-11-12 12:42:23 PST; 24s ago
Process: 1089 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 1117 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
Process: 1208 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
Main PID: 1248 (nginx)
Tasks: 2 (limit: 383816)
Memory: 4.6M
CPU: 53ms
CGroup: /system.slice/nginx.service
├─1248 “nginx: master process /usr/sbin/nginx”
└─1250 “nginx: worker process”
lines 22-52/52 (END)

#8

OK, since I have no picture about what you are trying to accomplish let me restate my first question:
How are you running Pulp? How is it installed in the first place? Where did you get “status” working and then not working?

Couple of thoughts that may or may not apply depending on how you deploy:

  • We know installing Pulp “by hand” is a complex thing to do so we recommend using the prebuilt containers. Welcome to Pulp OCI-images - Pulp Project
  • If you are unhappy with the selected set of plugins therein, you can still adjust this Containerfile https://github.com/pulp/pulp-oci-images/blob/latest/images/pulp/stable/Containerfile and build your own image.
  • Depending on the size/scope of your installation it may be useful to consider the podman-compose or the operator installation method.
  • If for some reason you need / really want to do the installation by hand, there is mainly one thing to warn you about:
    • pip is not good with maintaining installed packages. It’s dependency solution only cares about packages specified on the last call to pip install. In doubt start fresh and install all plugins in one go. If it still want’s to use that very old version of pulp_container, help it by saying pulp_container>=2.21.
2 Likes
#9

I checked:
pulp_container 2.14.16 depends on pulpcore>=3.21.6,<3.25
The fact that you have it installed alongside pulpcore 3.66 is definitely a pip screwup.
It’s a known bug, and there is nothing we can improve on our side: `pip install` breaks existing install (and prints that it did so) instead of refusing to perform · Issue #9482 · pypa/pip · GitHub

#10

we had cleaned up everything and re-installed pulp3 on the RHEL host. after the installation ‘pulp status’ was good but we couldn’t access the pulp via URL so we restarted all the pulp services, after restarting the services below 3 services are now failing and ‘pulp status’ giving below 502 error.

Could someone help us in fixing this issue. please let me know if you need any more details.

● pulpcore-api.service - Pulp WSGI Server
Loaded: loaded (/usr/lib/systemd/system/pulpcore-api.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2024-11-18 14:35:30 PST; 2s ago
Process: 28683 ExecStart=/usr/local/lib/pulp/bin/gunicorn pulpcore.app.wsgi:application --name pulp-api --bind 127.0.0.1:24817 --workers 1 --timeout 90 (code=exited, status=>
Main PID: 28683 (code=exited, status=3)
Status: “Gunicorn arbiter booted”
CPU: 260ms

● pulpcore-worker@1.service - Pulp Worker
Loaded: loaded (/usr/lib/systemd/system/pulpcore-worker@.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2024-11-18 14:35:30 PST; 2s ago
Process: 28684 ExecStart=/usr/local/lib/pulp/bin/pulpcore-worker (code=exited, status=1/FAILURE)
Main PID: 28684 (code=exited, status=1/FAILURE)
CPU: 216ms

● pulpcore-worker@2.service - Pulp Worker
Loaded: loaded (/usr/lib/systemd/system/pulpcore-worker@.service; enabled; preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2024-11-18 14:35:30 PST; 2s ago
Process: 28685 ExecStart=/usr/local/lib/pulp/bin/pulpcore-worker (code=exited, status=1/FAILURE)
Main PID: 28685 (code=exited, status=1/FAILURE)
CPU: 215ms

pulp status :

(pulp) [root@produs3-c1-yumrepo-s1-z1-0 pulp_installer]# pulp stattus
Usage: pulp [OPTIONS] COMMAND [ARGS]…
Try ‘pulp --help’ for help.

Error: No such command ‘stattus’.
(pulp) [root@produs3-c1-yumrepo-s1-z1-0 pulp_installer]# pulp status
status_read : get https://produs3-c1-yumrepo-s1-z1-0.us1.gcpint.ariba.com/pulp/api/v3/status/
Response: 502
Error:

502 Bad Gateway

502 Bad Gateway

nginx

(pulp) [root@produs3-c1-yumrepo-s1-z1-0 a_reghatte_sap_com]# pip list |grep -i pulp
pulp-cli 0.29.2
pulp-container 2.14.16
pulp-file 1.12.0
pulp-glue 0.29.2
pulp-rpm 3.19.12
pulpcore 3.22.33

#11

I need like probably 20 times more details about this specific part, in order to understand what’s going on.