Error while running pulp status

midhuhk · March 21, 2024, 10:02am

Hi All ,

I have installed pulp operator in openshift and pulp api route is accessible in browser . I have installed pulp cli using pip and up on chekcing pulp status , it throws me below error.

[myserver:~]# pulp status
Traceback (most recent call last):
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 137, in load_api
with open(apidoc_cache, “rb”) as f:
FileNotFoundError: [Errno 2] No such file or directory: ‘/root/.cache/squeezer/https______pulp_api_v3_docs_api.jsonapi.json’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/bin/pulp”, line 11, in
sys.exit(main())
File “/usr/local/lib/python3.6/site-packages/pulp_cli/init.py”, line 38, in main
return _main()
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1128, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1053, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python3.6/site-packages/pulpcore/cli/common/generic.py”, line 211, in invoke
return super().invoke(ctx)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python3.6/site-packages/pulpcore/cli/common/generic.py”, line 211, in invoke
return super().invoke(ctx)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 754, in invoke
return __callback(*args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/click/decorators.py”, line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 754, in invoke
return __callback(*args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/pulpcore/cli/core/status.py”, line 25, in status
result = pulp_ctx.call(“status_read”)
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/context.py”, line 357, in call
if self.domain_enabled:
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/context.py”, line 293, in domain_enabled
return cast(bool, self.api.api_spec.get(“info”, {}).get(“x-pulp-domain-enabled”, False))
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/context.py”, line 314, in api
doc_path=f"{self._api_root}api/v3/docs/api.json", **self._api_kwargs
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 124, in init
self.load_api(refresh_cache=refresh_cache)
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 142, in load_api
data = self._download_api()
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 167, in _download_api
response.raise_for_status()
File “/usr/local/lib/python3.6/site-packages/requests/models.py”, line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https:///pulp/api/v3/docs/api.json

My cli.toml is as below:
[cli]
base_url = “”
api_root = “/pulp/”
domain = “default”
username = “admin”
password = “password”
cert = “”
key = “”
verify_ssl = false
format = “json”
dry_run = false
timeout = 0
verbose = 0

quba42 · March 21, 2024, 10:08am

You need to set the base URL where your pulp instance is reachable in your cli.toml.

Right now the CLI is trying to reach the Pulp API at https:///pulp/api/v3, because your base URL is an empty string.

midhuhk · March 21, 2024, 11:18am

My bad …It was a typo while pasting here …I have provided the route which is reachabel in base url.

Errror
Traceback (most recent call last):
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 137, in load_api
with open(apidoc_cache, “rb”) as f:
FileNotFoundError: [Errno 2] No such file or directory: ‘/root/.cache/squeezer/https___pulp-server.apps.az2-ost00.danskenet.net__pulp_api_v3_docs_api.jsonapi.json’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/bin/pulp”, line 11, in
sys.exit(main())
File “/usr/local/lib/python3.6/site-packages/pulp_cli/init.py”, line 38, in main
return _main()
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1128, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1053, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python3.6/site-packages/pulpcore/cli/common/generic.py”, line 211, in invoke
return super().invoke(ctx)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python3.6/site-packages/pulpcore/cli/common/generic.py”, line 211, in invoke
return super().invoke(ctx)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 754, in invoke
return __callback(*args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/click/decorators.py”, line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 754, in invoke
return __callback(*args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/pulpcore/cli/core/status.py”, line 25, in status
result = pulp_ctx.call(“status_read”)
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/context.py”, line 357, in call
if self.domain_enabled:
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/context.py”, line 293, in domain_enabled
return cast(bool, self.api.api_spec.get(“info”, {}).get(“x-pulp-domain-enabled”, False))
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/context.py”, line 314, in api
doc_path=f"{self._api_root}api/v3/docs/api.json", **self._api_kwargs
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 124, in init
self.load_api(refresh_cache=refresh_cache)
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 142, in load_api
data = self._download_api()
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 167, in _download_api
response.raise_for_status()
File “/usr/local/lib/python3.6/site-packages/requests/models.py”, line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://pulp-server.apps.az2-ost00.danskenet.net/pulp/api/v3/docs/api.json

[root@yb1334.danskenet.net TEST:~]# cat /root/.config/pulp/cli.toml
[cli]
base_url = “https://pulp-server.apps.az2-ost00.danskenet.net”
api_root = “/pulp/”
domain = “default”
username = “admin”
password = “passwd”
cert = “”
key = “”
verify_ssl = false
format = “json”
dry_run = false
timeout = 0
verbose = 0

x9c4 · March 21, 2024, 11:24am

Now your server is actually responding, but it looks like we only get to the reverse proxy.

can you cURL this address? It looks valid to me.

midhuhk · March 21, 2024, 11:29am

Bad gateway

[root@yb1334.danskenet.net TEST:~]# curl https://pulp-server.apps.az2-ost00.danskenet.net/pulp/api/v3/docs/api.json

502 Bad Gateway

The server returned an invalid or incomplete response.

x9c4 · March 21, 2024, 1:25pm

So pinned down to being a server issue, not a client problem.

I’d suggest digging into the reverse proxy logs, and checking whether the pulpcore-api services are running.
Or just cut though the truth right away: It’s always DNS. (just kidding)

midhuhk · March 22, 2024, 7:19am

I can see pods are running fine in ocp. Also route is accessible

x9c4 · March 22, 2024, 8:23am

The claim to be using “http” instead of “https”. I’d still guess it’s something in the reverse proxy config.

midhuhk · March 28, 2024, 5:28am

Any suggestion on how to proceed ?

I can curl the v3 endpoint from server , but cannot check the status via cli from same server

[root@yb1334.danskenet.net TEST:~]# curl https://pulp-server.apps.az2-ost00.danskenet.net/pulp/api/v3/status/
{“versions”:[{“component”:“core”,“version”:“3.21.2”,“package”:“pulpcore”},{“component”:“rpm”,“version”:“3.18.8”,“package”:“pulp-rpm”},{“component”:“python”,“version”:“3.7.3”,“package”:“pulp-python”},{“component”:“deb”,“version”:“2.20.0”,“package”:“pulp_deb”},{“component”:“container”,“version”:“2.14.2”,“package”:“pulp-container”},{“component”:“file”,“version”:“1.11.2”,“package”:“pulp-file”},{“component”:“certguard”,“version”:“1.5.5”,“package”:“pulp-certguard”},{“component”:“ansible”,“version”:“0.15.0”,“package”:“pulp-ansible”},{“component”:“maven”,“version”:“0.3.3”,“package”:“pulp-maven”}],“online_workers”:[{“pulp_href”:"/pulp/api/v3/workers/024ca9ef-3d94-43d9-a2e1-77f275fd8b15/",“pulp_created”:“2024-03-27T09:51:12.562379Z”,“name”:“221@pulp-test-worker-7d5d485f4b-rhx2v”,“last_heartbeat”:“2024-03-27T14:05:22.011557Z”,“current_task”:null},{“pulp_href”:"/pulp/api/v3/workers/270c1a1f-84dc-41e8-b6c3-ce5b4d5cc921/",“pulp_created”:“2024-03-27T09:51:06.415971Z”,“name”:“895@pulp-test-worker-7d5d485f4b-vlrnk”,“last_heartbeat”:“2024-03-27T14:05:25.631433Z”,“current_task”:null}],“online_content_apps”:[{“name”:“363@pulp-test-content-5f68c987fc-shx2r”,“last_heartbeat”:“2024-03-27T14:05:19.955941Z”},{“name”:“364@pulp-test-content-5f68c987fc-shx2r”,“last_heartbeat”:“2024-03-27T14:05:21.595041Z”}],“database_connection”:{“connected”:true},“redis_connection”:{“connected”:true},“storage”:{“total”:10737418240,“used”:0,“free”:10737418240}}

[root@yb1334.danskenet.net TEST:~]# pulp status
Traceback (most recent call last):
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 137, in load_api
with open(apidoc_cache, “rb”) as f:
FileNotFoundError: [Errno 2] No such file or directory: ‘/root/.cache/squeezer/https___pulp-server.apps.az2-ost00.danskenet.net___pulp_api_v3_docs_api.jsonapi.json’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/bin/pulp”, line 11, in
sys.exit(main())
File “/usr/local/lib/python3.6/site-packages/pulp_cli/init.py”, line 38, in main
return _main()
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1128, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1053, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python3.6/site-packages/pulpcore/cli/common/generic.py”, line 211, in invoke
return super().invoke(ctx)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python3.6/site-packages/pulpcore/cli/common/generic.py”, line 211, in invoke
return super().invoke(ctx)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 754, in invoke
return __callback(*args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/click/decorators.py”, line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/click/core.py”, line 754, in invoke
return __callback(*args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/pulpcore/cli/core/status.py”, line 25, in status
result = pulp_ctx.call(“status_read”)
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/context.py”, line 357, in call
if self.domain_enabled:
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/context.py”, line 293, in domain_enabled
return cast(bool, self.api.api_spec.get(“info”, {}).get(“x-pulp-domain-enabled”, False))
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/context.py”, line 314, in api
doc_path=f"{self._api_root}api/v3/docs/api.json", **self._api_kwargs
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 124, in init
self.load_api(refresh_cache=refresh_cache)
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 142, in load_api
data = self._download_api()
File “/usr/local/lib/python3.6/site-packages/pulp_glue/common/openapi.py”, line 167, in _download_api
response.raise_for_status()
File “/usr/local/lib/python3.6/site-packages/requests/models.py”, line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://pulp-server.apps.az2-ost00.danskenet.net/pulp/api/v3/docs/api.json

midhuhk · March 29, 2024, 8:30am

Managed to fix this by increasing pod memory . Thank you

x9c4 · April 2, 2024, 7:25am

That was unexpected. Can you maybe leave some numbers for people later?
Did the services OOM repeatedly?

midhuhk · April 3, 2024, 8:24am

Yes , there were oom errors .

[2024-03-28 13:56:00 +0000] [2836] [INFO] Booting worker with pid: 2836
[2024-03-28 13:56:28 +0000] [1] [WARNING] Worker with pid 2815 was terminated due to signal 9

I had to unmanage the operator and change the memory to higher value (random value if 1G was given) and then pods were fine .

Probably we can take it as a bug and increase the memory of pod so that operator can mange the resoruce by itlsef ?