Error to list remotes after reinstalling 3.21.2

Problem:
Got an error “Server Error (500)” when try to list remotes with
http ‘http://localhost/pulp/api/v3/remotes/rpm/rpm/
This happened when we reinstall pulp with pulp_installer. We use external S3 and postgres db

Expected outcome:
Expect to see a list of all remote since both external s3 and postgres remain the same

Pulpcore version:
3.21.2

Pulp plugins installed and their versions:
pulp-rpm 3.18.8

Operating system - distribution and version:
rhel8.6

Other relevant data:
We don’t see any additional errors in /var/log/messages besides “Server Error (500)” when we run http ‘http://localhost/pulp/api/v3/remotes/rpm/rpm/’ We rebuilt new hosts with the same 3.21.2 version and configuration while keeping the external S3 and postgres data. We are able to list repositories, distributions, publications but not remotes after the rebuild.

$ http localhost/pulp/api/v3/status/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 4568
Content-Type: application/json
Correlation-ID: 1c2f5802908140ec81aeb5a1f92a549b
Date: Wed, 09 Nov 2022 17:38:38 GMT
Referrer-Policy: same-origin
Server: nginx/1.14.1
Vary: Accept
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
“database_connection”: {
“connected”: true
},
“online_content_apps”: [
{
“last_heartbeat”: “2022-11-09T17:38:32.558954Z”,
“name”: “60390@pulp-d15”
},
{
“last_heartbeat”: “2022-11-09T17:38:33.867532Z”,
“name”: “63375@pulp-d10”
},
{
“last_heartbeat”: “2022-11-09T17:38:33.872858Z”,
“name”: “63374@pulp-d10”
},
{
“last_heartbeat”: “2022-11-09T17:38:28.343465Z”,
“name”: “60392@pulp-d15”
},
{
“last_heartbeat”: “2022-11-09T17:38:28.344217Z”,
“name”: “60396@pulp-d15”
},
{
“last_heartbeat”: “2022-11-09T17:38:28.344506Z”,
“name”: “60412@pulp-d15”
},
{
“last_heartbeat”: “2022-11-09T17:38:28.030859Z”,
“name”: “63354@pulp-d10”
},
{
“last_heartbeat”: “2022-11-09T17:38:28.954523Z”,
“name”: “63373@pulp-d10”
},
{
“last_heartbeat”: “2022-11-09T17:38:28.963106Z”,
“name”: “63353@pulp-d10”
},
{
“last_heartbeat”: “2022-11-09T17:38:31.180673Z”,
“name”: “60391@pulp-d15”
}
],
“online_workers”: [
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:34.170723Z”,
“name”: “59454@pulp-d10”,
“pulp_created”: “2022-11-09T01:58:02.590901Z”,
“pulp_href”: “/pulp/api/v3/workers/3a0e1644-7d92-44eb-9fbf-10de7e390335/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:35.379022Z”,
“name”: “56187@pulp-d15”,
“pulp_created”: “2022-11-09T01:57:56.879460Z”,
“pulp_href”: “/pulp/api/v3/workers/e2663a0b-dd28-4b03-886c-90f4a341fb30/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:35.638358Z”,
“name”: “58696@pulp-d10”,
“pulp_created”: “2022-11-09T01:57:54.907265Z”,
“pulp_href”: “/pulp/api/v3/workers/05aac73b-4641-415a-8bfd-00734766d5fb/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:36.939703Z”,
“name”: “55726@pulp-d15”,
“pulp_created”: “2022-11-09T01:57:52.885405Z”,
“pulp_href”: “/pulp/api/v3/workers/9f7f9617-fc11-4bcf-8e90-7b20958c4179/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:37.076805Z”,
“name”: “56034@pulp-d15”,
“pulp_created”: “2022-11-09T01:57:55.484560Z”,
“pulp_href”: “/pulp/api/v3/workers/5d8a7d94-4530-4fac-89ba-d16eb3d351d1/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:36.897097Z”,
“name”: “58998@pulp-d10”,
“pulp_created”: “2022-11-09T01:57:58.106349Z”,
“pulp_href”: “/pulp/api/v3/workers/0bbe6553-34a6-4a30-b6b9-a31a86cadc94/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:36.922760Z”,
“name”: “59150@pulp-d10”,
“pulp_created”: “2022-11-09T01:57:59.446175Z”,
“pulp_href”: “/pulp/api/v3/workers/03380468-e924-44c4-bf72-723b3a6dccbf/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:37.150664Z”,
“name”: “59302@pulp-d10”,
“pulp_created”: “2022-11-09T01:58:00.889938Z”,
“pulp_href”: “/pulp/api/v3/workers/748a8095-44d5-4546-84a8-4daeaf48d29b/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:30.093974Z”,
“name”: “58364@pulp-d10”,
“pulp_created”: “2022-11-09T01:57:51.725856Z”,
“pulp_href”: “/pulp/api/v3/workers/8f31b9ed-5490-4374-b155-086d1749fa81/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:30.358928Z”,
“name”: “58526@pulp-d10”,
“pulp_created”: “2022-11-09T01:57:52.840795Z”,
“pulp_href”: “/pulp/api/v3/workers/a12669bd-5583-44ea-b36e-158209617733/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:33.553203Z”,
“name”: “56492@pulp-d15”,
“pulp_created”: “2022-11-09T01:57:59.613343Z”,
“pulp_href”: “/pulp/api/v3/workers/d89a68b0-5f62-4bd5-9214-1b55effcf10b/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:33.801005Z”,
“name”: “55880@pulp-d15”,
“pulp_created”: “2022-11-09T01:57:54.159406Z”,
“pulp_href”: “/pulp/api/v3/workers/79fd28d6-f6a3-41f7-93ea-1ec13cb99428/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:33.828904Z”,
“name”: “56339@pulp-d15”,
“pulp_created”: “2022-11-09T01:57:58.107422Z”,
“pulp_href”: “/pulp/api/v3/workers/a0107842-b3a1-4142-8f95-2ea5e834064a/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:33.060610Z”,
“name”: “58848@pulp-d10”,
“pulp_created”: “2022-11-09T01:57:56.455609Z”,
“pulp_href”: “/pulp/api/v3/workers/c171362a-0bbc-4786-959c-7e54adee9a12/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:34.327789Z”,
“name”: “55414@pulp-d15”,
“pulp_created”: “2022-11-09T01:57:51.488844Z”,
“pulp_href”: “/pulp/api/v3/workers/d4d0a84c-5490-45ee-9bb1-b198acdb6725/”
},
{
“current_task”: null,
“last_heartbeat”: “2022-11-09T17:38:34.508042Z”,
“name”: “55570@pulp-d15”,
“pulp_created”: “2022-11-09T01:57:51.908645Z”,
“pulp_href”: “/pulp/api/v3/workers/888a5b1d-092d-4c08-a76e-73ef076bd0c4/”
}
],
“redis_connection”: {
“connected”: true
},
“storage”: null,
“versions”: [
{
“component”: “core”,
“package”: “pulpcore”,
“version”: “3.21.2”
},
{
“component”: “rpm”,
“package”: “pulp-rpm”,
“version”: “3.18.8”
},
{
“component”: “file”,
“package”: “pulp-file”,
“version”: “1.11.2”
},
{
“component”: “deb”,
“package”: “pulp_deb”,
“version”: “2.20.0”
},
{
“component”: “ansible”,
“package”: “pulp-ansible”,
“version”: “0.15.0”
}
]
}

Can you post the 500 error traceback if you have it?

I don’t see any traceback in this case. Not in the /var/log/messages. Do you need any settings to turn the traceback on?