Api timeout

Support
#1

Problem:*
Api timeout

Expected outcome:

Pulpcore version:
3.22.2
Pulp plugins installed and their versions:
pulp-ansible 0.16.1
pulp-cli 0.18.0
pulp-deb 2.20.2
pulp-file 1.12.0
pulp-glue 0.18.0
pulp-rpm 3.19.3
pulpcore 3.22.4

Operating system - distribution and version:
Rhel 8.6
Other relevant data:

We frequently see the api server timed out when all of our 5 pulp-workers are busy creating publications. We have 5 hosts and each runs api and pulpcore-workers. The hosts have enought cpu and memorys. Could the bottleneck be the external postgresql or five workers put too much load to the postgres? I am not sure if we should add more cpu and memory to the postgres or reduce number of pulpcore-works?

We increased the timeout for nginx already.
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_send_timeout 300;

#2

There’s a lot that could be going on here. What are the workers doing? How many connections is postgres allowing? What does the load on the postgres machine/container look like? Are all API requests timing out? If not - which specific ones are a problem?

Generally, “allow less concurrency” (ie, fewer pulpcore-workers/api-handlers) can help you get on top of the immediate pain for your users, at the expense of “things take longer” and “some API requests will be refused outright”. Without knowing more (a LOT more) specifics of exactly what’s going on when you see these timeouts, it’s hard to give any specific advice.