Is pulp in one container suitable for production

[xm1234567]

Hello everyone

We have one pulp2 server: it is a physical machine, not a VM, there are +100 rpm repositories on it for our internal use, total repo size ~1TB, and DB is provided by our external DB server.

I am wondering , for pulp3, is pulp-in-one-container enough for our case? or better use single process images(docker-composer) ?
Any experience, suggestions?

Another question: to use container, is it possible to use a external DB?

Thanks in advance

xm

Other relevant data:

[x9c4]

Yes, using an external db is absolutely possible, as is using cloud storage technology (e.g. run your own minio container…). It’s just a matter of choosing the proper settings. When doing so you might think about using the single process images in order to not carry the installed postgres around for no good reason.
I will also lean out of the window and claim that while keeping the db, the storage and the db encryption key, you can switch forth and back between both container models.

[x9c4]

Has anyone ever tried mixing the two models? Like adding an external pulpcore-worker fleet to the single container while using external postgres, redis and cloud storage?

[tytan]

Our prd Pulp3 is running in container, no external DB connection.

[xm1234567]

Hello @tytan
So you have pulp-in-one-container in production? I am very interested to your experience if you don’t mind to share ?
Is it for rpm repository? If yes, have you got some rpm repository sync problem?
Thanks
xm

[mikch06]

Hi, we also have a pulp container in production. for ostree and rpm so far. About 90 repositories. It goes welle. I think the repo mgmt is still bad with the cli and no better or more comforttable way. But the container is stable.
And the setup was quite fast and worked very well.

We have no sync problems so far.

[xm1234567]

Hi @mikch06
Thanks for your reply.

Do you use podman or docker? The host machine has what OS?
Do you need to set any special permissions?
How do you debug? I mean, in this pulp-in-on-container, there is no systemctl status pulp-worker1, these kind of thing available?

I follow this doc simply: Multi-Process Images - Pulp OCI Images
I don’t have much experience of container (docker or podman)
My host machine is Rocky9. I use docker, the container is created, I can create repo, upload a rpm by hand. But to do rpm syno, I just see the task stays in state running for ever. I have to cancel this task after one day in running state. I have no idea, why, and how to debug?
Appreciate any help for this problem

[mikch06]

Hi @xm1234567
Yes, I use podman ond a Rocky9. We started with Fedora Coreos, but there are to much side effects for us. So we use our main OS R9.
And we go by non-root user - and facing NO problems.

so far i never used debugging functions for pulp - but you can use every command inside the container.
Just access your container podman exec -it pulp bash and you are inside of it.

Nowadays i would prefer podman… but yes, you need a bit of know how.

For the pulp functions you still need the pulp-cli or go by web-api. We do it by pulp-cli - and everything works fine. Biggest problem for us was to migrate the database.
→ hint: drop the primary installed pulp databse before you import yours.
→ copy all your artifacts to the new system.

[xm1234567]

Hello @mikch06 , thank you very much for sharing your experience.
For me,
First I tried Podman+as root + on rocky9, it was because I was lazy to add subuid/gid, and net.ipv4.ip_unprivileged_port_start.conf, => but rpm repo sync stucked in running state forever

2nd I tried Podman+rootless+Rocky9+ subgid/uid+ ip_unprivileged_port_start.conf => rpm repo sync OK

3rd: I tried docker, because we use Puppet and mainly docker here, so I would like to take advantage of the existing things, while docker + root or not root + rocky8 => rpm repo sync stucked

I have tried pulp debug mode and also reading container logs , but not much more information to tell why the task is stay running and never finished.
Here are some details on the docker container and pulp debug logs, if you are interested to have a look:

Summary

[xxx@xxxxx ~]$ docker logs -f pulp
pulp [18ce01c225a04a9db773fdbe1fc372b8]: pulpcore.tasking.tasks:INFO: Starting task 018c2560-d94d-7d17-a7fe-2bd9dce6c6a4
pulp [18ce01c225a04a9db773fdbe1fc372b8]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=rocky9_extras remote=rocky9_extras_remote
(‘pulp [18ce01c225a04a9db773fdbe1fc372b8]: ::ffff:127.0.0.1 - admin [01/Dec/2023:12:35:53 +0000] “GET /pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/ HTTP/1.0” 200 767 “-” “Pulp-CLI/0.21.2”’,)
(‘pulp [18ce01c225a04a9db773fdbe1fc372b8]: ::ffff:127.0.0.1 - admin [01/Dec/2023:12:35:55 +0000] “GET /pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/ HTTP/1.0” 200 767 “-” “Pulp-CLI/0.21.2”’,)
(‘pulp [18ce01c225a04a9db773fdbe1fc372b8]: ::ffff:127.0.0.1 - admin [01/Dec/2023:12:35:56 +0000] “GET /pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/ HTTP/1.0” 200 767 “-” “Pulp-CLI/0.21.2”’,)
(‘pulp [18ce01c225a04a9db773fdbe1fc372b8]: ::ffff:127.0.0.1 - admin [01/Dec/2023:12:35:58 +0000] “GET /pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/ HTTP/1.0” 200 767 “-” “Pulp-CLI/0.21.2”’,)
(‘pulp [18ce01c225a04a9db773fdbe1fc372b8]: ::ffff:127.0.0.1 - admin [01/Dec/2023:12:35:59 +0000] “GET /pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/ HTTP/1.0” 200 767 “-” “Pulp-CLI/0.21.2”’,)
(‘pulp [18ce01c225a04a9db773fdbe1fc372b8]: ::ffff:127.0.0.1 - admin [01/Dec/2023:12:36:01 +0000] “GET /pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/ HTTP/1.0” 200 767 “-” “Pulp-CLI/0.21.2”’,)
(‘pulp [18ce01c225a04a9db773fdbe1fc372b8]: ::ffff:127.0.0.1 - admin [01/Dec/2023:12:36:03 +0000] “GET /pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/ HTTP/1.0” 200 767 “-” “Pulp-CLI/0.21.2”’,)

And the pulp in debug mode:

(pulp_venv) [xxx@xxxxx ~]$ pulp -vvv --profile admin rpm repository sync --name rocky9_extras
repositories_rpm_rpm_list : get http://ccosfip00322.in2p3.fr:8080/pulp/api/v3/repositories/rpm/rpm/?name=rocky9_extras&offset=0&limit=1
User-Agent: Pulp-CLI/0.21.2
Accept-Encoding: gzip, deflate
Accept: application/json
Connection: keep-alive
Authorization: Basic YWRtaW46dGVzdA==
Response: 200
Server: nginx/1.22.1
Date: Fri, 01 Dec 2023 12:35:52 GMT
Content-Type: application/json
Content-Length: 785
Connection: keep-alive
Vary: Accept
Allow: GET, POST, HEAD, OPTIONS
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Cross-Origin-Opener-Policy: same-origin
Correlation-ID: 18ce01c225a04a9db773fdbe1fc372b8
Access-Control-Expose-Headers: Correlation-ID
{“count”:1,“next”:null,“previous”:null,“results”:[{“pulp_href”:"/pulp/api/v3/repositories/rpm/rpm/018c1ba1-6c6e-753e-8658-3c034b12dae5/",“pulp_created”:“2023-11-29T15:10:13.103998Z”,“versions_href”:"/pulp/api/v3/repositories/rpm/rpm/018c1ba1-6c6e-753e-8658-3c034b12dae5/versions/",“pulp_labels”:{},“latest_version_href”:"/pulp/api/v3/repositories/rpm/rpm/018c1ba1-6c6e-753e-8658-3c034b12dae5/versions/0/",“name”:“rocky9_extras”,“description”:“Repo rocky9_extras”,“retain_repo_versions”:1,“remote”:"/pulp/api/v3/remotes/rpm/rpm/018c1ba1-656f-7dc8-b833-a30113f1ebef/",“autopublish”:false,“metadata_signing_service”:null,“retain_package_versions”:0,“metadata_checksum_type”:null,“package_checksum_type”:null,“gpgcheck”:null,“repo_gpgcheck”:null,“sqlite_metadata”:false,“repo_config”:{}}]}
repositories_rpm_rpm_sync : post http://ccosfip00322.in2p3.fr:8080/pulp/api/v3/repositories/rpm/rpm/018c1ba1-6c6e-753e-8658-3c034b12dae5/sync/
User-Agent: Pulp-CLI/0.21.2
Accept-Encoding: gzip, deflate
Accept: application/json
Connection: keep-alive
Correlation-ID: 18ce01c225a04a9db773fdbe1fc372b8
Content-Length: 2
Content-Type: application/json
Authorization: Basic YWRtaW46dGVzdA==
b’{}’
Response: 202
Server: nginx/1.22.1
Date: Fri, 01 Dec 2023 12:35:53 GMT
Content-Type: application/json
Content-Length: 67
Connection: keep-alive
Vary: Accept
Allow: POST, OPTIONS
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Cross-Origin-Opener-Policy: same-origin
Correlation-ID: 18ce01c225a04a9db773fdbe1fc372b8
Access-Control-Expose-Headers: Correlation-ID
{“task”:"/pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/"}
tasks_read : get http://ccosfip00322.in2p3.fr:8080/pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/
User-Agent: Pulp-CLI/0.21.2
Accept-Encoding: gzip, deflate
Accept: application/json
Connection: keep-alive
Correlation-ID: 18ce01c225a04a9db773fdbe1fc372b8
Authorization: Basic YWRtaW46dGVzdA==
Response: 200
Server: nginx/1.22.1
Date: Fri, 01 Dec 2023 12:35:53 GMT
Content-Type: application/json
Content-Length: 767
Connection: keep-alive
Vary: Accept
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Cross-Origin-Opener-Policy: same-origin
Correlation-ID: 18ce01c225a04a9db773fdbe1fc372b8
Access-Control-Expose-Headers: Correlation-ID
{“pulp_href”:"/pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/",“pulp_created”:“2023-12-01T12:35:53.294229Z”,“state”:“running”,“name”:“pulp_rpm.app.tasks.synchronizing.synchronize”,“logging_cid”:“18ce01c225a04a9db773fdbe1fc372b8”,“created_by”:"/pulp/api/v3/users/1/",“started_at”:“2023-12-01T12:35:53.392229Z”,“finished_at”:null,“error”:null,“worker”:"/pulp/api/v3/workers/018c1bbe-c4fe-780f-88df-8b08ee3a2ee4/",“parent_task”:null,“child_tasks”:[],“task_group”:null,“progress_reports”:[],“created_resources”:[],“reserved_resources_record”:["/pulp/api/v3/repositories/rpm/rpm/018c1ba1-6c6e-753e-8658-3c034b12dae5/",“shared:/pulp/api/v3/remotes/rpm/rpm/018c1ba1-656f-7dc8-b833-a30113f1ebef/”,“shared:/pulp/api/v3/domains/018c1b92-8e75-7aae-8e7e-ddcb1ab0ad7c/”]}
Started background task /pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/

(pulp_venv) [xxx@xxxxx ~]$ pulp task list --state running
[
{
“pulp_href”: “/pulp/api/v3/tasks/018c2560-d94d-7d17-a7fe-2bd9dce6c6a4/”,
“pulp_created”: “2023-12-01T12:35:53.294229Z”,
“state”: “running”,
“name”: “pulp_rpm.app.tasks.synchronizing.synchronize”,
“logging_cid”: “18ce01c225a04a9db773fdbe1fc372b8”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: “2023-12-01T12:35:53.392229Z”,
“finished_at”: null,
“error”: null,
“worker”: “/pulp/api/v3/workers/018c1bbe-c4fe-780f-88df-8b08ee3a2ee4/”,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/rpm/rpm/018c1ba1-6c6e-753e-8658-3c034b12dae5/”,
“shared:/pulp/api/v3/remotes/rpm/rpm/018c1ba1-656f-7dc8-b833-a30113f1ebef/”,
“shared:/pulp/api/v3/domains/018c1b92-8e75-7aae-8e7e-ddcb1ab0ad7c/”
]
}
]

I feel strange, I was hopping, using container should be more consistent, and should be working everywhere?

[xm1234567]

a little update for my docker + root or not root + rocky8 test, the problem I meet, it is because the special MTU settings of my openstack vm . As soon as I change its value, rpm repo sync working perfectly !!!

The details are here:

Rpm repository sync task stucked in pulp-in-one-container

Hello @gerrod , thank you very much for the test, your words inspire me something! Yes, the VM I used is from our openstack infrastructure, they have some predefined custom settings from our site, and selinux and firewalld are not enabled. And YES there is something with our system for these kind of vm, I just discovered, it is related with the MTU for docker daemon. I have to add : cat /etc/docker/daemon.json { "mtu": 1442 } Now, rpm repo sync is working perfectly under docker+rootless+r…

[mikch06]

What we struggled with was disabling of ipv6 in this network the false ways. The container inside uses it, i believe.
Then mind: selinux, and firewall both enabled by default on rocky.

Otherwise we had no issues while sync or other tasks.
But ours are vm’s on vmware.