Rpm repository sync task stucked in pulp-in-one-container

Support
#1

Problem:
Following Multi-Process Images - Pulp OCI Images
I create a container by docker as a non-privilege user on a rocky8 VM, set the admin password, check pulp status ok.

Pulp images is: quay.io/pulp/pulp latest 26d96dd810e5 13 hours ago 2.07GB

Then following Create, Sync and Publish a Repository — Pulp RPM Support 3.24.0 documentation
I create a rpm repo and test sync of this small repo, but the sync task stucked.
pulp task list show the sync task running, but in normal situation, the task should be finished in 1-2 mins, it is a very small repo with not many rpm pakages.

Here are some information of this small repo if needed:

Summary

(pulp_venv) [xxx@xxxxxx ~]$ pulp rpm repository show --name rocky9_extras
{
“pulp_href”: “/pulp/api/v3/repositories/rpm/rpm/018c1628-4370-7eaa-8aa3-ef75dafd5294/”,
“pulp_created”: “2023-11-28T13:39:46.674019Z”,
“versions_href”: “/pulp/api/v3/repositories/rpm/rpm/018c1628-4370-7eaa-8aa3-ef75dafd5294/versions/”,
“pulp_labels”: {},
“latest_version_href”: “/pulp/api/v3/repositories/rpm/rpm/018c1628-4370-7eaa-8aa3-ef75dafd5294/versions/0/”,
“name”: “rocky9_extras”,
“description”: “Repo rocky9_extras”,
“retain_repo_versions”: 1,
“remote”: “/pulp/api/v3/remotes/rpm/rpm/018c1628-3d78-7c00-a011-9e58047605d2/”,
“autopublish”: false,

(pulp_venv) [xxx@xxxxxx ~]$ pulp rpm remote show --name rocky9_extras_remote
{
“pulp_href”: “/pulp/api/v3/remotes/rpm/rpm/018c1628-3d78-7c00-a011-9e58047605d2/”,
“pulp_created”: “2023-11-28T13:39:45.145863Z”,
“name”: “rocky9_extras_remote”,
“url”: “Index of /rocky/9/extras/x86_64/os”,
“ca_cert”: null,
“client_cert”: null,
“tls_validation”: true,
“proxy_url”: null,

Expected outcome:
The sync task should finish quickly, not in the state running forever.

Pulpcore version:
“version”: “3.41.1”,
“package”: “pulpcore”,

Pulp plugins installed and their versions:
“version”: “3.24.0”,
“package”: “pulp-rpm”,

Details of pulp status if needed:

Summary

pulp status
{
“versions”: [
{
“component”: “core”,
“version”: “3.41.1”,
“package”: “pulpcore”,
“domain_compatible”: true
},
{
“component”: “ansible”,
“version”: “0.21.0”,
“package”: “pulp-ansible”,
“domain_compatible”: false
},
{
“component”: “certguard”,
“version”: “1.7.1”,
“package”: “pulp-certguard”,
“domain_compatible”: true
},
{
“component”: “container”,
“version”: “2.17.0”,
“package”: “pulp-container”,
“domain_compatible”: false
},
{
“component”: “deb”,
“version”: “3.1.0”,
“package”: “pulp_deb”,
“domain_compatible”: false
},
{
“component”: “gem”,
“version”: “0.4.0”,
“package”: “pulp-gem”,
“domain_compatible”: true
},
{
“component”: “maven”,
“version”: “0.7.0”,
“package”: “pulp-maven”,
“domain_compatible”: false
},
{
“component”: “ostree”,
“version”: “2.2.1”,
“package”: “pulp-ostree”,
“domain_compatible”: false
},
{
“component”: “python”,
“version”: “3.11.0”,
“package”: “pulp-python”,
“domain_compatible”: false
},
{
“component”: “rpm”,
“version”: “3.24.0”,
“package”: “pulp-rpm”,
“domain_compatible”: true
},
{
“component”: “file”,
“version”: “3.41.1”,
“package”: “pulp-file”,
“domain_compatible”: true
}
],
“online_workers”: [
{
“pulp_href”: “/pulp/api/v3/workers/018c1621-d8aa-7e3f-a336-8276cc3e7860/”,
“pulp_created”: “2023-11-28T13:32:46.142665Z”,
“name”: “385@bc541f4bd344”,
“last_heartbeat”: “2023-11-28T14:48:14.290311Z”,
“versions”: {
“deb”: “3.1.0”,
“gem”: “0.4.0”,
“rpm”: “3.24.0”,
“core”: “3.41.1”,
“file”: “3.41.1”,
“maven”: “0.7.0”,
“ostree”: “2.2.1”,
“python”: “3.11.0”,
“ansible”: “0.21.0”,
“certguard”: “1.7.1”,
“container”: “2.17.0”
},
“current_task”: null
},
{
“pulp_href”: “/pulp/api/v3/workers/018c1621-d917-7fa5-b276-94652db3d93f/”,
“pulp_created”: “2023-11-28T13:32:46.232920Z”,
“name”: “429@bc541f4bd344”,
“last_heartbeat”: “2023-11-28T14:48:14.366486Z”,
“versions”: {
“deb”: “3.1.0”,
“gem”: “0.4.0”,
“rpm”: “3.24.0”,
“core”: “3.41.1”,
“file”: “3.41.1”,
“maven”: “0.7.0”,
“ostree”: “2.2.1”,
“python”: “3.11.0”,
“ansible”: “0.21.0”,
“certguard”: “1.7.1”,
“container”: “2.17.0”
},
“current_task”: “/pulp/api/v3/tasks/018c164c-5197-7069-9587-981cf1f3f35c/”
}
],
“online_api_apps”: [
{
“name”: “438@bc541f4bd344”,
“last_heartbeat”: “2023-11-28T14:48:23.526952Z”,
“versions”: {
“deb”: “3.1.0”,
“gem”: “0.4.0”,
“rpm”: “3.24.0”,
“core”: “3.41.1”,
“file”: “3.41.1”,
“maven”: “0.7.0”,
“ostree”: “2.2.1”,
“python”: “3.11.0”,
“ansible”: “0.21.0”,
“certguard”: “1.7.1”,
“container”: “2.17.0”
}
},
{
“name”: “437@bc541f4bd344”,
“last_heartbeat”: “2023-11-28T14:48:23.540028Z”,
“versions”: {
“deb”: “3.1.0”,
“gem”: “0.4.0”,
“rpm”: “3.24.0”,
“core”: “3.41.1”,
“file”: “3.41.1”,
“maven”: “0.7.0”,
“ostree”: “2.2.1”,
“python”: “3.11.0”,
“ansible”: “0.21.0”,
“certguard”: “1.7.1”,
“container”: “2.17.0”
}
}
],
“online_content_apps”: [
{
“name”: “459@bc541f4bd344”,
“last_heartbeat”: “2023-11-28T14:48:19.588775Z”,
“versions”: {
“deb”: “3.1.0”,
“gem”: “0.4.0”,
“rpm”: “3.24.0”,
“core”: “3.41.1”,
“file”: “3.41.1”,
“maven”: “0.7.0”,
“ostree”: “2.2.1”,
“python”: “3.11.0”,
“ansible”: “0.21.0”,
“certguard”: “1.7.1”,
“container”: “2.17.0”
}
},
{
“name”: “460@bc541f4bd344”,
“last_heartbeat”: “2023-11-28T14:48:19.595592Z”,
“versions”: {
“deb”: “3.1.0”,
“gem”: “0.4.0”,
“rpm”: “3.24.0”,
“core”: “3.41.1”,
“file”: “3.41.1”,
“maven”: “0.7.0”,
“ostree”: “2.2.1”,
“python”: “3.11.0”,
“ansible”: “0.21.0”,
“certguard”: “1.7.1”,
“container”: “2.17.0”
}
}
],
“database_connection”: {
“connected”: true
},
“redis_connection”: {
“connected”: false
},
“storage”: {
“total”: 21464350720,
“used”: 262459392,
“free”: 21201891328
},
“content_settings”: {
“content_origin”: “http://xx.xxx.xx.xx:8080”,
“content_path_prefix”: “/pulp/content/”
},
“domain_enabled”: false
}

Operating system - distribution and version:
NAME=“Rocky Linux”
VERSION=“8.9 (Green Obsidian)”

containerd.io-1.6.25-3.1.el8.x86_64
docker-buildx-plugin-0.11.2-1.el8.x86_64
docker-ce-24.0.7-1.el8.x86_64
docker-ce-cli-24.0.7-1.el8.x86_64
docker-ce-rootless-extras-24.0.7-1.el8.x86_64
docker-compose-plugin-2.21.0-1.el8.x86_64

More other information:
I have tried:
pulp-all-in-one container by podman as non-privilege user=> rpm repo sync OK
pulp-all-in-one container by podman as root user => rpm repo sync NOK

Any idea why this problem? And how to fix it?

Thanks in advance !

Other relevant data:

#2

Great use of the hide details feature!

#3

@xm1234567 I’ve tried to reproduce the error you are seeing, but I haven’t been able to. Here are the steps I’ve taken:

  1. Set up a Rocky 9 VM (9.3 Blue-Onyx) [-1]
  2. Install Docker engine [0] and add my user to the docker group so I don’t need to use sudo [1]
  3. Follow the Multi-Process Images quickstart guide [2], making sure to use docker instead of podman (and also using the SElinux command variant).
  4. Set up the rpm repository and remote as you have listed and performed the sync. Completed in less than 5 seconds.
CLI Commands 
pulp rpm remote create --name rocky9_extras_remote --url http://miroir.univ-lorraine.fr/rocky/9/extras/x86_64/os/
pulp rpm repository create --name rocky9_extras --retain-repo-versions 1 --remote rocky9_extras_remote
pulp rpm repository sync --name rocky9_extras

I am not sure what is going on with your system. Can you share the output of pulp task list --state-in waiting --state-in running?

[-1] Rocky on VirtualBox - Documentation
[0] Install Docker Engine on CentOS | Docker Docs
[1] Linux post-installation steps for Docker Engine | Docker Docs
[2] Multi-Process Images - Pulp OCI Images

1 Like
#4

Hello @gerrod , thank you very much for the test, your words inspire me something!
Yes, the VM I used is from our openstack infrastructure, they have some predefined custom settings from our site, and selinux and firewalld are not enabled.

And YES there is something with our system for these kind of vm, I just discovered, it is related with the MTU for docker daemon. I have to add :

cat /etc/docker/daemon.json
{
  "mtu": 1442
}

Now, rpm repo sync is working perfectly under docker+rootless+rocky8 I am really really happy for that.

Many thanks sincerely :fireworks: :pray:

3 Likes
Is pulp in one container suitable for production
#5

Another update, to use docker-composer on openstack vm for my usecase, I have to add one networks part into compose.yaml:

volumes:
  pulp:
    name: pulp${DEV_VOLUME_SUFFIX:-dev}
...
networks:                                
  default:                               
    driver: bridge                       
    driver_opts:                         
      com.docker.network.driver.mtu: 1454

Seems it is well known for openstack vm +docker: Docker MTU issues and solutions - Matthias Lohr

2 Likes