Some errors in pulp container under Rocky9: unable to chown /var/lib/pgsql, and some other python error

xm1234567 · September 1, 2023, 2:38pm

Problem:
I am new to podman and pulp3. We have pulp2 in our production, while pulp2 and 3 are quite different.
I followed the Pulp in One Container | software repository management to test pulp3-in-one-container as root on a Rocky9 Virtual machine. I can view pulp status page, but I also saw some errors during the podman run , so I am not sure if I properly set up the container. Here are the messages:

s6-chown: fatal: unable to chown /var/lib/pgsql//////////: Filename too long

and some python errors:
Traceback (most recent call last):
File “/usr/local/bin/pulpcore-manager”, line 8, in
sys.exit(manage())
File “/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py”, line 11, in manage
execute_from_command_line(sys.argv)
File “/usr/local/lib/python3.8/site-packages/django/core/management/init.py”, line 442, in execute_from_command_line
utility.execute()
File “/usr/local/lib/python3.8/site-packages/django/core/management/init.py”, line 382, in execute
settings.INSTALLED_APPS
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 102, in getattr
self._setup(name)
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 89, in _setup
self._wrapped = Settings(settings_module)
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 217, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 843, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py”, line 378, in
settings = DjangoDynaconf(
File “/usr/local/lib/python3.8/site-packages/dynaconf/contrib/django_dynaconf_v2.py”, line 84, in load
lazy_settings.populate_obj(django_settings_module)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 121, in getattr
self._setup()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 180, in _setup
self._wrapped = Settings(
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 262, in init
self.execute_loaders()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1155, in execute_loaders
self.pre_load(env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1180, in pre_load
self.load_file(path=preloads, env=env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1220, in load_file
if py_loader.try_to_load_from_py_module_name(
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 95, in try_to_load_from_py_module_name
mod = importlib.import_module(str(name))
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 843, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.8/site-packages/pulp_ansible/app/settings.py”, line 24, in
ANSIBLE_CONTENT_HOSTNAME = settings.CONTENT_ORIGIN + “/pulp/content”
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 121, in getattr
self._setup()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 180, in _setup
self._wrapped = Settings(
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 262, in init
self.execute_loaders()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1171, in execute_loaders
execute_module_hooks(“post”, self, env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/init.py”, line 109, in execute_module_hooks
hook_module = py_loader.import_from_filename(
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 121, in import_from_filename
if filename in [item.filename for item in inspect.stack()]:
File “/usr/lib64/python3.8/inspect.py”, line 1526, in stack
return getouterframes(sys._getframe(1), context)
File “/usr/lib64/python3.8/inspect.py”, line 1503, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File “/usr/lib64/python3.8/inspect.py”, line 1473, in getframeinfo
filename = getsourcefile(frame) or getfile(frame)
File “/usr/lib64/python3.8/inspect.py”, line 708, in getsourcefile
if getattr(getmodule(object, filename), ‘loader’, None) is not None:
File “/usr/lib64/python3.8/inspect.py”, line 737, in getmodule
file = getabsfile(object, _filename)
File “/usr/lib64/python3.8/inspect.py”, line 721, in getabsfile
return os.path.normcase(os.path.abspath(_filename))
File “/usr/lib64/python3.8/posixpath.py”, line 379, in abspath
cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory
2023-09-01 13:02:29.781 UTC [12608] LOG: redirecting log output to logging collector process
2023-09-01 13:02:29.781 UTC [12608] HINT: Future log output will appear in directory “log”.
Traceback (most recent call last):
File “/usr/local/bin/pulpcore-manager”, line 8, in
sys.exit(manage())
File “/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py”, line 11, in manage
execute_from_command_line(sys.argv)
File “/usr/local/lib/python3.8/site-packages/django/core/management/init.py”, line 442, in execute_from_command_line
utility.execute()
File “/usr/local/lib/python3.8/site-packages/django/core/management/init.py”, line 382, in execute
settings.INSTALLED_APPS
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 102, in getattr
self._setup(name)
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 89, in _setup
self._wrapped = Settings(settings_module)
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 217, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 843, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py”, line 378, in
settings = DjangoDynaconf(
File “/usr/local/lib/python3.8/site-packages/dynaconf/contrib/django_dynaconf_v2.py”, line 84, in load
lazy_settings.populate_obj(django_settings_module)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 121, in getattr
self._setup()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 180, in _setup
self._wrapped = Settings(
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 262, in init
self.execute_loaders()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1155, in execute_loaders
self.pre_load(env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1180, in pre_load
self.load_file(path=preloads, env=env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1220, in load_file
if py_loader.try_to_load_from_py_module_name(
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 95, in try_to_load_from_py_module_name
mod = importlib.import_module(str(name))
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 843, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.8/site-packages/pulp_ansible/app/settings.py”, line 24, in
ANSIBLE_CONTENT_HOSTNAME = settings.CONTENT_ORIGIN + “/pulp/content”
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 121, in getattr
self._setup()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 180, in _setup
self._wrapped = Settings(
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 262, in init
self.execute_loaders()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1171, in execute_loaders
execute_module_hooks(“post”, self, env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/init.py”, line 109, in execute_module_hooks
hook_module = py_loader.import_from_filename(
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 121, in import_from_filename
if filename in [item.filename for item in inspect.stack()]:
File “/usr/lib64/python3.8/inspect.py”, line 1526, in stack
return getouterframes(sys._getframe(1), context)
File “/usr/lib64/python3.8/inspect.py”, line 1503, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File “/usr/lib64/python3.8/inspect.py”, line 1473, in getframeinfo
filename = getsourcefile(frame) or getfile(frame)
File “/usr/lib64/python3.8/inspect.py”, line 708, in getsourcefile
if getattr(getmodule(object, filename), ‘loader’, None) is not None:
File “/usr/lib64/python3.8/inspect.py”, line 737, in getmodule
file = getabsfile(object, _filename)
File “/usr/lib64/python3.8/inspect.py”, line 721, in getabsfile
return os.path.normcase(os.path.abspath(_filename))
File “/usr/lib64/python3.8/posixpath.py”, line 379, in abspath
cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory
Database migrated!
ready
Database migrated!
Traceback (most recent call last):
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 106, in get_module
mod = importlib.import_module(filename)
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 961, in _find_and_load_unlocked
File “”, line 219, in _call_with_frames_removed
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named ‘/etc/pulp/settings’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/bin/pulpcore-manager”, line 8, in
sys.exit(manage())
File “/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py”, line 11, in manage
execute_from_command_line(sys.argv)
File “/usr/local/lib/python3.8/site-packages/django/core/management/init.py”, line 442, in execute_from_command_line
utility.execute()
File “/usr/local/lib/python3.8/site-packages/django/core/management/init.py”, line 382, in execute
settings.INSTALLED_APPS
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 102, in getattr
self._setup(name)
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 89, in _setup
self._wrapped = Settings(settings_module)
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 217, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 843, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py”, line 378, in
settings = DjangoDynaconf(
File “/usr/local/lib/python3.8/site-packages/dynaconf/contrib/django_dynaconf_v2.py”, line 84, in load
lazy_settings.populate_obj(django_settings_module)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 121, in getattr
self._setup()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 180, in _setup
self._wrapped = Settings(
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 262, in init
self.execute_loaders()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1155, in execute_loaders
self.pre_load(env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1180, in pre_load
self.load_file(path=preloads, env=env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1220, in load_file
if py_loader.try_to_load_from_py_module_name(
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 95, in try_to_load_from_py_module_name
mod = importlib.import_module(str(name))
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 843, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.8/site-packages/pulp_ansible/app/settings.py”, line 24, in
ANSIBLE_CONTENT_HOSTNAME = settings.CONTENT_ORIGIN + “/pulp/content”
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 121, in getattr
self._setup()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 180, in _setup
self._wrapped = Settings(
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 262, in init
self.execute_loaders()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1156, in execute_loaders
settings_loader(
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/init.py”, line 275, in settings_loader
py_loader.load(obj, mod_file, key=key, validate=validate)
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 36, in load
mod, loaded_from = get_module(obj, settings_module, silent)
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 110, in get_module
mod = import_from_filename(obj, filename, silent=silent)
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 121, in import_from_filename
if filename in [item.filename for item in inspect.stack()]:
File “/usr/lib64/python3.8/inspect.py”, line 1526, in stack
return getouterframes(sys._getframe(1), context)
File “/usr/lib64/python3.8/inspect.py”, line 1503, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File “/usr/lib64/python3.8/inspect.py”, line 1473, in getframeinfo
filename = getsourcefile(frame) or getfile(frame)
File “/usr/lib64/python3.8/inspect.py”, line 708, in getsourcefile
if getattr(getmodule(object, filename), ‘loader’, None) is not None:
File “/usr/lib64/python3.8/inspect.py”, line 737, in getmodule
file = getabsfile(object, _filename)
File “/usr/lib64/python3.8/inspect.py”, line 721, in getabsfile
return os.path.normcase(os.path.abspath(_filename))
File “/usr/lib64/python3.8/posixpath.py”, line 379, in abspath
cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory
ready

Expected outcome:
Should not have error during podman run

Pulpcore version:
According to the status page, core is “version”: “3.32.0”,

Pulp plugins installed and their versions:
All the plugins included in the container images.

Operating system - distribution and version:
A virtual machine with Rocky9. This vm has some preinstall and configurations required by our environments.
NAME=“Rocky Linux”
VERSION=“9.2 (Blue Onyx)”

Other relevant data:
Another question, is it possbile to just install some plugins? For us, we mainly use rpm.

Thanks in advance

mikedep333 · September 1, 2023, 3:36pm

Hi,

The “Filename too long” error is probably not breaking anything, but is an issue. It was introduced recently.

It looks like the problem is that /etc/pulp/settings.py does not exist. Did you follow these instructions at the beginning of the page?

$ mkdir settings pulp_storage pgsql containers
$ echo "CONTENT_ORIGIN='http://$(hostname):8080'
ANSIBLE_API_HOSTNAME='http://$(hostname):8080'
ANSIBLE_CONTENT_HOSTNAME='http://$(hostname):8080/pulp/content'
CACHE_ENABLED=True" >> settings/settings.py

xm1234567 · September 3, 2023, 4:39pm

Sorry for my late reply. I did follow the instructions including the env setting at the beginning of the page. I just copy&paste, add no other settings. Inside the container, it looks like this:
[root@caba53de0597 /]# cat /etc/pulp/settings.py
CONTENT_ORIGIN=‘http://ccosfip00276.in2p3.fr:8080’
ANSIBLE_API_HOSTNAME=‘http://ccosfip00276.in2p3.fr:8080’
ANSIBLE_CONTENT_HOSTNAME=‘http://ccosfip00276.in2p3.fr:8080/pulp/content’
CACHE_ENABLED=True

I am wondering, should I add PULP_SETTINGS somewhere?
export PULP_SETTINGS=/etc/pulp/settings.py

My virtual machine is Rocky9. Could this be a reason ?

xm1234567 · September 4, 2023, 8:03pm

I added
SECRET_KEY=‘xxxxxxxxxxxxxx’

into settings.py, but still got the python error

lubosmj · September 5, 2023, 9:03am

The PULP_SETTINGS variable should be already defined inside the container for each process. Is that correct, @mikedep333?

@xm1234567, have you ruled out possible permission issues related to the SELinux installation?
https://docs.pulpproject.org/pulpcore/configuration/applying.html#configuration-file
https://pulpproject.org/pulp-in-one-container/#with-selinux

xm1234567 · September 5, 2023, 6:45pm

Selinux is disabled on this Rocky9 vm.
getsebool: SELinux is disabled
I created the container without Selinux
I tried to create a rpm repo foo with pulp cli, it seems working even I got the above python errors during podman run. I will test some other pulp cli for rpm.

xm1234567 · September 7, 2023, 3:59pm

I tried to upload a rpm package manually into a repo:
artificate upload => rpm content create => rpm publication create => rpm distribution => OK

But repo sync is not working, I see the task stay at running forever

If anyone sees the same problem?

Or maybe it is because I got error during the creation of the container?

gerrod · September 13, 2023, 5:50pm

Can you run pulp status and post the output here? It should give info on the workers available to Pulp. Also, if you can run pulp task list --state-in waiting --state-in running this should give info on the currently running tasks.

xm1234567 · September 14, 2023, 3:25pm

Here is the output of pulp status :

{
“pulp_href”: “/pulp/api/v3/remotes/rpm/rpm/018a9447-6a4a-78db-ad08-2c433669c899/”,
“pulp_created”: “2023-09-14T15:20:22.859602Z”,
“name”: “rocky9_extras_remote”,
“url”: “Index of /pub/linux/rocky/9/extras/x86_64/os”,
“ca_cert”: null,
“client_cert”: null,
“tls_validation”: true,
“proxy_url”: null,
“pulp_labels”: {},
“pulp_last_updated”: “2023-09-14T15:20:22.859623Z”,
“download_concurrency”: null,
“max_retries”: null,
“policy”: “immediate”,
“total_timeout”: null,
“connect_timeout”: null,
“sock_connect_timeout”: null,
“sock_read_timeout”: null,
“headers”: null,
“rate_limit”: null,
“hidden_fields”: [
{
“name”: “client_key”,
“is_set”: false
},
{
“name”: “proxy_username”,
“is_set”: false
},
{
“name”: “proxy_password”,
“is_set”: false
},
{
“name”: “username”,
“is_set”: false
},
{
“name”: “password”,
“is_set”: false
}
],
“sles_auth_token”: null
}

xm1234567 · September 14, 2023, 3:39pm

And here is the task list:

pulp task list --state-in waiting --state-in running
[
{
“pulp_href”: “/pulp/api/v3/tasks/018a9448-632f-7498-b558-d01edb57af79/”,
“pulp_created”: “2023-09-14T15:21:26.576399Z”,
“state”: “running”,
“name”: “pulp_rpm.app.tasks.synchronizing.synchronize”,
“logging_cid”: “d162e74b29214889a0bece45aaa0eefa”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: “2023-09-14T15:21:26.658407Z”,
“finished_at”: null,
“error”: null,
“worker”: “/pulp/api/v3/workers/018a756d-ce93-72e9-ac06-315e9b4aa665/”,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/rpm/rpm/018a9448-2496-7431-a586-3feabacbb5b9/”,
“shared:/pulp/api/v3/remotes/rpm/rpm/018a9447-6a4a-78db-ad08-2c433669c899/”,
“shared:/pulp/api/v3/domains/018a756c-3d85-78da-ae76-1582e92c8185/”
]
}
]

The task looks stucked:

pulp rpm repository sync --name rocky9_extras
Started background task /pulp/api/v3/tasks/018a9448-632f-7498-b558-d01edb57af79/
…

The thing is:
it is a small remote repo for syno.
I tested this synco job on another Pulp 3 instance (installed by the way of ansible play book, pulp 3.22.0), the task finished quickly.
So there is something wrong with this pulp3 instance (installed by way of pulp-in-one-container 3.32).

That is why, I put at the the beginning of this thread, asking about the python errors I got during the podman run.

gerrod · September 14, 2023, 3:49pm

Looking at the worker info in your status output shows that there are two online workers, but neither of them are running a task even though your task output shows that the sync task is running. This means the task somehow got lost (maybe a worker picked up the task before it got killed) and will never complete. So I recommend cancelling the task with pulp task cancel --running and then restart the sync.

xm1234567 · September 14, 2023, 7:23pm

Sorry, strange the output of previous pulp status, it is not correct at all. I did a bad copy-paste, and didn’t notice.

Here is the right one. I cancel all job, re start the sync, then check the pulp status, and we can see one worker is running:
→ re start sync

pulp rpm repository sync --name rocky9_extras
Started background task /pulp/api/v3/tasks/018a9520-5dc4-7884-9c50-6041600cbac4/
…^CTask /pulp/api/v3/tasks/018a9520-5dc4-7884-9c50-6041600cbac4/ sent to background.

→ task status

pulp task list --state-in running --state-in waiting
[
{
“pulp_href”: “/pulp/api/v3/tasks/018a9520-5dc4-7884-9c50-6041600cbac4/”,
“pulp_created”: “2023-09-14T19:17:20.966130Z”,
“state”: “running”,
“name”: “pulp_rpm.app.tasks.synchronizing.synchronize”,
“logging_cid”: “c4bca8628e9a4fdba7f3f9ff7bdb2484”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: “2023-09-14T19:17:21.177665Z”,
“finished_at”: null,
“error”: null,
“worker”: “/pulp/api/v3/workers/018a756d-d102-71d7-89bf-30418469e151/”,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/rpm/rpm/018a9448-2496-7431-a586-3feabacbb5b9/”,
“shared:/pulp/api/v3/remotes/rpm/rpm/018a9447-6a4a-78db-ad08-2c433669c899/”,
“shared:/pulp/api/v3/domains/018a756c-3d85-78da-ae76-1582e92c8185/”
]
}
]

→ pulp status

pulp status
{
“versions”: [
{
“component”: “core”,
“version”: “3.32.0”,
“package”: “pulpcore”,
“domain_compatible”: true
},
{
“component”: “ansible”,
“version”: “0.19.0”,
“package”: “pulp-ansible”,
“domain_compatible”: false
},
{
“component”: “certguard”,
“version”: “1.6.5”,
“package”: “pulp-certguard”,
“domain_compatible”: true
},
{
“component”: “container”,
“version”: “2.16.0”,
“package”: “pulp-container”,
“domain_compatible”: false
},
{
“component”: “deb”,
“version”: “2.21.1”,
“package”: “pulp_deb”,
“domain_compatible”: false
},
{
“component”: “file”,
“version”: “1.14.4”,
“package”: “pulp-file”,
“domain_compatible”: true
},
{
“component”: “gem”,
“version”: “0.2.0”,
“package”: “pulp-gem”,
“domain_compatible”: false
},
{
“component”: “maven”,
“version”: “0.6.0”,
“package”: “pulp-maven”,
“domain_compatible”: false
},
{
“component”: “ostree”,
“version”: “2.1.1”,
“package”: “pulp-ostree”,
“domain_compatible”: false
},
{
“component”: “python”,
“version”: “3.10.0”,
“package”: “pulp-python”,
“domain_compatible”: false
},
{
“component”: “rpm”,
“version”: “3.22.3”,
“package”: “pulp-rpm”,
“domain_compatible”: true
}
],
“online_workers”: [
{
“pulp_href”: “/pulp/api/v3/workers/018a756d-ce93-72e9-ac06-315e9b4aa665/”,
“pulp_created”: “2023-09-08T15:34:05.205268Z”,
“name”: “12653@2712badf2f30”,
“last_heartbeat”: “2023-09-14T19:17:41.307424Z”,
“current_task”: null
},
{
“pulp_href”: “/pulp/api/v3/workers/018a756d-d102-71d7-89bf-30418469e151/”,
“pulp_created”: “2023-09-08T15:34:05.835121Z”,
“name”: “12703@2712badf2f30”,
“last_heartbeat”: “2023-09-14T19:17:41.434519Z”,
“current_task”: “/pulp/api/v3/tasks/018a9520-5dc4-7884-9c50-6041600cbac4/”
}
],
“online_content_apps”: [
{
“name”: “12715@2712badf2f30”,
“last_heartbeat”: “2023-09-14T19:17:36.342205Z”
},
{
“name”: “12714@2712badf2f30”,
“last_heartbeat”: “2023-09-14T19:17:36.871418Z”
}
],
“database_connection”: {
“connected”: true
},
“redis_connection”: {
“connected”: false
},
“storage”: {
“total”: 21464350720,
“used”: 262213632,
“free”: 21202137088
},
“content_settings”: {
“content_origin”: “http://ccosfip00276.in2p3.fr:80”,
“content_path_prefix”: “/pulp/content/”
},
“domain_enabled”: false
}

xm1234567 · September 14, 2023, 7:29pm

Another question: for pulp-in-one-container, I see there is something s6-layer…, but I don’t know much about how to manage pulp in container style. Eg, how to check /restart the status of service, like Postgres? Redis? pulp-worker? How to access the DB if needed? How to trouble-shooting?
If you have any experience to share, it is hgihly appreciate

mikedep333 · October 9, 2023, 8:06pm

Hi @xm1234567 ,

Sorry for the delayed reply.

You can restart services like:
podman exec pulp s6-svc -r /run/service/nginx

You can manage postgres lke:
podman exec pulp -it psql

See also:
https://docs.pulpproject.org/pulp_oci_images/multi-process-images/#debugging-the-services

xm1234567 · November 10, 2023, 4:47pm

Hello @mikedep333
Thanks for your reply.
Sorry for my late reply also. I am new to container and django, I take time to read some doc, to try to better understand pulp.

I have set up the container (run as root on port 443), pulp status ok, pulp admin access OK
Create repo, OK.
Upload a rpm to repo, OK
But a repo sync is not OK.

The sync task is running forever, here is the debug:

(pulp_venv) [root@mysever ~]# pulp -v --profile admin rpm repository sync --sync-policy "mirror_content_only" --name "rocky9_extras"
repositories_rpm_rpm_list : get https://mysever.example.com/pulp/api/v3/repositories/rpm/rpm/?name=rocky9_extras&offset=0&limit=1
Response: 200
repositories_rpm_rpm_sync : post https://mysever.example.com/pulp/api/v3/repositories/rpm/rpm/018bb9e6-57ca-7898-bd0b-5c506f1eba5d/sync/
Response: 202
tasks_read : get https://mysever.example.com/pulp/api/v3/tasks/018bba03-eafd-706e-a04e-2f8d0c171b21/
Response: 200
Started background task /pulp/api/v3/tasks/018bba03-eafd-706e-a04e-2f8d0c171b21/
.tasks_read : get https://mysever.example.com/pulp/api/v3/tasks/018bba03-eafd-706e-a04e-2f8d0c171b21/
Response: 200
.tasks_read : get https://mysever.example.com/pulp/api/v3/tasks/018bba03-eafd-706e-a04e-2f8d0c171b21/
Response: 200
.tasks_read : get https://mysever.example.com/pulp/api/v3/tasks/018bba03-eafd-706e-a04e-2f8d0c171b21/
Response: 200
.tasks_read : get https://mysever.example.com/pulp/api/v3/tasks/018bba03-eafd-706e-a04e-2f8d0c171b21/
Response: 200
.tasks_read : get https://mysever.example.com/pulp/api/v3/tasks/018bba03-eafd-706e-a04e-2f8d0c171b21/
Response: 200
^CTask /pulp/api/v3/tasks/018bba03-eafd-706e-a04e-2f8d0c171b21/ sent to background.
(pulp_venv) [root@mysever ~]# pulp task cancel --running

Is this a network problem of the container?
Or it is related with the error when the container was started? the error is as I describe in my ticket: the
python error and ModuleNotFoundError: No module named '/etc/pulp/settings',

I have tried the same sync job on the pulp instance installed by pulp_installer, the sync finished in less than 30s. So there is something NOK on my pulp container instance .

xm1234567 · November 10, 2023, 4:31pm

or should I open another ticket for my repo synco problem?

xm1234567 · November 21, 2023, 2:15pm

I have redo from scratch for pulp container, but this time as rootless container, not as root.
The first time podman run , I still see these python error in logs, but repo sync is working!
After I restart the container, I didn’t see these python error.

So for me, I feel :
run container as root => repo sync NOT working
run container rootless => repo sync working