Pulp in one container errors

Support
#1

Problem:
Getting errors to start pulp in one container
Expected outcome:

Pulpcore version:
3.36.0
Pulp plugins installed and their versions:
pulp-ansible 0.20.0
pulp-certguard 1.6.5
pulp-container 2.16.2
pulp-deb 3.0.0
pulp-file 1.14.5
pulp-gem 0.3.0
pulp-glue 0.21.3
pulp-maven 0.6.0
pulp-ostree 2.1.3
pulp-python 3.10.0
pulp-rpm 3.22.5
pulpcore 3.36.0
Operating system - distribution and version:
rhel 8.6
Other relevant data:

Getting errors to run the pulp in one container
Use below command to run the container:
podman run
–publish 8080:80
–name pulp
–volume “$(pwd)/settings”:/etc/pulp:Z
–volume “$(pwd)/pulp_storage”:/var/lib/pulp:Z
–volume “$(pwd)/pgsql”:/var/lib/pgsql:Z
–volume “$(pwd)/containers”:/var/lib/containers:Z
–volume “/etc/ssl/certs/ca-bundle.trust.crt:/etc/ssl/certs/ca-bundle.trust.crt”
–device /dev/fuse
pulp/stable

Error:
023-10-06 18:00:36.128 UTC [333] LOG: redirecting log output to logging collector process
2023-10-06 18:00:36.128 UTC [333] HINT: Future log output will appear in directory “log”.
Traceback (most recent call last):
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 106, in get_module
mod = importlib.import_module(filename)
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 961, in _find_and_load_unlocked
File “”, line 219, in _call_with_frames_removed
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named ‘/etc/pulp/settings’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/bin/pulpcore-manager”, line 8, in
sys.exit(manage())
File “/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py”, line 11, in manage
execute_from_command_line(sys.argv)
File “/usr/local/lib/python3.8/site-packages/django/core/management/init.py”, line 442, in execute_from_command_line
utility.execute()
File “/usr/local/lib/python3.8/site-packages/django/core/management/init.py”, line 382, in execute
settings.INSTALLED_APPS
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 102, in getattr
self._setup(name)
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 89, in _setup
self._wrapped = Settings(settings_module)
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 217, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 843, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py”, line 378, in
settings = DjangoDynaconf(
File “/usr/local/lib/python3.8/site-packages/dynaconf/contrib/django_dynaconf_v2.py”, line 86, in load
lazy_settings.populate_obj(django_settings_module)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 122, in getattr
self._setup()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 183, in _setup
self._wrapped = self._wrapper_class(
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 271, in init
self.execute_loaders()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1159, in execute_loaders
self.pre_load(env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1184, in pre_load
self.load_file(path=preloads, env=env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1224, in load_file
if py_loader.try_to_load_from_py_module_name(
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 95, in try_to_load_from_py_module_name
mod = importlib.import_module(str(name))
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 843, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.8/site-packages/pulp_ansible/app/settings.py”, line 24, in
ANSIBLE_CONTENT_HOSTNAME = settings.CONTENT_ORIGIN + “/pulp/content”
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 122, in getattr
self._setup()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 183, in _setup
self._wrapped = self._wrapper_class(
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 271, in init
self.execute_loaders()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1160, in execute_loaders
settings_loader(
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/init.py”, line 303, in settings_loader
py_loader.load(obj, mod_file, key=key, validate=validate)
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 36, in load
mod, loaded_from = get_module(obj, settings_module, silent)
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 110, in get_module
mod = import_from_filename(obj, filename, silent=silent)
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 121, in import_from_filename
if filename in [item.filename for item in inspect.stack()]:
File “/usr/lib64/python3.8/inspect.py”, line 1526, in stack
return getouterframes(sys._getframe(1), context)
File “/usr/lib64/python3.8/inspect.py”, line 1503, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File “/usr/lib64/python3.8/inspect.py”, line 1473, in getframeinfo
filename = getsourcefile(frame) or getfile(frame)
File “/usr/lib64/python3.8/inspect.py”, line 708, in getsourcefile
if getattr(getmodule(object, filename), ‘loader’, None) is not None:
File “/usr/lib64/python3.8/inspect.py”, line 737, in getmodule
file = getabsfile(object, _filename)
File “/usr/lib64/python3.8/inspect.py”, line 721, in getabsfile
return os.path.normcase(os.path.abspath(_filename))
File “/usr/lib64/python3.8/posixpath.py”, line 379, in abspath
cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory
Database migrated!
ready

I confirmed the container has the /etc/pulp volume mounted.
[pulp@pt-app322m pulp]$ podman exec -it 2edc664cb37f bash
[root@2edc664cb37f /]# ls /etc/pulp
certs save settings.py
[root@2edc664cb37f /]# more /etc/pulp/settings.py
CONTENT_ORIGIN=‘http://localhost:8080’
ANSIBLE_API_HOSTNAME=‘http://localhost:8080’
ANSIBLE_CONTENT_HOSTNAME=‘http://localhost:8080/pulp/content’
CACHE_ENABLED=True

If I mount to /etc/pulp/setting. I don’t have the first error "ModuleNotFoundError: No module named ‘/etc/pulp/settings’ but still getting
Traceback (most recent call last):
File “/usr/local/bin/pulpcore-manager”, line 8, in
sys.exit(manage())
File “/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py”, line 11, in manage
execute_from_command_line(sys.argv)
File “/usr/local/lib/python3.8/site-packages/django/core/management/init.py”, line 442, in execute_from_command_line
utility.execute()
File “/usr/local/lib/python3.8/site-packages/django/core/management/init.py”, line 382, in execute
settings.INSTALLED_APPS
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 102, in getattr
self._setup(name)
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 89, in _setup
self._wrapped = Settings(settings_module)
File “/usr/local/lib/python3.8/site-packages/django/conf/init.py”, line 217, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 843, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py”, line 378, in
settings = DjangoDynaconf(
File “/usr/local/lib/python3.8/site-packages/dynaconf/contrib/django_dynaconf_v2.py”, line 86, in load
lazy_settings.populate_obj(django_settings_module)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 122, in getattr
self._setup()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 183, in _setup
self._wrapped = self._wrapper_class(
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 271, in init
self.execute_loaders()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1159, in execute_loaders
self.pre_load(env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1184, in pre_load
self.load_file(path=preloads, env=env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1224, in load_file
if py_loader.try_to_load_from_py_module_name(
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 95, in try_to_load_from_py_module_name
mod = importlib.import_module(str(name))
File “/usr/lib64/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 843, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.8/site-packages/pulp_ansible/app/settings.py”, line 24, in
ANSIBLE_CONTENT_HOSTNAME = settings.CONTENT_ORIGIN + “/pulp/content”
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 122, in getattr
self._setup()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 183, in _setup
self._wrapped = self._wrapper_class(
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 271, in init
self.execute_loaders()
File “/usr/local/lib/python3.8/site-packages/dynaconf/base.py”, line 1175, in execute_loaders
execute_module_hooks(“post”, self, env, silent=silent, key=key)
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/init.py”, line 116, in execute_module_hooks
hook_module = py_loader.import_from_filename(
File “/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py”, line 121, in import_from_filename
if filename in [item.filename for item in inspect.stack()]:
File “/usr/lib64/python3.8/inspect.py”, line 1526, in stack
return getouterframes(sys._getframe(1), context)
File “/usr/lib64/python3.8/inspect.py”, line 1503, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File “/usr/lib64/python3.8/inspect.py”, line 1473, in getframeinfo
filename = getsourcefile(frame) or getfile(frame)
File “/usr/lib64/python3.8/inspect.py”, line 708, in getsourcefile
if getattr(getmodule(object, filename), ‘loader’, None) is not None:
File “/usr/lib64/python3.8/inspect.py”, line 737, in getmodule
file = getabsfile(object, _filename)
File “/usr/lib64/python3.8/inspect.py”, line 721, in getabsfile
return os.path.normcase(os.path.abspath(_filename))
File “/usr/lib64/python3.8/posixpath.py”, line 379, in abspath
cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory

I also try pulp-minimum image and I did have any errors when mount the settings to /etc/pulp.
Please advise how to trouble this issue.

#2

I tried reproducing the problem with the following command:

podman run --publish 8080:80 --name pulp --volume "$(pwd)/settings":/etc/pulp:Z --volume "$(pwd)/pulp_storage":/var/lib/pulp:Z --volume "$(pwd)/pgsql":/var/lib/pgsql:Z --volume "$(pwd)/containers":/var/lib/containers:Z --device /dev/fuse pulp/pulp:stable

However, everything started correctly. Check the permissions on directories you are mounting as volumes.

My exact image:

REPOSITORY                                    TAG             IMAGE ID      CREATED        SIZE
quay.io/pulp/pulp                             stable          fbd157e97eee  3 days ago     2.13 GB
1 Like
#3

Den, It looks it works perfectly the first time I run the container. This only happens after stop and remove the container. Any idea how I should rerun podman to make it work the second time?

#4

Hello @bli111

Have you found anything about the error you got?
I got the similar error ModuleNotFoundError: No module named '/etc/pulp/settings', and my ticket is here.

My environment is

OS: Rocky9. 
OS Python:  Python 3.9.16

I use the pulp image.

And have you tried repo sync with the pulp container? It is not working for me.

But create repo and upload rpm to repo => OK for me.

#5

I am still getting the same message. It looks like the error is from /etc/init/postgres-prepare but it works fine when I run it manually inside the container. The error only occurred when I stop and restart the container again. I don’t have the error when I run the first time. I wonder if it is related to podman because I had to run ‘podman unshare -R chown 26:26’ to fix some permission issue.

#6

Hi @bli111

I redo the container from scratch (delete images, delete container, delete logs…), and this time I do rootless container. This time I still see these python error when container first begins, but repo sync is working at lease ! Then I stop, and restart the container, during the restart of container, I didn’t see the python error. I don’t know why, but at least a good news for me is the repo sync working!

All my previous test, I did container as root, maybe that was the reason of repo sync not working.

I didn’t do podman unshare thing, seems it is done by the container image already. (of course I have to set /etc/subuid and subgid before)
(pulpcli_venv) [xxx@xxxxxx ~]$ ls /opt/pulp/ -l
total 0
drwxr-xr-x 2 xxx xxxxx 6 Nov 21 14:44 containers
drwxr-x— 3 165561 165561 18 Nov 21 14:47 pgsql
drwxr-xr-x 6 166235 166235 59 Nov 21 14:47 pulp_storage
drwxr-xr-x 3 xxx xxxxx 38 Nov 21 14:47 settings

1 Like