Installation error of 3.21.2: CONTENT_ORIGIN is required

Problem:
Pulp_installer complains “CONTENT_ORIGIN is a required”

Expected outcome:
Expect to run database migrations

Pulpcore version:
3.21.2
Pulp plugins installed and their versions:
pulp-rpm 3.18.8
Operating system - distribution and version:
rhel 8.6

Other relevant data:
We got the below error when we run pulp_installer. It is interesting that we see this error on some hosts but not on others. We use the same playbook which has
environment:
DJANGO_SETTINGS_MODULE: pulpcore.app.settings
The /etc/pulp/settings.py has CONTENT_ORIGIN on the failed host
$grep ORIG /etc/pulp/settings.py
CONTENT_ORIGIN = ‘http://pulpserver

The migration works when we manually running
PULP_SETTINGS=/etc/pulp/settings.py /opt/utils/venv/pulp/3.9.7/bin/pulpcore-manager migrate --no-input
and it fails with the same error when run
DJANGO_SETTINGS_MODULE=pulpcore.app.settings /opt/utils/venv/pulp/3.9.7/bin/pulpcore-manager migrate --no-input

Please advise what could cause this and how to fix it

TASK [pulp_database_config : Run database migrations] ******************************************************************
fatal: [pulpserver]: FAILED! => {“changed”: true, “cmd”: ["/opt/utils/venv/pulp/3.9.7/bin/pulpcore-manager", “migrate”, “–no-input”], “delta”: “0:00:01.125680”, “end”: “2022-11-30 06:53:06.662637”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2022-11-30 06:53:05.536957”, “stderr”: “Traceback (most recent call last):\n File “/opt/utils/venv/pulp/3.9.7/bin/pulpcore-manager”, line 8, in \n sys.exit(manage())\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulpcore/app/manage.py”, line 11, in manage\n execute_from_command_line(sys.argv)\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/core/management/init.py”, line 419, in execute_from_command_line\n utility.execute()\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/core/management/init.py”, line 363, in execute\n settings.INSTALLED_APPS\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/conf/init.py”, line 82, in getattr\n self._setup(name)\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/conf/init.py”, line 69, in _setup\n self._wrapped = Settings(settings_module)\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/conf/init.py”, line 170, in init\n mod = importlib.import_module(self.SETTINGS_MODULE)\n File “/usr/lib64/python3.9/importlib/init.py”, line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File “”, line 1030, in _gcd_import\n File “”, line 1007, in _find_and_load\n File “”, line 986, in _find_and_load_unlocked\n File “”, line 680, in _load_unlocked\n File “”, line 850, in exec_module\n File “”, line 228, in _call_with_frames_removed\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulpcore/app/settings.py”, line 361, in \n settings = DjangoDynaconf(\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/contrib/django_dynaconf_v2.py”, line 84, in load\n lazy_settings.populate_obj(django_settings_module)\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 115, in getattr\n self._setup()\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 174, in _setup\n self._wrapped = Settings(\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 253, in init\n self.execute_loaders()\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 1025, in execute_loaders\n self.pre_load(env, silent=silent, key=key)\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 1044, in pre_load\n self.load_file(path=preloads, env=env, silent=silent, key=key)\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 1071, in load_file\n if py_loader.try_to_load_from_py_module_name(\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/loaders/py_loader.py”, line 68, in try_to_load_from_py_module_name\n mod = importlib.import_module(str(name))\n File “/usr/lib64/python3.9/importlib/init.py”, line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File “”, line 1030, in _gcd_import\n File “”, line 1007, in _find_and_load\n File “”, line 986, in _find_and_load_unlocked\n File “”, line 680, in _load_unlocked\n File “”, line 850, in exec_module\n File “”, line 228, in _call_with_frames_removed\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulp_ansible/app/settings.py”, line 24, in \n ANSIBLE_CONTENT_HOSTNAME = settings.CONTENT_ORIGIN + “/pulp/content”\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 115, in getattr\n self._setup()\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 174, in _setup\n self._wrapped = Settings(\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 256, in init\n self.validators.validate(\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/validator.py”, line 461, in validate\n validator.validate(\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/validator.py”, line 213, in validate\n self._validate_items(\n File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/validator.py”, line 276, in _validate_items\n raise ValidationError(_message, details=[(self, _message)])\ndynaconf.validator.ValidationError: CONTENT_ORIGIN is a required setting but it was not configured. This may be caused by invalid read permissions of the settings file. Note that CONTENT_ORIGIN is set by the installer automatically.”, “stderr_lines”: [“Traceback (most recent call last):”, " File “/opt/utils/venv/pulp/3.9.7/bin/pulpcore-manager”, line 8, in “, " sys.exit(manage())”, " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulpcore/app/manage.py”, line 11, in manage", " execute_from_command_line(sys.argv)", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/core/management/init.py”, line 419, in execute_from_command_line", " utility.execute()", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/core/management/init.py”, line 363, in execute", " settings.INSTALLED_APPS", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/conf/init.py”, line 82, in getattr", " self._setup(name)", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/conf/init.py”, line 69, in _setup", " self._wrapped = Settings(settings_module)", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/conf/init.py”, line 170, in init", " mod = importlib.import_module(self.SETTINGS_MODULE)", " File “/usr/lib64/python3.9/importlib/init.py”, line 127, in import_module", " return _bootstrap._gcd_import(name[level:], package, level)", " File “”, line 1030, in _gcd_import", " File “”, line 1007, in _find_and_load", " File “”, line 986, in _find_and_load_unlocked", " File “”, line 680, in _load_unlocked", " File “”, line 850, in exec_module", " File “”, line 228, in _call_with_frames_removed", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulpcore/app/settings.py”, line 361, in “, " settings = DjangoDynaconf(”, " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/contrib/django_dynaconf_v2.py”, line 84, in load", " lazy_settings.populate_obj(django_settings_module)", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 115, in getattr", " self._setup()", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 174, in _setup", " self._wrapped = Settings(", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 253, in init", " self.execute_loaders()", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 1025, in execute_loaders", " self.pre_load(env, silent=silent, key=key)", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 1044, in pre_load", " self.load_file(path=preloads, env=env, silent=silent, key=key)", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 1071, in load_file", " if py_loader.try_to_load_from_py_module_name(", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/loaders/py_loader.py”, line 68, in try_to_load_from_py_module_name", " mod = importlib.import_module(str(name))", " File “/usr/lib64/python3.9/importlib/init.py”, line 127, in import_module", " return _bootstrap._gcd_import(name[level:], package, level)", " File “”, line 1030, in _gcd_import", " File “”, line 1007, in _find_and_load", " File “”, line 986, in _find_and_load_unlocked", " File “”, line 680, in _load_unlocked", " File “”, line 850, in exec_module", " File “”, line 228, in _call_with_frames_removed", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulp_ansible/app/settings.py”, line 24, in “, " ANSIBLE_CONTENT_HOSTNAME = settings.CONTENT_ORIGIN + “/pulp/content””, " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 115, in getattr", " self._setup()", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 174, in _setup", " self._wrapped = Settings(", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/base.py”, line 256, in init", " self.validators.validate(", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/validator.py”, line 461, in validate", " validator.validate(", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/validator.py”, line 213, in validate", " self._validate_items(", " File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/dynaconf/validator.py”, line 276, in _validate_items", " raise ValidationError(_message, details=[(self, _message)])", “dynaconf.validator.ValidationError: CONTENT_ORIGIN is a required setting but it was not configured. This may be caused by invalid read permissions of the settings file. Note that CONTENT_ORIGIN is set by the installer automatically.”], “stdout”: “”, “stdout_lines”: []}

This is expected behavior. We had discussions on whether pulpcore-manager should default use /etc/pulp/settings.py if not specified in the env var PULP_SETTINGS, but we decided against it. Here is the issue tracking this problem and comment with reasoning why we won’t fixed it: https://github.com/pulp/pulpcore/issues/2038#issuecomment-1079351361