Postgresql database exists but will be upgraded from 13 to 16 : Fails

Problem.

Please assist.

From container log.(log below)

Postgresql database exists but will be upgraded from 13 to 16 : Fails

Updating from /pulp/pulp3:84 to /pulp/pulp3.86
Then 3.84.0 gives same error.

Note:
Proxy information has never been part of any running instance of Pulp.

Gives this error on starting.

Proxy details were never made available before this version 3.84
Seems 3.84.0 started to need it.
Proxy details are given in the image startup environment and in the settings.py file.

At the time the curl is tried here,
CentOS Stream 9 - BaseOS                        0.0  B/s |   0  B     00:00
Errors during downloading metadata for repository 'baseos':
  - Curl error (6): Couldn't resolve host name,

the proxy detail is not available it seems.

Trying the URL from a login to the container the URL can be reached as proxy environment variables are set.

https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http

bash-5.1$ curl https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http
[1] 1931
[2] 1932
bash-5.1$ <?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Tue, 26 Aug 2025 11:22:11 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">
<!-- # either path=, or repo= and arch= must be specified -->

 
bash-5.1$ env|grep prox
no_proxy=localhost,127.0.0.1,.nuc,.group.net
https_proxy=http://lpmaster0.group.net:3128
http_proxy=http://lpmaster0.group.net:3128

bash-5.1$ id
uid=700(pulp) gid=700(pulp) groups=700(pulp)

Are there a new/changed way to start the image?
Please point me to the URL (that I am missing to implement 3.84.0 onwards)

Pulpcore version:

Image version still Fine

          "Id": "b1d9f444c04f2ffdc03bd5f75f2d6cb840debbb49ea60f84ba5466104babbf37",
          "Digest": "sha256:3fff1f19df868d535c4a18a95d882e76c5a64e3de0aa3b7799ccf59f236c5ecf",

/pulp/pulp:3.84 2.17GB     

Image Version broken

          "Id": "336676e8ad2f93ca53b56548308a8a10a18815858b164a30f556cb3ae57a44c7",
          "Digest": "sha256:ec4b00a93d17d7957ea9813c58525cb5d5011c35fda92a7eeb0740155f2a2427",

/pulp/pulp:3.84 1.63 GB

Operating system - distribution and version:

SUSE Linux Enterprise Server 15 SP7

Container startup error.

s6-rc: info: service certs: starting
Checking /var/lib/pulp/media permissions
Calling /etc/init/certs
Checking /var/lib/pulp/scripts permissions
Checking /var/lib/pulp/tmp permissions
s6-rc: info: service fix-pulp-perms successfully started
[oneshot] certs: adding webserver certificate to the certificate store
[oneshot] postgres-init:  Postgresql database exists but will be upgraded from 13 to 16
CentOS Stream 9 - BaseOS                        0.0  B/s |   0  B     00:00
Errors during downloading metadata for repository 'baseos':
  - Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Could not resolve host: mirrors.centos.org]
  - Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http&countme=1 [Could not resolve host: mirrors.centos.org]
Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Could not resolve host: mirrors.centos.org]
bash: line 1: /usr/pgsql-13/bin/pg_ctl: No such file or directory
psql: error: connection to server on socket "/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
bash: line 1: /usr/pgsql-13/bin/pg_ctl: No such file or directory
[oneshot] postgres-init: initdb -D /var/lib/pgsql/data_new -E  --locale= --auth=trust
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C".
initdb: error: "--locale=" is not a valid server encoding name
[oneshot] postgres-init:  failed to initialize the new database
s6-rc: warning: unable to start service postgres-init: command exited 1
[oneshot] certs: finished adding webserver certificate to the certificate store
s6-rc: info: service certs successfully started
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more
 information.

@Janr In pulpcore 3.85 we decided to upgrade the postgresql version from 13 to 16. Since it took us a little while to get all the plugin 3.85 releases out it seems that the later 3.84 images picked up the database upgrade code. This upgrade is a one time process and it involves downloading the old 13 binaries to perform the upgrade. We didn’t want to ship the 13 & 16 binaries for all future images, so it is only downloaded the first time you upgrade, but looks like we forgot to think about the scenarios where Pulp is behind a proxy and can’t perform the download. In your case we can still manually upgrade the database.

On a pulp/pulp:3.86 image, shell into the image and perform the following commands:

# if you haven't set your proxy env vars, do so now
dnf -y install postgresql13 postgresql13-server postgresql13-contrib
/etc/init/postgres-init

After this you should see in your pgsql/data folder the file PG_VERSION should be 16. You should also have a new folder called pgsql/data_old where the old version 13 files are if you need to downgrade to a pulpcore version <3.84. Now you should be able to start up 3.86 image again. Let me know if you run into anymore problems.

2 Likes

Hello gerrod.

Thanks so much, this worked 100%.
3.86 started.

[root@pulp_mirror /]# /etc/init/postgres-init
[oneshot] postgres-init:  Postgresql database exists but will be upgraded from 13 to 16
.
.
.

Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, gem, maven, npm, ostree, python, rpm, sessions
Running migrations:
  Applying ansible.0065_collectionversion_ansible_col_namespa_ea6f75_idx... OK
  Applying core.0136_delete_basedistribution... OK
  Applying core.0137_appstatus... OK
  Applying core.0138_vulnerabilityreport... OK
  Applying core.0139_task_app_lock... OK
  Applying deb.0031_add_domains... OK
  Applying python.0015_alter_pythonpackagecontent_options... OK
Access policy for content/python/packages updated.

.
.
.

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade.
Once you start the new server, consider running:
    /usr/pgsql-16/bin/vacuumdb --all --analyze-in-stages

Thanks a mil!

3 Likes

gerrod

TypeError: unhashable type: 'list'

We are getting this error with 3.86.
Only in the Distribution task, there are ±32 repos.

Some Distributions work, many fails.

We have a CI/CD with 5 hosts in a Matrix.

Only 3.86 gives this error

This is for information only.
We will wait for a later 3.86…

TASK [Create rpm Distribution of Repositories] *********************************
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES12-SP5', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5/', 'base': 'MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5'})
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP7', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP7/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP7/', 'base': 'MR/SLES15/SP7/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP7'})
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP6', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6/', 'base': 'MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6'})
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP5', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP5/', 'base': 'MR/SLES15/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP5'})
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP4', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4/', 'base': 'MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4'})
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES12-SP4', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES12/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP4/', 'base': 'MR/SLES12/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP4'})
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: unhashable type: 'list'
failed: [localhost] (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP4-latest', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4-latest/', 'base': 'MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4-latest'}) => changed=false 
  ansible_index_var: d_idx
  ansible_loop_var: item
  d_idx: 6
  item:
    backend: https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4-latest/
    base: MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4-latest
    name: active-P-2025Q03.00-ABC-SLES15-SP4-latest
  msg: 'unhashable type: ''list'''
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: unhashable type: 'list'
failed: [localhost] (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP6-latest', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6-latest/', 'base': 'MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6-latest'}) => changed=false 
  ansible_index_var: d_idx
  ansible_loop_var: item
  d_idx: 7
  item:
    backend: https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6-latest/
    base: MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6-latest
    name: active-P-2025Q03.00-ABC-SLES15-SP6-latest
  msg: 'unhashable type: ''list'''
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: unhashable type: 'list'
failed: [localhost] (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP5-latest', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP5-latest/', 'base': 'MR/SLES15/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP5-latest'}) => changed=false 
  ansible_index_var: d_idx
  ansible_loop_var: item
  d_idx: 8

Thanks again.

This looks like an Ansible playbook run. But then again not very familiar out put. Can you find a post the very distribution task from the playbook that fails here? (Obviously, please strip sensitive info.)

Thank you.

Just a repeat, four out of the five servers are fine from the same Playbook.
With the same pulp image configuration, save the one with the 3.86 image.

Working builds: pulp/pulp 3.83 153feacb8645
pulp.squeezer:0.1.0

    - name: Run-2 Create rpm Publication, 1st run do not create packages.
      pulp.squeezer.rpm_publication:
        pulp_url: '{{ PULP_MIRROR_SVR }}'
        username: admin
        password: '{{ PASSWD }}'
        repository: "{{ item.name }}"
        validate_certs: false
        state: present
        timeout: 7200
      register: pub_result_2
      loop: "{{ repo_data.Repository }}"
      loop_control:
         index_var: p_idx_2


    - name: Create rpm Distribution of Repositories
      pulp.squeezer.rpm_distribution:
        pulp_url: '{{ PULP_MIRROR_SVR }}'
        username: admin
        password: '{{ PASSWD }}'
        name: "{{ item.name }}"
        base_path: "{{ item.base }}"
        publication: "{{ pub_result_2.results[d_idx].publication.pulp_href }}"
        validate_certs: false
        state: present
        timeout: 7200
      register: dist_result
      loop: "{{ repo_data.Repository }}"
      loop_control:
         index_var: d_idx

Repodata.yml as in the loop - “repo_data.Repository”

Repository:

 - name: active-P-2025Q03.00-ABC-SLES12-SP5
   backend: https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5/
   base: MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5

 - name: active-P-2025Q03.00-ABC-SLES15-SP7
   backend: https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP7/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP7/
   base: MR/SLES15/SP7/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP7

 - name: active-P-2025Q03.00-ABC-SLES15-SP6
   backend: https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6/
   base: MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6

I cannot see what list it is talking about.
Also squeezer 0.1.0 is rather old. Can you update it?

Hi.

The problem was that 3.86’s sync was not completed by the time the publication task ran.
On 3.83/4 this was not the case.
The playbook was adjusted to wait for the sync to complete.

The sync in 3.86 took longer.

2025-09-01T08:38:24.806659000+02:00 pulp [24f230f635824c09b53574509b66452c]: pulpcore.tasking.tasks:INFO: Task completed 019903f7-f45e-778d-b095-2c720eb05998 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 71860950 μs
2025-09-01T08:37:34.720977000+02:00 pulp [6a16efd3b0884f3daf0bacd2f458ea8c]: pulpcore.tasking.tasks:INFO: Task completed 019903f7-f1bc-7d0b-91e5-26a8a2f63152 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 84567227 μs
2025-09-01T08:39:10.724830000+02:00 pulp [2e93c6fbc8814baa8d22fe1e220f388d]: pulpcore.tasking.tasks:INFO: Task completed 019903f7-fba7-7306-a54a-15b8b03761a7 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 91773600 μs
2025-09-01T08:44:46.042523000+02:00 pulp [ec0322dab44c47f2b3c2ec51dd1f1025]: pulpcore.tasking.tasks:INFO: Task completed 019903f8-0ec8-7103-a633-b4e48047b983 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 328351543 μs
2025-09-01T08:44:36.292905000+02:00 pulp [3a752a318a8249b79f19dba53426b726]: pulpcore.tasking.tasks:INFO: Task completed 019903f7-f929-7fb9-8042-3f24e3f409f0 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 340074624 μs
2025-09-01T08:36:09.254611000+02:00 pulp [ff6199a8e8a7452b9bd8a640ebc3c636]: pulpcore.tasking.tasks:INFO: Task completed 019903f7-ea77-726b-89f3-5360e1e3592f in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 366043272 μs
2025-09-01T08:36:17.669032000+02:00 pulp [424d39f9005c4ceeb52366b23aee4b17]: pulpcore.tasking.tasks:INFO: Task completed 019903f7-ef37-75c4-b8d1-509ecface4e0 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 373225019 μs
2025-09-01T08:45:27.617528000+02:00 pulp [30012349ac6648d282fe01bf8ca7de74]: pulpcore.tasking.tasks:INFO: Task completed 019903f7-fded-7340-9fbc-4f4689bb3233 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 417526220 μs
2025-09-01T08:37:12.041126000+02:00 pulp [c8862780696d48fd95b0cb0350c742f6]: pulpcore.tasking.tasks:INFO: Task completed 019903f7-e82e-7d06-9d4f-3b62c91a9232 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 429390917 μs
2025-09-01T08:45:07.824555000+02:00 pulp [13e7363a6016434c9ac5381cb9ef3ba4]: pulpcore.tasking.tasks:INFO: Task completed 019903f7-f6c8-7799-b987-306d9a1fc965 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 529324531 μs
2025-09-01T08:38:55.409029000+02:00 pulp [8a24316a44634ec3b54a76508fd10a56]: pulpcore.tasking.tasks:INFO: Task completed 019903f7-ecdc-7105-ab92-7cdad6d508a0 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 531557239 μs

This now works, thanks for the help.
Squeezer will be updated, sure thing.

1 Like

Hi x94c

Pulp squeezer is updated to 0.2.1 in our docker image.

pulp-cli           0.35.0
pulp-glue          0.35.0

The pulp-minimal image running:

pulp/pulp        3.87          4405271a1d74

Here pulp-glue 0.31.2 is installed.

The error now is:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: Installed 'pulp-glue' version '0.35.0' is not in '>=0.29.2,<0.35'.
failed: [localhost] (item={'name': 'active-P-2025Q03.00-ABC-SLES12-SP5', 'backend': 'https://ltcli01.abcgroup.net:82/pulp/content/MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5/', 'base': 'MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5'}) => changed=false 
  ansible_loop_var: item
  item:
    backend: https://ltcli01.abcgroup.net:30082/pulp/content/MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5/
    base: MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5
    name: active-P-2025Q03.00-ABC-SLES12-SP5
  msg: Failed to import the required Python library (pulp-glue) on 7564c8ed38f4's Python /usr/bin/python3.11. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter

Please suggest what to do?
Tanks