Postgresql database exists but will be upgraded from 13 to 16 : Fails

Problem.

Please assist.

From container log.(log below)

Postgresql database exists but will be upgraded from 13 to 16 : Fails

Updating from /pulp/pulp3:84 to /pulp/pulp3.86
Then 3.84.0 gives same error.

Note:
Proxy information has never been part of any running instance of Pulp.

Gives this error on starting.

Proxy details were never made available before this version 3.84
Seems 3.84.0 started to need it.
Proxy details are given in the image startup environment and in the settings.py file.

At the time the curl is tried here,
CentOS Stream 9 - BaseOS                        0.0  B/s |   0  B     00:00
Errors during downloading metadata for repository 'baseos':
  - Curl error (6): Couldn't resolve host name,

the proxy detail is not available it seems.

Trying the URL from a login to the container the URL can be reached as proxy environment variables are set.

https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http

bash-5.1$ curl https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http
[1] 1931
[2] 1932
bash-5.1$ <?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Tue, 26 Aug 2025 11:22:11 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">
<!-- # either path=, or repo= and arch= must be specified -->

 
bash-5.1$ env|grep prox
no_proxy=localhost,127.0.0.1,.nuc,.group.net
https_proxy=http://lpmaster0.group.net:3128
http_proxy=http://lpmaster0.group.net:3128

bash-5.1$ id
uid=700(pulp) gid=700(pulp) groups=700(pulp)

Are there a new/changed way to start the image?
Please point me to the URL (that I am missing to implement 3.84.0 onwards)

Pulpcore version:

Image version still Fine

          "Id": "b1d9f444c04f2ffdc03bd5f75f2d6cb840debbb49ea60f84ba5466104babbf37",
          "Digest": "sha256:3fff1f19df868d535c4a18a95d882e76c5a64e3de0aa3b7799ccf59f236c5ecf",

/pulp/pulp:3.84 2.17GB     

Image Version broken

          "Id": "336676e8ad2f93ca53b56548308a8a10a18815858b164a30f556cb3ae57a44c7",
          "Digest": "sha256:ec4b00a93d17d7957ea9813c58525cb5d5011c35fda92a7eeb0740155f2a2427",

/pulp/pulp:3.84 1.63 GB

Operating system - distribution and version:

SUSE Linux Enterprise Server 15 SP7

Container startup error.

s6-rc: info: service certs: starting
Checking /var/lib/pulp/media permissions
Calling /etc/init/certs
Checking /var/lib/pulp/scripts permissions
Checking /var/lib/pulp/tmp permissions
s6-rc: info: service fix-pulp-perms successfully started
[oneshot] certs: adding webserver certificate to the certificate store
[oneshot] postgres-init:  Postgresql database exists but will be upgraded from 13 to 16
CentOS Stream 9 - BaseOS                        0.0  B/s |   0  B     00:00
Errors during downloading metadata for repository 'baseos':
  - Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Could not resolve host: mirrors.centos.org]
  - Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http&countme=1 [Could not resolve host: mirrors.centos.org]
Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Could not resolve host: mirrors.centos.org]
bash: line 1: /usr/pgsql-13/bin/pg_ctl: No such file or directory
psql: error: connection to server on socket "/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
psql: error: connection to server on socket "/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
bash: line 1: /usr/pgsql-13/bin/pg_ctl: No such file or directory
[oneshot] postgres-init: initdb -D /var/lib/pgsql/data_new -E  --locale= --auth=trust
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C".
initdb: error: "--locale=" is not a valid server encoding name
[oneshot] postgres-init:  failed to initialize the new database
s6-rc: warning: unable to start service postgres-init: command exited 1
[oneshot] certs: finished adding webserver certificate to the certificate store
s6-rc: info: service certs successfully started
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more
 information.

@Janr In pulpcore 3.85 we decided to upgrade the postgresql version from 13 to 16. Since it took us a little while to get all the plugin 3.85 releases out it seems that the later 3.84 images picked up the database upgrade code. This upgrade is a one time process and it involves downloading the old 13 binaries to perform the upgrade. We didn’t want to ship the 13 & 16 binaries for all future images, so it is only downloaded the first time you upgrade, but looks like we forgot to think about the scenarios where Pulp is behind a proxy and can’t perform the download. In your case we can still manually upgrade the database.

On a pulp/pulp:3.86 image, shell into the image and perform the following commands:

# if you haven't set your proxy env vars, do so now
dnf -y install postgresql13 postgresql13-server postgresql13-contrib
/etc/init/postgres-init

After this you should see in your pgsql/data folder the file PG_VERSION should be 16. You should also have a new folder called pgsql/data_old where the old version 13 files are if you need to downgrade to a pulpcore version <3.84. Now you should be able to start up 3.86 image again. Let me know if you run into anymore problems.

2 Likes

Hello gerrod.

Thanks so much, this worked 100%.
3.86 started.

[root@pulp_mirror /]# /etc/init/postgres-init
[oneshot] postgres-init:  Postgresql database exists but will be upgraded from 13 to 16
.
.
.

Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, gem, maven, npm, ostree, python, rpm, sessions
Running migrations:
  Applying ansible.0065_collectionversion_ansible_col_namespa_ea6f75_idx... OK
  Applying core.0136_delete_basedistribution... OK
  Applying core.0137_appstatus... OK
  Applying core.0138_vulnerabilityreport... OK
  Applying core.0139_task_app_lock... OK
  Applying deb.0031_add_domains... OK
  Applying python.0015_alter_pythonpackagecontent_options... OK
Access policy for content/python/packages updated.

.
.
.

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade.
Once you start the new server, consider running:
    /usr/pgsql-16/bin/vacuumdb --all --analyze-in-stages

Thanks a mil!

3 Likes

gerrod

TypeError: unhashable type: 'list'

We are getting this error with 3.86.
Only in the Distribution task, there are ±32 repos.

Some Distributions work, many fails.

We have a CI/CD with 5 hosts in a Matrix.

Only 3.86 gives this error

This is for information only.
We will wait for a later 3.86…

TASK [Create rpm Distribution of Repositories] *********************************
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES12-SP5', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5/', 'base': 'MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5'})
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP7', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP7/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP7/', 'base': 'MR/SLES15/SP7/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP7'})
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP6', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6/', 'base': 'MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6'})
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP5', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP5/', 'base': 'MR/SLES15/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP5'})
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP4', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4/', 'base': 'MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4'})
ok: [localhost] => (item={'name': 'active-P-2025Q03.00-ABC-SLES12-SP4', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES12/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP4/', 'base': 'MR/SLES12/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP4'})
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: unhashable type: 'list'
failed: [localhost] (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP4-latest', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4-latest/', 'base': 'MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4-latest'}) => changed=false 
  ansible_index_var: d_idx
  ansible_loop_var: item
  d_idx: 6
  item:
    backend: https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4-latest/
    base: MR/SLES15/SP4/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP4-latest
    name: active-P-2025Q03.00-ABC-SLES15-SP4-latest
  msg: 'unhashable type: ''list'''
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: unhashable type: 'list'
failed: [localhost] (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP6-latest', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6-latest/', 'base': 'MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6-latest'}) => changed=false 
  ansible_index_var: d_idx
  ansible_loop_var: item
  d_idx: 7
  item:
    backend: https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6-latest/
    base: MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6-latest
    name: active-P-2025Q03.00-ABC-SLES15-SP6-latest
  msg: 'unhashable type: ''list'''
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: unhashable type: 'list'
failed: [localhost] (item={'name': 'active-P-2025Q03.00-ABC-SLES15-SP5-latest', 'backend': 'https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP5-latest/', 'base': 'MR/SLES15/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP5-latest'}) => changed=false 
  ansible_index_var: d_idx
  ansible_loop_var: item
  d_idx: 8

Thanks again.

This looks like an Ansible playbook run. But then again not very familiar out put. Can you find a post the very distribution task from the playbook that fails here? (Obviously, please strip sensitive info.)

Thank you.

Just a repeat, four out of the five servers are fine from the same Playbook.
With the same pulp image configuration, save the one with the 3.86 image.

Working builds: pulp/pulp 3.83 153feacb8645
pulp.squeezer:0.1.0

    - name: Run-2 Create rpm Publication, 1st run do not create packages.
      pulp.squeezer.rpm_publication:
        pulp_url: '{{ PULP_MIRROR_SVR }}'
        username: admin
        password: '{{ PASSWD }}'
        repository: "{{ item.name }}"
        validate_certs: false
        state: present
        timeout: 7200
      register: pub_result_2
      loop: "{{ repo_data.Repository }}"
      loop_control:
         index_var: p_idx_2


    - name: Create rpm Distribution of Repositories
      pulp.squeezer.rpm_distribution:
        pulp_url: '{{ PULP_MIRROR_SVR }}'
        username: admin
        password: '{{ PASSWD }}'
        name: "{{ item.name }}"
        base_path: "{{ item.base }}"
        publication: "{{ pub_result_2.results[d_idx].publication.pulp_href }}"
        validate_certs: false
        state: present
        timeout: 7200
      register: dist_result
      loop: "{{ repo_data.Repository }}"
      loop_control:
         index_var: d_idx

Repodata.yml as in the loop - “repo_data.Repository”

Repository:

 - name: active-P-2025Q03.00-ABC-SLES12-SP5
   backend: https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5/
   base: MR/SLES12/SP5/2025Q03.00/active-P-2025Q03.00-ABC-SLES12-SP5

 - name: active-P-2025Q03.00-ABC-SLES15-SP7
   backend: https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP7/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP7/
   base: MR/SLES15/SP7/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP7

 - name: active-P-2025Q03.00-ABC-SLES15-SP6
   backend: https://ltpulpcli01.abcgroup.net:30082/pulp/content/MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6/
   base: MR/SLES15/SP6/2025Q03.00/active-P-2025Q03.00-ABC-SLES15-SP6