Pulp as single container: s6rc-fdholder not starting

Support
#1

Hello,

on my newly build pulp as a single container s6rc-fdholder is not starting. I don’t see any logs referring to this service in container. I can also start it manually however it’s not auto-staring. Is it expected? I’m not very familiar with s6 how can I troubleshoot it?

# s6-rc -a list
s6rc-oneshot-runner
nginx
postgresql
pulpcore-api
pulpcore-content
pulpcore-worker@1
pulpcore-worker@2
redis
add-workers
certs
db-fields-key-create
fix-pulp-perms
postgres-init
postgres-prepare
# s6-rc -da list
s6rc-fdholder
# s6-rc start s6rc-fdholder
# s6-rc -da list
# s6-rc -a list
s6rc-oneshot-runner
s6rc-fdholder
nginx
postgresql
pulpcore-api
pulpcore-content
pulpcore-worker@1
pulpcore-worker@2
redis
add-workers
certs
db-fields-key-create
fix-pulp-perms
postgres-init
postgres-prepare

Logs from starting container:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
fix-attrs: info: applying /etc/fix-attrs.d/postgres
s6-chown: fatal: unable to chown /var/lib/pgsql/dump.sql
data: No such file or directory
fix-attrs: warning: fix-attrs is deprecated, please fix volume permissions in your container manager instead
fix-attrs: warning: some fix files failed to apply
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service postgres-init: starting
s6-rc: info: service fix-pulp-perms: starting
s6-rc: info: service db-fields-key-create: starting
s6-rc: info: service redis: starting
Calling /etc/init/postgres-init
Calling /etc/init/fix-pulp-perms
Calling /etc/init/db-fields-key-create
s6-rc: info: service redis successfully started
[oneshot] db-fields-key-create: creating database fields key
s6-rc: info: service db-fields-key-create successfully started
s6-rc: info: service certs: starting
Checking /var/lib/pulp/media permissions
Checking /var/lib/pulp/scripts permissions
Calling /etc/init/certs
Checking /var/lib/pulp/tmp permissions
[oneshot] certs: adding webserver certificate to the certificate store
s6-rc: info: service fix-pulp-perms successfully started
s6-rc: info: service postgres-init successfully started
s6-rc: info: service postgresql: starting
s6-rc: info: service postgresql successfully started
s6-rc: info: service postgres-prepare: starting
Calling /etc/init/postgres-prepare
2024-04-22 09:44:24.589 UTC [120] LOG:  redirecting log output to logging collector process
2024-04-22 09:44:24.589 UTC [120] HINT:  Future log output will appear in directory "log".
ALTER DATABASE
[oneshot] postgres-prepare: running Pulp migrations
[oneshot] certs: finished adding webserver certificate to the certificate store
s6-rc: info: service certs successfully started
error: Failed to initialize NSS library
Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, gem, maven, ostree, python, rpm, sessions
Running migrations:
  No migrations to apply.
s6-rc: info: service postgres-prepare successfully started
s6-rc: info: service add-workers: starting
s6-rc: info: service pulpcore-worker@1: starting
s6-rc: info: service pulpcore-content: starting
s6-rc: info: service pulpcore-api: starting
Calling /etc/init/add-workers
s6-rc: info: service pulpcore-worker@1 successfully started
Checking for database migrations
Checking for database migrations
s6-rc: info: service pulpcore-api successfully started
s6-rc: info: service pulpcore-content successfully started
Checking for database migrations
s6-rc: info: service nginx: starting
s6-rc: info: service nginx successfully started
Calling /etc/init/nginx
[oneshot] add-workers: Adding workers 2 through 2
[oneshot] add-workers: Workers will be added shortly after the boot finishes
s6-rc: info: service add-workers successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
Calling /etc/init/postgres-init
Calling /etc/init/db-fields-key-create
[oneshot] db-fields-key-create: creating database fields key
Calling /etc/init/fix-pulp-perms
Calling /etc/init/certs
Checking /var/lib/pulp/media permissions
Checking /var/lib/pulp/scripts permissions
Checking /var/lib/pulp/tmp permissions
2024-04-22 09:44:39.580 UTC [336] LOG:  redirecting log output to logging collector process
2024-04-22 09:44:39.580 UTC [336] HINT:  Future log output will appear in directory "log".
Calling /etc/init/postgres-prepare
error: Failed to initialize NSS library
ALTER DATABASE
[oneshot] postgres-prepare: running Pulp migrations
error: Failed to initialize NSS library
error: Failed to initialize NSS library
error: Failed to initialize NSS library
error: Failed to initialize NSS library
Database migrated!
ready
Database migrated!
ready
Database migrated!
ready
Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, gem, maven, ostree, python, rpm, sessions
Running migrations:
  No migrations to apply.
Calling /etc/init/add-workers
Already migrated!
Already migrated!
Already migrated!
Calling /etc/init/pulpcore-content
Calling /etc/init/pulpcore-worker
/usr/local/bin/pulpcore-content
Calling /etc/init/pulpcore-api
/usr/local/bin/pulpcore-api
[oneshot] add-workers: Adding workers 2 through 2
Calling /etc/init/nginx
[oneshot] add-workers: Workers will be added shortly after the boot finishes
Already migrated!
Calling /etc/init/pulpcore-worker
[2024-04-22 09:44:55 +0000] [405] [INFO] Starting gunicorn 21.2.0
[2024-04-22 09:44:55 +0000] [405] [INFO] Listening at: http://[::]:24817 (405)
[2024-04-22 09:44:55 +0000] [405] [INFO] Using worker: pulpcore.app.entrypoint.PulpApiWorker
[2024-04-22 09:44:55 +0000] [450] [INFO] Booting worker with pid: 450
[2024-04-22 09:44:55 +0000] [451] [INFO] Booting worker with pid: 451
error: Failed to initialize NSS library
error: Failed to initialize NSS library
error: Failed to initialize NSS library
error: Failed to initialize NSS library
error: Failed to initialize NSS library
error: Failed to initialize NSS library
pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
pulp [None]: pulpcore.tasking.worker:INFO: New worker '440@404baaf57d65' discovered
pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
pulp [None]: pulpcore.tasking.worker:INFO: New worker '399@404baaf57d65' discovered
[2024-04-22 09:45:10 +0000] [402] [INFO] Starting gunicorn 21.2.0
[2024-04-22 09:45:10 +0000] [402] [INFO] Listening at: http://[::]:24816 (402)
[2024-04-22 09:45:10 +0000] [402] [INFO] Using worker: aiohttp.GunicornWebWorker
[2024-04-22 09:45:10 +0000] [476] [INFO] Booting worker with pid: 476
[2024-04-22 09:45:10 +0000] [479] [INFO] Booting worker with pid: 479
#2

That log looks like everything is up. fdholder (as I understand it, anyway) is very “internal”. What problem are you seeing with your container? What happens if you ping the /status/ url?

This error is harmless - see the discussion here.

#3

Hi ggainey,

thanks for your reply. I don’t see any impact now, API works just fine. I wonder why this service don’t come up and might it have any impact on pulp?

#4

I don’t know that “we” do anything with fdholder - as noted, it is very internal to s6. Unless you see failures in Pulp or in the logs, I expect you’re fine.

1 Like