Pulp ocp operator - pods stuck in database migration after restart/patch

#1

Hi Team ,

I have an issue with pulp operator in ocb. Issue is observed when ever cluster is patched .
Aspart of patching , nodes are rebooted and then pods are stuck in database migration step.

Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…

Issue can be wrokarounded by running migration manully from any pod or by removing init pods .
Is there a way to fix this issue with out unmanaging operator ?

#2

This is a known issue related to how we build our images: Workers stuck due to nonexistent migrations · Issue #1228 · pulp/pulp-operator · GitHub
There is no need to put the operator in unmanaged state to manually run the migrations: Common issues - Pulp Project

1 Like
#3

Ok , thanks for the information onthe issue. It failed over weekend and one of my colleague removed init contianer by unmanaging the cluster.
image pull policy is set to always in our setup
Apart from manual migration do we have any other workaround for now ?

#4

Hum… for now, unfortunately no.
Identifying when to automatically trigger the migration or finding a way to avoid this scenario is somewhat tricky. Here is another issue impacted by the need for manual intervention: Create a manual database migration trigger · Issue #1295 · pulp/pulp-operator · GitHub
Maybe it would be a matter of modifying our image initialization, but I have not had time to do a thorough investigation on this issue yet.