Pulp stuck with syncing with workder node showing Incompatible versions to execute task

midhuhk · July 25, 2024, 9:09am

Hi Team ,

ASK [Gathering Facts] *********************************************************
ok: [localhost]
TASK [Include role] ************************************************************
TASK [2vlx-midhul-test-template : set_fact] ************************************
ok: [localhost]
TASK [2vlx-midhul-test-template : Create test-“baseos-rhel8” respoistory] ******
changed: [localhost]
TASK [2vlx-midhul-test-template : Create base-“baseos-rhel8” repository] *******
changed: [localhost]
TASK [2vlx-midhul-test-template : Sync remote into local repo] *****************

Worker node logs says:
pulp [None]: pulpcore.tasking.worker:INFO: Incompatible versions to execute task 0190538f-f543-7473-8150-95bace1675f6 by worker 1@pulp-server-nonprod-worker-5cfcc464f4-7lflg: task: rpm>=3.25.3 worker: 3.25.2
pulp [None]: pulpcore.tasking.worker:INFO: Incompatible versions to execute task 0190538f-f543-7473-8150-95bace1675f6 by worker 1@pulp-server-nonprod-worker-5cfcc464f4-7lflg: task: rpm>=3.25.3 worker: 3.25.2
pulp [None]: pulpcore.tasking.worker:INFO: Incompatible versions to execute task 0190538f-f543-7473-8150-95bace1675f6 by worker 1@pulp-server-nonprod-worker-5cfcc464f4-7lflg: task: rpm>=3.25.3 worker: 3.25.2

Restarted the worker bide , but no luck.

Any suggestions ?

x9c4 · June 26, 2024, 8:32am

Well, pulp is not stuck. This task is. But anyway, you seem to have an api process running that is newer than all your workers. Just bring your workers up to at least pulp_rpm version 3.25.3 and that task will be served.
If you decided to roll back an upgrade, than your only viable option is to cancel all tasks the old worker code will never dare to touch.

midhuhk · June 26, 2024, 10:19am

How do you suggest to update worker when installed via operator ?
image is being pulled from quay.io/pulp/pulp-minimal:stable

x9c4 · June 26, 2024, 10:27am

Well, as soon as all the pods run from the same container image, they should have the same versions installed.
Can you run a pulp status check and compare for any version inconsistencies?

midhuhk · June 28, 2024, 5:44am

Version issue seems fixed with latest image pull .
I am trying to sync rhel9 baseos repo from redhat and getting this error as below .

{
“changed”: false,
“exception”: "Traceback (most recent call last):\n\n File “/tmp/ansible_pulp.squeezer.rpm_sync_payload_nxs6qcxa/ansible_pulp.squeezer.rpm_sync_payload.zip/ansible_collections/pulp/squeezer/plugins/modules/rpm_sync.py”, line 109, in main\n\n File “/tmp/ansible_pulp.squeezer.rpm_sync_payload_nxs6qcxa/ansible_pulp.squeezer.rpm_sync_payload.zip/ansible_collections/pulp/squeezer/plugins/module_utils/pulp.py”, line 320, in process_sync\n sync_task = self.sync(remote.href, parameters)\n\n File “/tmp/ansible_pulp.squeezer.rpm_sync_payload_nxs6qcxa/ansible_pulp.squeezer.rpm_sync_payload.zip/ansible_collections/pulp/squeezer/plugins/module_utils/pulp.py”, line 281, in sync\n return PulpTask(self.module, {“pulp_href”: response[“task”]}).wait_for()\n\n File “/tmp/ansible_pulp.squeezer.rpm_sync_payload_nxs6qcxa/ansible_pulp.squeezer.rpm_sync_payload.zip/ansible_collections/pulp/squeezer/plugins/module_utils/pulp.py”, line 466, in wait_for\n self.entity[“state”], self.entity[“error”][“description”]\n\nKeyError: ‘description’\n",
** “msg”: “‘description’”,**

github.com

pulp/squeezer/blob/develop/plugins/module_utils/pulp.py#L466

    
      
          
          
    def wait_for(self, desired_state="completed"):
                  self.find()
                  while self.entity["state"] not in ["completed", "failed", "canceled"]:
                      sleep(2)
                      self.read()
                  if self.entity["state"] != desired_state:
                      if self.entity["state"] == "failed":
                          raise Exception(
                              "Task failed to complete. ({0}; {1})".format(
                                  self.entity["state"], self.entity["error"]["description"]
                              )
                          )
                      raise Exception("Task did not reach {0} state".format(desired_state))
                  return self.entity
          
          

          
class PulpUpload(PulpEntity):
              _href = "upload_href"
              _create_id = "uploads_create"
              _update_id = "uploads_update"

Nothing much in log file about the error. Any suggestions ?

x9c4 · June 28, 2024, 8:31am

Well this is for sure a different issue. Not all errors seem to have a “description”.
Can you change that to self.entity["error"].get("description")?

But again this is unrelated to the issue described in the original post.