Repository sync is stuck in waiting state

Community
#1

Hi Team,
I am failing to sync the repository, help is much appreciated.

Please find the details:

pulp rpm repository sync --name centos-7-baseos1

Started background task /pulp/api/v3/tasks/018a1d87-c8dc-71fa-9c5f-610fbfa94dbf/

…

pulp task show --uuid 018a1d87-c8dc-71fa-9c5f-610fbfa94dbf
{
“pulp_href”: “/pulp/api/v3/tasks/018a1d87-c8dc-71fa-9c5f-610fbfa94dbf/”,
“pulp_created”: “2023-08-22T13:55:52.668864Z”,
“state”: “waiting”,
“name”: “pulp_rpm.app.tasks.synchronizing.synchronize”,
“logging_cid”: “4f902c05491345539231c5c7c9957e40”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: null,
“finished_at”: null,
“error”: null,
“worker”: null,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/rpm/rpm/018a1d87-a57e-7a36-ad5d-465dafe9385e/”,
“shared:/pulp/api/v3/remotes/rpm/rpm/018a1d87-5558-76f7-97e3-5053e59f0b4d/”
]
}

Thank you!

#2

Does pulp status show any workers online? Are there other tasks in running state?

#3

Hi,
I see online workers, and I see only waiting tasks for rpm sync but nothing is in a running state.

Please find the below output and let me know if you want me the change something.

“online_workers”: [
{
“pulp_href”: “/pulp/api/v3/workers/018a1d41-af10-71c6-9f4e-c422cf4a39cc/”,
“pulp_created”: “2023-08-22T12:39:18.545465Z”,
“name”: “2480@ip-xxx-xx-xx-xxx.ec2.internal”,
“last_heartbeat”: “2023-08-23T01:00:00.321516Z”,
“current_task”: null
}
],

Pulp Status output

pulp status

Notice: Cached api is outdated. Refreshing…
{
“versions”: [
{
“component”: “core”,
“version”: “3.31.0”,
“package”: “pulpcore”,
“domain_compatible”: true
},
{
“component”: “rpm”,
“version”: “3.22.3”,
“package”: “pulp-rpm”,
“domain_compatible”: true
},
{
“component”: “container”,
“version”: “2.16.0”,
“package”: “pulp-container”,
“domain_compatible”: false
},
{
“component”: “file”,
“version”: “1.14.4”,
“package”: “pulp-file”,
“domain_compatible”: true
}
],
“online_workers”: [
{
“pulp_href”: “/pulp/api/v3/workers/018a1d41-af10-71c6-9f4e-c422cf4a39cc/”,
“pulp_created”: “2023-08-22T12:39:18.545465Z”,
“name”: “2480@ip-xxx-xx-xx-xxx.ec2.internal”,
“last_heartbeat”: “2023-08-23T01:00:00.321516Z”,
“current_task”: null
}
],
“online_content_apps”: [
{
“name”: “2579@ip-xxx-xx-xx-xxx.ec2.internal”,
“last_heartbeat”: “2023-08-23T01:00:02.139764Z”
},
{
“name”: “2572@ip-xxx-xx-xx-xxx.ec2.internal”,
“last_heartbeat”: “2023-08-23T01:00:02.141253Z”
}
],
“database_connection”: {
“connected”: true
},
“redis_connection”: {
“connected”: false
},
“storage”: {
“total”: 34280026112,
“used”: 3929198592,
“free”: 30350827520
},
“content_settings”: {
“content_origin”: “http://localhost:24816”,
“content_path_prefix”: “/pulp/content/”
},
“domain_enabled”: false
}

Lists of tasks are in waiting state

{
“pulp_href”: “/pulp/api/v3/tasks/018a1e3d-a7bb-76e8-a1af-3507a9fbb819/”,
“pulp_created”: “2023-08-22T17:14:31.740184Z”,
“state”: “waiting”,
“name”: “pulp_rpm.app.tasks.synchronizing.synchronize”,
“logging_cid”: “9450e27688c948e89e272283e661cf56”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: null,
“finished_at”: null,
“error”: null,
“worker”: null,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/rpm/rpm/018a1d87-a57e-7a36-ad5d-465dafe9385e/”,
“shared:/pulp/api/v3/remotes/rpm/rpm/018a1d87-5558-76f7-97e3-5053e59f0b4d/”
]
},
{
“pulp_href”: “/pulp/api/v3/tasks/018a1da1-f5b0-78cc-8813-5816add60524/”,
“pulp_created”: “2023-08-22T14:24:28.081528Z”,
“state”: “waiting”,
“name”: “pulp_file.app.tasks.synchronizing.synchronize”,
“logging_cid”: “0954138eeb6d41d7a93a08ae00a32f5a”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: null,
“finished_at”: null,
“error”: null,
“worker”: null,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/file/file/018a0922-9bc9-7eed-8b6c-dec99708b37b/”,
“shared:/pulp/api/v3/remotes/file/file/018a0926-54cc-7eb3-9758-3f3170a209a9/”
]
},
{
“pulp_href”: “/pulp/api/v3/tasks/018a1d91-0db0-7418-b1b6-d1688d9a79b0/”,
“pulp_created”: “2023-08-22T14:06:00.113645Z”,
“state”: “waiting”,
“name”: “pulpcore.app.tasks.base.general_create”,
“logging_cid”: “9832aa59d4d44e14ab1e559aa41d1610”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: null,
“finished_at”: null,
“error”: null,
“worker”: null,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [],
“created_resources”: [],
“reserved_resources_record”: []
},
{
“pulp_href”: “/pulp/api/v3/tasks/018a1d87-c8dc-71fa-9c5f-610fbfa94dbf/”,
“pulp_created”: “2023-08-22T13:55:52.668864Z”,
“state”: “waiting”,
“name”: “pulp_rpm.app.tasks.synchronizing.synchronize”,
“logging_cid”: “4f902c05491345539231c5c7c9957e40”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: null,
“finished_at”: null,
“error”: null,
“worker”: null,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/rpm/rpm/018a1d87-a57e-7a36-ad5d-465dafe9385e/”,
“shared:/pulp/api/v3/remotes/rpm/rpm/018a1d87-5558-76f7-97e3-5053e59f0b4d/”
]
},

FYI: remote details
#pulp rpm remote list
[
{
“pulp_href”: “/pulp/api/v3/remotes/rpm/rpm/018a1d87-5558-76f7-97e3-5053e59f0b4d/”,
“pulp_created”: “2023-08-22T13:55:23.096727Z”,
“name”: “centos-7-baseos1”,
“url”: “CentOS Mirror”,
“ca_cert”: null,
“client_cert”: null,
“tls_validation”: false,
“proxy_url”: null,
“pulp_labels”: {},
“pulp_last_updated”: “2023-08-22T13:55:23.096750Z”,
“download_concurrency”: null,
“max_retries”: null,
“policy”: “immediate”,
“total_timeout”: null,
“connect_timeout”: null,
“sock_connect_timeout”: null,
“sock_read_timeout”: null,
“headers”: null,
“rate_limit”: null,
“hidden_fields”: [
{
“name”: “client_key”,
“is_set”: false
},
{
“name”: “proxy_username”,
“is_set”: false
},
{
“name”: “proxy_password”,
“is_set”: false
},
{
“name”: “username”,
“is_set”: false
},
{
“name”: “password”,
“is_set”: false
}
],
“sles_auth_token”: null
}
]

Appreciate your help!

#4

Hi,
I have destroyed all the tasks, I tried re-sync and observed the worker as null. please let me know if any thoughts.

I see workers running
[root@ip ~]# ps -aux | grep pulpcore-worker
pulp 16567 0.0 1.1 345568 94088 ? Ss 04:35 0:04 /usr/local/lib/pulp/bin/python3 /usr/local/lib/pulp/bin/pulpcore-worker
pulp 16571 0.0 1.1 345584 93840 ? Ss 04:35 0:04 /usr/local/lib/pulp/bin/python3 /usr/local/lib/pulp/bin/pulpcore-worker

[root@ip ~]# pulp task list --state waiting
[
{
“pulp_href”: “/pulp/api/v3/tasks/018a20e8-9929-73ba-aec7-5dd30b72918d/”,
“pulp_created”: “2023-08-23T05:40:29.097953Z”,
“state”: “waiting”,
“name”: “pulp_rpm.app.tasks.synchronizing.synchronize”,
“logging_cid”: “40d0a7d739e743f5a9429d87ea13c3d3”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: null,
“finished_at”: null,
“error”: null,
“worker”: null,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/rpm/rpm/018a20e8-5e81-7c88-9a1f-556ddc879414/”,
“shared:/pulp/api/v3/remotes/rpm/rpm/018a20e6-ee6d-7405-a686-acb60fd3b22a/”
]
},
{
“pulp_href”: “/pulp/api/v3/tasks/018a20db-f432-77b3-a01c-c0218f0024cf/”,
“pulp_created”: “2023-08-23T05:26:40.435666Z”,
“state”: “waiting”,
“name”: “pulp_rpm.app.tasks.synchronizing.synchronize”,
“logging_cid”: “7ed9f8f6e4cc4b8094c5b74cd629331c”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: null,
“finished_at”: null,
“error”: null,
“worker”: null,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/rpm/rpm/018a1d87-a57e-7a36-ad5d-465dafe9385e/”,
“shared:/pulp/api/v3/remotes/rpm/rpm/018a1d87-5558-76f7-97e3-5053e59f0b4d/”
]
}
]

Thank you!

#5

Hi @dkliban

I see below an incompatible version message in log.

Aug 23 10:26:37 ip-xxx-xx-xx-xxx pulpcore-worker[16472]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: New worker ‘16472@localhost.localdomain’ discovered

Aug 23 10:26:37 ip-xxx-xx-xx-xxx pulpcore-worker[16472]: pulp [None]: pulpcore.tasking.pulpcore_worker:DEBUG: Worker heartbeat from ‘16472@localhost.localdomain’ at time 2023-08-23 10:26:37.515670+00:00

Aug 23 10:26:37 ip-xxx-xx-xx-xxx pulpcore-worker[16472]: pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: Incompatible versions to execute task 018a21ea-8181-7dba-8386-172b9918af11 by worker 16472@localhost.localdomain: task: rpm>=3.22.3 worker: None

Thank you!

#6

This sounds like you upgraded an existing system without restarting the pulpcore-workers?

1 Like
#7

Hi @ggainey,
I am not sure about it, please help us to fix the issue.
Let us know if you need any further details.
Thank you!

#8

Here are some tips to try

  • Restart the workers and verify that you no longer see the Incompatible versions to execute log line

  • Is your installation on one machine or multiple? If it’s multiple, it’s possible one environment got upgraded and the other didn’t. If that’s the case restarting the workers won’t help, you’ll also need to have all envronments upgraded to the same version

  • It’s unlikely, but maybe it’s possible that the pulp-api process is the one not up to date enough. Make sure that is also upgraded to the same version as your workers.

  • Also unlikely, but it’s possible everything is upgraded correctly and restarted correctly, but the task in the queue is incompatible with the worker that would pick it up. Cancel all of the outstanding Pulp tasks and ensure when you list running and waiting tasks there are none, then dispatch the sync again. This would be the thing to do after you are very sure all of your pulp processes are upgraded to the latest version.

#9

Hi @ggainey,

Restarting of Workers working fine now. I am not sure what was the issue because earlier I rebooted the instance multiple times but that did not help. I will perform multiple tests before going for production and keep you posted in case of any issues.

Appreciate your help @ggainey @dkliban.

2 Likes
#10

Hey @gouthamtrip!

I’m glad you got things working! I def can’t explain how “still had old workers running” happened, but at least final-restart got you past it!

2 Likes