Sync Task Queueing

Support
#1

Problem:
We are planning to include at least 30 repos for mirroring on a big EC2 instance with enough CPU to support several workers, we were trying to determine the best schedule for cron job to trigger the sync tasks, initially we thought of grouping repos with a certain pattern depending on their size so we will run in batches, let’s say, one group at 1am then next at 2am and so on, but we think that is inefficient since some tasks can finish within seconds when no updates are found, the question is, pulp is able to queue all the syncs if we trigger the tasks for example separated by 1 minute intervals in cron or some other pattern? in this case what I expect is if the server has 8 CPUs then 8 tasks will run and 22 will be in waiting state but at the end all of them will be run? If so what is the limit of this queue?
Thank you for your input!

Expected outcome:

Pulpcore version:
“core”: “3.48.0”,
Pulp plugins installed and their versions:
“versions”: {
“rpm”: “3.25.1”,
“core”: “3.48.0”,
“file”: “3.48.0”,
“certguard”: “3.48.0”
},
Operating system - distribution and version:
RHEL 9
Other relevant data:

#2

Not sure this is the answer you mean, but one worker performs one task at a time.
So with 8 CPUs you could start 8 workers.
The only thing is, it’s not limited to sync tasks.

#3

Hi @x9c4 , thanks for your reply, well what I would like to understand is if pulp can queue for example in my case 30 tasks if all workers are busy at any given point in time. I would like to optimize run time for the sync task so once one of them finishes next one is taken from queue, this way whole 30 repo sync will finish quicker and will use compute capacity in a more optimized manner.

#4

Yes, absolutely. There is no limit to the size of the task queue.
You could even start more workers later and they will start picking up the next available (and unblocked) task right away.

3 Likes
#5

Hi @x9c4 , I tested and scheduled 30 tasks separated by one minute in cron and pulp effectively queued several tasks in waiting state hence my resources are used in a much better way, all tasks were run successfully. Sorry for the delay in my reply but I wanted to test.

4 Likes