Monitoring Sync Tasks for Repos

Support
#1

Problem:
With pulp v2 we were monitoring pulp repo sync tasks to evaluate proper completion using specific api endpoint:
https://ip_address/pulp/api/v2/repositories/{repo_name}/history/sync?limit=1

We processed output to get relevant information, however in pulp v3 which we are migrating to, I have not been able to find a key that can link a task to a specific repo hence I cannot correlate them so monitoring is almost impossible since task href can be only related to repo at the moment of task execution, we are running sync tasks using cron and I think storing in a file task href everytime task is run is not feasible since we use lambda to monitor so we need a proper API method. Maybe I overlooked something please bear with me, but as of now I don’t have a clear view on how to implement this monitoring. Do you have any suggestions/ideas?

Off topic question, task list is automatically rotated by pulp? or do I need to schedule purge task?

Expected outcome:

Pulpcore version:
“core”: “3.48.0”,

Pulp plugins installed and their versions:
“versions”: {
“rpm”: “3.25.1”,
“core”: “3.48.0”,
“file”: “3.48.0”,
“certguard”: “3.48.0”
},

Operating system - distribution and version:
RHEL 9

Other relevant data:

#2

There isn’t a direct way, but the information is there in the task. A sync “knows” it needs to lock the repo being sync’d, so that repository shows up as a “reserved_resources_record”. You can get the task-state of the sync-tasks that involved a given-named-repository using the CLI something like so:

$ pulp task list \
  --name='pulp_rpm.app.tasks.synchronizing.synchronize'
  --reserved-resource \
    `pulp rpm repository show --name zoo | jq -r .pulp_href`  \
  --field "state"  --field "pulp_created" 
  --field "started_at" --field "finished_at"
[
  {
    "pulp_created": "2024-04-03T14:43:41.714772Z",
    "state": "completed",
    "started_at": "2024-04-03T14:43:41.758464Z",
    "finished_at": "2024-04-03T14:43:43.488597Z"
  }
]

(oci-env)
2 Likes
#3

Hi @ggainey , thanks a lot for your reply, that was the relation I was looking for, I did not know what reserved_resources_record meant in the task. New to pulp here.

#4

Glad I could help!

“reserved” resources are “entities the task needs/will impact as it gets its job done”. The tasking system runs tasks concurrently, the reserved-resources are used to make sure two tasks that need the same Thing a) won’t collide and b) will be executed in the order they were dispatched. Resources can be ‘shared’ or ‘exclusive’ - ‘shared’ means the requesting task can run in parallel with any other task that also only needs shared access to the entity, ‘exclusive’ means “nobody else asking for this resource can run till I’m done with it”.

Does that clarify some?

#5

Hi @ggainey yes it is crystal clear now, thank you!!, Regarding the other question, task list is automatically rotated by pulp? or do I need to schedule purge task?

#6

Oops, missed that - pulp doesn’t purge on its own, you’ll want to call pulp task purge on whatever schedule makes sense for you.

3 Likes