Pulp task-groups

Support
#1

Problem:
unable to find documentation and help to create and use task-groups. can you please point to existing documentation for the same? looking for using pulp-cli to create,list, and use task-groups

Expected outcome:
Looking for documentation on task-groups with RPM plugin

Thanks,

#2

Is this a user question?
In that case, sorry, you don’t get to “use” task groups. There are certain actions in Pulp, where a single task would not be sufficient, or multiple tasks help with paralellizing (like import). In those cases you get a task group instead of a task to wait on.
What do you want to accomplish? Maybe there’s another way?

1 Like
#3

Thanks, yes…
we are trying to synchronise multiple OS repositories and wanted to put them in sort of a loop and export the content out to the filesystem when done, and wanted to run all of this in parallel…
for example:
remote 1 → repo1 → auto_publish → distribute_1 → export rpm units to fs → export to s3 (for ex)
remote 2 → repo2 → auto_publish → distribute_2 → export rpm units to fs → export to s3 (for ex)

both these jobs perform the sync in the background, and the only to way to track the progress is to run
pulp task list --state running --feild pulp_href and wait in a loop,

if task_groups were available to end users to use, we could group tasks for a type of repo, say redhat 8 into its own group and then monitor the progress of the group until finished and then perform additional tasks…

again, this is just an idea what we wanted to try, but like you said there could be a more elegant solution

Thanks,

#4

It looks like your two chains of operations are completely independent. No one needs to wait on any step of the other. How about writing a shell function for each and execute it with “&” (assuming you are using some Burnshell).

function sync_and_publish (
  pulp sync ... $1 ... # Do not use `--background`
  pulp publish ... $1 ...
  pulp export ... $1 ...
)

sync_and_publish repo1 &
sync_and_publish repo2 &
fg %1
fg %2
#5

Have you considered using Ansible? There is a collection calles pulp/squeezer, but it may lack some features.

#6

Havent considered ansible to sync repos, our workflows include mostly bash scripts, i’ve tied the function approach that you suggested above, but had issues, cant remember what it was. Will try it again.

1 Like