Calling All Task Management Enthusiasts! Thoughts, views and issues of our Pulp instance

Development
#1

Yay folks! Hey, we’ve been running our instance for some time and we would like to share some info and ask for some collaboration in the discussion of some items.

Before anything, our Pulp instance runs on a Openshift/Kubernetes cluster, and it manages almost 20TB of artifact data, our DB uses 1.5 TB at least, and currently we run 24 pulp-workers, with an intent to scale it to 48 workers. I can’t say this is the biggest Pulp instance in the world, but we’re on the run for it. :sweat_smile:

After some heavy load tests and gathering some data, we had some impressions about the tasking system:

  1. Upper limit of how many workers we can start (assume completely idle system)
    1.1 Limited by the number of sessions opened to the database
    1.2 Limited by the number of heartbeats that can be written before workers start timing out.

  2. Upper limit of how concurrency of task dispatching / insertion
    2.1 Throughput is limited by this lock acquisition.

  3. Upper limit of tasks we process (assuming infinite workers)
    3.1 Limited by the architecture of the advisory lock concurrency

  4. Cost due to memory footprint
    4.1 Workers need a huge amount of memory. We set the base memory as 2GB, and the upper limit to 6GB.
    4.2 Things that need that upper limit are not garantee to run, it could be killed by the scheduler.

  5. Not clear which feature set for immediate task scheduling we are using now
    5.1 Which of our tasks are immediate ones? Where and how those immediate tasks run?

  6. Tasking logging is not exactly a standard
    6.1 Task logs doesn’t necessarily have a task id, so it’s hard to correlate log entries.
    6.2 It’s not clear what the tasking system started to run or completed using the logs.

Creating a Tasking working group to discuss those things would be great.

So, I want to hear from you:

  • Do you share some of those impressions?
  • Would you want to work on any of those points?
  • Do you have any insight about it?

PS: Feel free to ask for more data if it could support the discussion here. We’ll work to get it ASAP.

1 Like
#2

About immediate tasks:

Where they are used

The ones I know of in pulpcore are:

  • API: the update and remove async mixins, which are used in some viewsets (repository, distribution, acs and exporter)
  • content-app pull-trough (which caused the recent regression).

Also, Maven and Container plugins use it in specific endpoints.

Reference:

How are they used

This is defined in the dispatch call, which can receive immediate and deferred arguments.
These names are atually a bit confusing. They existed before we decided to do short-task optimization.

  1. The immediate task property really means short_task now
  2. The exeuction is immediate (on api/content worker) if task is “short”, the “defered” argument is not True and the resources are available at dispatch-time.
    • In this case, Task.worker=None (can use that fact to filter where task is executed)
  3. The execution is deferrd otherwise:
    • In this case, Task.worker!=None

This distinction is relevant, because now we do have some optimizations for short task running in a task worker. Also some plugins have special requirements, like this usage in Maven where the task should fail if it can’t be executed in the Api worker.

TLDR:
A task can be of kind ‘long’ or ‘short’ (actually marked in the task as ‘Task.immediate=True’).
If its short, it can have its execution ‘imediate/deferred’, depending on dispatch args and resoures.

Other considerations

Some immediate deletes are disabled because:
“Too many cascaded deletes to block the gunicorn worker.”

#3

Hey folks, thanks for sharing these problem statements.
Having such insights from a big production environment is quite valuable.

From the last meeting I think I understand problem (4) better and I had some thoughts about it.
What I heard is that admins running their instance needs more knobs to handle resource more efficiently, and that ideally we shouldnt try to make Pulp very smart about it (e.g, not make Pulp take decisions on what efficient resource management is).

Trying to put the problem a bit out of Pulp, my idea is introduce an optional feature of “Worker Groups” as an opt-in in the deployment/installation level. Given it is enabled:

  • All API, content and task workers must have a group assigned at deployment/installation level
  • Groups are arbitrary label strings
  • Dispatching a task will mark it with the group of the component that dispatched it
  • A worker only pick tasks that belong to its group

Given that capability, an admin could setup how many groups they need and specify routing rules to redirect any given request to the appropriate worker group. These rules could include url patterns, special headers or what not.

That approach kind puts all the complexity of handling the resources in the admins hands (and out of Pulp’s core), but I feel like that is what services is asking for. Does that make sense?

1 Like