A new Tale - Pulp gets queueless distributed task workers | software repository management

Pulp is about to introduce a new replacement for its tasking system. This complete redesign comes with a much simpler architecture that will drop the need for the resource-manager, the last singleton in the Pulp 3 infrastructure. Additionally, the distributed nature of the new task worker type drops the use of Redis Queue (rq). With this simplification there is no longer a need to replicate task data accross two databases. The main Postgres Pulp database will therefore be the single source of truth about tasks. Last but not least, the new worker type will use Postgres advisory locks to ensure exclusive access to tasks. This is a much more lightweight approach than storing access locks in a separate table. Also it simplifies the cleanup process vastly if a worker dies as postgres will clean the locks with its connection.


This is a companion discussion topic for the original entry at https://pulpproject.org/2021/06/21/a-new-tale/