Temporary storage on S3

Support
#1

Problem:
Can we relocate the temporary storage during the sync to s3?

Expected outcome:

Pulpcore version:
3.21

Pulp plugins installed and their versions:
pulp-ansible 0.16.1
pulp-cli 0.16.0
pulp-deb 2.20.1
pulp-file 1.11.3
pulp-rpm 3.19.3
pulp-rpm-client 3.18.9
pulpcore 3.21.3
pulpcore-client 3.21.3
Operating system - distribution and version:
Rhel 8.6

Other relevant data:
We have limited amount the local space to save the tmp files during multiple syncs. Is it possible to use s3 for these tmp files?

#2

Hi @bli111,

Unfortunately, it is not possible to relocate temporary storage to use s3 or azure object storage.

You can submit a feature request here.

-Mike

#3

If you can/are willing to trade time for space, you can experiment with:

  • reducing the number of pulpcore-workers (so fewer/only-one task is writing to tmp at a time)
  • reducing download_concurrency on your Remotes (so fewer blobs are potentially downloaded at once)
    • note: not sure about this one, I can’t remember when tmp gets cleaned up
  • use “on_demand” instead of “immediate” mode, so you never have the binary blobs in tmp (until someone asks for them)
2 Likes
#4

Thanks. I have already configured pulpcore-workers to one. I will try reducing the download_concurrency and on_demand option.

1 Like
#5

You’re saying your default storage is local FS but you want only for temp file to use S3?
At the moment we don’t have such flexibility. Here is another issue that demonstrates opposite problem https://github.com/pulp/pulpcore/pull/3510, https://github.com/pulp/pulpcore/issues/3827