Pulp_rpm filling up disk with tempfiles

Support
#1

Hi all,

there’s something behaving really odd in my pulp setup.
I’m running pulpcore 3.71.2 with pulp_rpm 3.28.0 and four workers to provide a bunch of mirror repositories to my clients.
The mirrors are configured, for example, as follows:

remotes:

{
  "name": "remote_ol9_UEK",
  "url": "https://yum.oracle.com/repo/OracleLinux/OL9/UEKR8/x86_64/",
  "proxy_url": null,
  "pulp_labels": {},
  "download_concurrency": null,
  "max_retries": null,
  "policy": "on_demand",
  "total_timeout": null,
  "connect_timeout": null,
  "sock_connect_timeout": null,
  "sock_read_timeout": null,
  "headers": null,
  "rate_limit": null,
etc.
}

repo:

{
  "name": "repo_ol9_UEK",
  "description": null,
  "retain_repo_versions": 1,
  "remote": "/pulp/api/v3/remotes/rpm/rpm/0196382e-8303-7bde-ae6b-f47334029ffa/",
  "autopublish": false,
  "metadata_signing_service": null,
  "package_signing_service": null,
  "package_signing_fingerprint": "",
  "retain_package_versions": 0,
  "checksum_type": null,
  "metadata_checksum_type": null,
  "package_checksum_type": null,
  "gpgcheck": null,
  "repo_gpgcheck": null,
  "sqlite_metadata": false,
  "repo_config": {},
  "compression_type": null,
 etc.
}

And I’m using “mirror_complete” to sync the repos every night.

Most of the time the repositories contain no artifacts until a client requests an rpm file. Then, pulp fetches this rpm from the remote repository into a temporary file, and once completed it makes it an artifact and provides it to the client, if I understood this correctly.

Now, when multiple clients request the same rpm file at virtually the same time I see several (non-identical) temporary copies of the same source rpm filling up the disk:

-rw-------  1 pulp pulp    606208 May 13 10:39 tmp05lznndl-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp   7508364 May 13 10:40 tmp073su03q-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  79421440 May 13 10:41 tmp0fmno6q8-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp 116867072 May 13 10:41 tmp0k4fvmch-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp  77651968 May 13 10:41 tmp0nkqxr4a-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm                                                                                                               
-rw-------  1 pulp pulp  88244224 May 13 10:41 tmp0urb3vxu-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp  22094770 May 13 10:38 tmp0v6mcnfe-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  23296178 May 13 10:37 tmp1575erjt-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm                                                                                                               
-rw-------  1 pulp pulp  19300352 May 13 10:38 tmp1pnvohem-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm                                                                                                               
-rw-------  1 pulp pulp 126320640 May 13 10:41 tmp1pzsmlxq-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp  95702858 May 13 10:37 tmp1sixug4s-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm                                                                                                               
-rw-------  1 pulp pulp  73908224 May 13 10:41 tmp1xzovrgk-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp    333290 May 13 10:40 tmp1y83cg88-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm                                                                                                               
-rw-------  1 pulp pulp 116146176 May 13 10:41 tmp2a5c4vxn-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp  13627392 May 13 10:41 tmp2c8iury5-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  65372160 May 13 10:41 tmp2ptvrvw0-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp   3481012 May 13 10:39 tmp2q5e55uu-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm                                                                                                               
-rw-------  1 pulp pulp 152342528 May 13 10:41 tmp2s3pkuba-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp 124264448 May 13 10:41 tmp_2zqb6jl-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  39279282 May 13 10:36 tmp3817zdkr-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  36689418 May 13 10:36 tmp3apu2vq0-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm                                                                                                               
-rw-------  1 pulp pulp  14014932 May 13 10:38 tmp3c3tkifb-kernel-uek-core-6.12.0-1.23.3.1.el9uek.x86_64.rpm                                                                                                                       
-rw-------  1 pulp pulp   8486912 May 13 10:41 tmp3jh2m69m-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  81231872 May 13 10:38 tmp3nksg979-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp 124715008 May 13 10:41 tmp3nu9iq1s-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  27869184 May 13 10:39 tmp3vhqq5pa-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  79560704 May 13 10:41 tmp3vid85rg-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp 179490816 May 13 10:41 tmp3y5q5rb9-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp     91256 May 13 10:36 tmp3zx1sy5f-linux-firmware-core-20250423-999.40.git32f3227b.el9.noarch.rpm                                                                                                          
-rw-------  1 pulp pulp  90734592 May 13 10:41 tmp__41z6jh-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  64487424 May 13 10:41 tmp4596ygph-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp  26134434 May 13 10:37 tmp4a1m4agk-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp   1921466 May 13 10:38 tmp4ohd_f1z-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  81723392 May 13 10:41 tmp4rnw2tg8-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp  70631424 May 13 10:41 tmp4vguzebp-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp   7707892 May 13 10:41 tmp51ps7rfr-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm                                                                                                               
-rw-------  1 pulp pulp  34323418 May 13 10:39 tmp55hgm315-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp     98304 May 13 10:39 tmp59aw4qgf-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  52379648 May 13 10:36 tmp5excvepa-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp   9781248 May 13 10:41 tmp5ni1b7f5-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  77045760 May 13 10:41 tmp5tz9w8mk-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm                                                                                                               
-rw-------  1 pulp pulp  20176786 May 13 10:38 tmp5xd4bhwq-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm                                                                                                               
-rw-------  1 pulp pulp  95862784 May 13 10:41 tmp60q78g98-mdatp-101.25032.0008-1.x86_64.rpm                                                                                                                                       
-rw-------  1 pulp pulp  18987954 May 13 10:38 tmp668nh0ox-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  18209938 May 13 10:38 tmp67u457ii-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  76800000 May 13 10:41 tmp6e0p30j5-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  23068672 May 13 10:38 tmp6gsins15-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp     98304 May 13 10:39 tmp71lxqael-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  84770816 May 13 10:41 tmp72sc4vfc-mdatp-101.25032.0008-1.x86_64.rpm
-rw-------  1 pulp pulp     76754 May 13 10:36 tmp75mggp8b-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp  13705216 May 13 10:41 tmp77bel35d-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm
-rw-------  1 pulp pulp   3774388 May 13 10:40 tmp7971e1zu-linux-firmware-20250423-999.40.git32f3227b.el8.noarch.rpm

Where does that come from and how to prevent this?

Looking forward to your comments :slight_smile:

#2

One of on_demand’s requirements is, “start streaming content to the client immediately, and clean up later”. When several/many clients ask for the same Content that doesn’t yet have an Artifact registered, at the same time, all of them get the binary streamed to them as separate items - because Pulp doesn’t “know” it’s The Same Thing until we have the whole thing and can sha256 it. That’s the duplication you’re seeing. The items are “non-identical” because the individual streams are at different points in the download process. This is especially noticeable when we’re talking about Large Binaries (like firmware) over not-so-fast links.

As soon as one stream has gotten/returned all the bits, it moves them to the /artifacts/ directory and creates the Artifact in the DB. Future access will then just stream the existing binary. All the “competing” streams will complete and, as part of cleanup notice the Artifact now exists and finish up/clean up their tempfiles.

There’s no “fix” for this - it’s a feature :slight_smile: Immediate-mode syncs don’t exhibit the behavior, but take up All The Disk. You could, after a sync, pre-fetch a list of RPMs to “seed” the /artifacts/ directory before all your client systems call home - kind of ad-hoc.

1 Like
#3

Thanks @ggainey for the insight. I guess I’ll add some delay to my daily host updates and figure out some mechanism to prefetch the newest packages, as you suggested :+1:

1 Like