Sync files in Pulp based on different download policies - immediate, on_demand, streamed

Support
#1

I want to sync different file types (e.g., manifests, tarballs, git repo, iso) in Pulp using download policies such as immediate, on_demand, or streamed.

I tried to set the policy field using a POST request to the endpoint: /pulp/api/v3/content/file/files/
However, I encountered an error stating ‘policy’: unexpected field

image
image3275×876 126 KB

On the API page also I couldn’t find any option to specify the download policy.

image
image2752×1655 184 KB

Additionally, I tried creating a pulp file remote with policy ‘on_demand’. The remote got created successfully:

image
image3235×1874 214 KB

but the sync operation failed with below error:

image
image2715×205 58.6 KB

Even the content upload doesn’t have any option for defining policy:

image
image2010×520 72.4 KB

Is there any other way I can sync files based on download policy?

#2

Pulp’s download-policy is a property of the Remote being used to sync-from, not at the individual-file-level. You can specify it when you create a FileRemote. We don’t support it on a per-content or per-content-type basis I’m afraid.

1 Like
#3

Yes, I have observed the same. I can create a pulp file remote using policy option but when I try to sync the remote, I get the below error:

’Error: Parsing of the manifest file failed on line:8.
Please make sure the remote URL is pointing to a valid manifest file.
The manifest file should be composed of lines in the following format: <relative_path>,,.'

As I understood, it is looking for a PULP_MANIFEST file with the same format as above, which my upstream url doesnt have. Since it is an external URL I am using to sync, it is not possible to add a PULP_MANIFEST to it.

Does pulp file plugin support syncing files which are not having a PULP_MANIFEST?
E.g - if I want to sync a manifest file: https://raw.githubusercontent.com/ROCm/k8s-device-plugin/r1.16/k8s-ds-amdgpu-dp.yaml

#4

What behavior do you want to see by trying to sync that file you linked? Is the entries in the yaml locations to other files that should also be synced into Pulp or do you just want to have Pulp store that yaml as an artifact in a file repository? If the latter, then you can do that easily by using the file_url option on file create.

pulp file content create --relative-path k8s-ds-amdgpu-dp.yaml --file-url https://raw.githubusercontent.com/ROCm/k8s-device-plugin/r1.16/k8s-ds-amdgpu-dp.yaml --repository foo

This will cause Pulp to go download the file and store it in the supplied repository.

If you wanted to do the former Pulp currently can’t do that. You would need to manually create a PULP_MANIFEST describing the location, size, & sha256 of every file you wanted to sync.

#5

This command helps to achieve what I want, thanks. One query however, as I see there are only the following options in pulp file content create command:

image
image1825×522 61 KB

Is there any way I can specify download policy in content create (on_demand, streamed) so that it doesnt download content immediately?

#6

No, the file-url option always downloads the content immediately.

1 Like
#7

Alright, thank you for the response. That helps!

2 Likes