Syncing a non-standard repository

Support
#1

Problem:
Hello everyone I ran into a problem when using pulp3. I need to synchronize the ALT_LINUX repository. Index of /altlinux/p11/branch/x86_64/. I am trying to synchronize as a file repository, since neither rpm works due to the non-standard repository structure. However, when I try to synchronize the repository as a file, I get an error indicating the absence of PULP_MANIFEST. The question is, can I somehow make an exact copy of the files from the link above using pulp3?
Expected outcome:
Sync complete
Pulpcore version:
core 3.53.0
Pulp plugins installed and their versions:

Operating system - distribution and version:
AlmaLinux - 8.9
Pulp is installed via podman
Other relevant data:

#2

If order to sync into file repository the remote source needs to have a PULP_MANIFEST otherwise it won’t be recognized by Pulp About File Repositories - Pulp Project

#3

@ipanova Do you know if it was ever considered to add a pulp_file sync feature that allows you to simply “sync everything in a folder” and have pulp_file create the manifest?

I guess it could be kind of dangerous since there is no way of telling ahead of time what you will get. Also handling symlinks might be an issue. Also, then there is no checksum to check downloaded files against.

I guess there are lots of reasons not to have such a “feature”.

1 Like
#4

that is, do I understand correctly that there is no way to make a copy of everything on the mirror using pulp3?

#5

No simple way that does not involve jumping through hoops.

For example, you could do something like mirror the repo somewhere yourself, where you can then add the PULP_MANIFEST yourself (there is a script to generate a PULP_MANIFEST for a local folder) and then sync that to pulp_file.

If what you have is an RPM repo that won’t sync to pulp_rpm for some reason, you are probably better off to open an issue with pulp_rpm

#6

Understood, thank you very much!

#7

You sort of answered it yourself. Mainly think of how pulp would be able to discover the remote source if it has no guarantee of any format or structure. pulp-manifest for us is basically metadata that allows us to ‘parse’ what we plan to pull into pulp ( filename, sha, size)

#8

This is not a standard yum repo that has defined repometadata. It might need a new type and special handling.

#9

that’s right, it’s like an rpm, but it doesn’t have a standard structure.

#10

Does dnf successfully install from it? Or is it built for some other package manager? Or more generally: How is it used?

#11

There is a DNF for this repository, it works the same way as in Alma, Centos. This is a separate OS, called AltLinux