I am trying to understand how to build releases for deb repos. I have been looking at the REST API docs but am still not able to grasp the workflow. I think my lack of knowledge of deb repos is probably not helping either.
Are there any docs or tests that cover such a workflow? Or could someone sketch out how the workflow with list of api endpoint calls or pulp-cli commands to run to build a release, add packages to it, etc?
You are pointing at a gaping hole in pulp_deb functionality. Currently there really isn’t any good way (not sure if there is any way) for users to create their own Releases and components, and add content (upload packages) to them.
Right now, “components”, “releases”, and “architectures”, can only really be created via sync. Even if that metadata content was created via sync, you cannot (easily) add additional manually uploaded packages to those repo structure elements. The current workaround for publishing manually uploaded packages is to use the option simple=True during publish, which basically invents an additional release named “default”, with a single component named “all” during publish, and simply adds all packages present in the repo version to that default/all release-component combination. This is currently the only way to include uploaded packages in a pulp_deb APT publication.
This is very far from ideal. We recently started a discussion with how Katello is currently handling package uploads for both pulp_rpm and pulp_deb (see here) because there are challenges even with the simpler case of pulp_rpm.
Personally I feel like the whole model for content upload across all of Pulp is not very user friendly, since it is essentially up to users to figure out what needs to become an artifact, and what needs to become a content unit, whether those artifacts and content units already exist in Pulp and how to add the final result to a repository. During sync, by comparison, all of that is up to the plugin author and the sync code, as it should be. As a user, I care about adding the file I have (for example some actual .deb package) to a particular Pulp repository. I should not have to worry about how Pulp turns that file into de-duplicated artifacts and content units. I believe this is true for all plugins, but it becomes really obvious in the case of pulp_deb, where packages are not just “present in the repo version or not present in the repo version”, but should also be associated with a particular release/distribution-component combination. It’s just not reasonable to expect users to understand pulp_deb’s internal plumbing on how those associations are represented in the DB.
Having described the Problem, I am going to pitch a possible solution that I have kicking around my brain (but which has not yet received a lot of discussion): What if there was a upload/ API endpoint, right next to the existing sync/ endpoint on repositories. This endpoint would provide a well defined interface for uploading a deb package to that repository, while also allowing the specification of a ReleaseComponent, that the package should be added to. It is then up to the upload code, to create artifacts and content or else retrieve existing artifacts and content as needed. If the user did not specify a ReleaseComponent, some default fallback ReleaseComponent could be used (perhaps uploaded/all). That way, uploaded packages would no longer depend on the (simple=True) publication workaround, and users would no longer need to worry about whether the things they are trying to add to some repo already exist in Pulp or not.
I am of course open to alternative ideas.
Thanks for the feedback. I definitely agree that managing content could be more user friendly especially when there are complex scenarios involving content that points to other content.
A couple questions. First, would it be possible currently for the user to construct the objects they need and then add them to the repository. It looks like it’s possible to create a Release, a ReleaseComponent, and a PackageReleaseComponent. What happens if you upload package and add all four objects (including the package) to a repo? I know this isn’t very user friendly but at least in our case, we’d be handling this programmatically.
Alternatively, since plugins can override the repo modify endpoint (see here) would it be possible to pulp_deb to introduce some extra fields to this endpoint? So like when you add a set of packages to a repository, you could specify the ReleaseComponent for example. I think the advantage of this solution is that it wouldn’t require changes in pulpcore.
In theory this should be possible. That being said, I have never tested it myself, and I have some vague memory of some user reporting problems with their attempt. => The only way to be sure, is to try it out. If you do try it, please record your commands/steps, I would be interested to look those over.
I think that would be a reasonable feature request, regardless of whether my half baked ideas on possible new API endpoints go anywhere or not. 
I think at this point it might be best to formulate an implementation agnostic issue in pulp_deb. Something like “Add workflows for adding packages to ReleaseComponents”.
FYI: There is an experimental “advanced copy API” feature in pulp_deb now, that provides support for copying packages between repositories, while automatically taking the associated Release, ReleaseComponent, and PackageReleaseComponent content along for the ride. Unfortunately, this feature isn’t fully ready/robust yet. It will let you copy multiple Releases with identical distribution field into a single repo version (which will fail to publish). However, the feature does provide an example for the kind of logic that is needed to implement what you are asking for.
That sounds like a fine idea, I imagine, that you would specify the release-component and modify would only add-remove packages to/from that component. (And maybe do some cleanup if packages are no longer in any component.)