Workflow with RPM modules and immutability

I was interested in the workflow described this issue about handling modular RPMs (in AlmaLinux), so I’ve decided to move its discussion here.

TLDR, the user would like to mutate Modulemds to make his workflow better. I know that our content is immutable by design, so I want to understand what is the best practice in such a case, or if we could do something to improve the usage at all.

Here is a description of the workflow from the author:

We do many things around modules, but I’ll try to focus on one scenario, creating a module after builds are finished.
This is how a modular build looks in ALBS

As you know, modules are made up of different packages, and these packages might be coming from different sources (think of them as different src.rpms), which turns this build into many tasks.

When tasks are finished, they get processed, and among many other things, we add the modular info into pulp. As a workaround for the current workflow in pulp around modules, as completed tasks are processed, we create “fake” versions of the module until all tasks are finished. When all tasks are finished, we create the final module in pulp with real NSVCA. In an ideal world, all tasks will finish okay, but the reality is that we might need to re-build failed tasks. Or we can even, complete the failed tasks in another build, that will only build a subset of the entire module. And as you may imagine, in this kind of situations, having the “final” version of the module already in pulp is a problem for us that we need to workaround.

Same applies if we make a partial update of a module, because maybe, only one of the modular components has an update, and in a big module, sometimes it just doesn’t make sense to rebuild all the module all over again every time there’s an update in one ref (consider a big module like this one javapackages-tools/modulemd.src.txt at c8-stream-201902 - javapackages-tools - AlmaLinux OS Git server)
(…)