I believe we need to reduce the amount of time it takes to merge a PR, especially a pulpcore PR. It has typical run times at around 40 minutes. There are a variety of ways to reduce the runtime here, but for this discussion I propose we run fewer tests at PR merge time and all tests at nightly.
For pulpcore specifically, I’d like to identify a representory sample of the tests, e.g. 30% and have these run at PR time. Then in the nightlies, let’s run 100% of the tests. To do this each test must have already been converted to a
pytest style test and then it needs the
pytest.mark.nightly mark see this example. The CI runs already use that mark all we have to do is apply the marks.
We may find the occasional revert, but I think it’s a small price for the ability to merge, e.g. 4x work in the same CI time. We need to look after our nightlies, but we need to do that anyway. I’ve started focusing on getting the various nightlies passing, e.g. pulpcore, pulp_file, pulp_ansible, etc.
What do others thing of this idea? Can we try this idea on pulpcore and pulp_file specifically?