Marking majority of tests to only run nightly

I believe we need to reduce the amount of time it takes to merge a PR, especially a pulpcore PR. It has typical run times at around 40 minutes. There are a variety of ways to reduce the runtime here, but for this discussion I propose we run fewer tests at PR merge time and all tests at nightly.

For pulpcore specifically, I’d like to identify a representory sample of the tests, e.g. 30% and have these run at PR time. Then in the nightlies, let’s run 100% of the tests. To do this each test must have already been converted to a pytest style test and then it needs the pytest.mark.nightly mark see this example. The CI runs already use that mark all we have to do is apply the marks.

We may find the occasional revert, but I think it’s a small price for the ability to merge, e.g. 4x work in the same CI time. We need to look after our nightlies, but we need to do that anyway. I’ve started focusing on getting the various nightlies passing, e.g. pulpcore, pulp_file, pulp_ansible, etc.

What do others thing of this idea? Can we try this idea on pulpcore and pulp_file specifically?

3 Likes

If our test suite had run for more than an hour, I would like the idea. Other than that, I think that we will be reverting commits more frequently. Or, we would outsource running the tests to developers, which is not wanted.

30% is too strict. I would personally vote for 50%+ if I had to.

Definitely agree that 40 min is too long.

Maybe another option is to run tests at merge time instead of at night? That wouldn’t slow down PRs and you’d get faster feedback if there’s a problem.

1 Like