Functional tests will now run inside the container running Pulp

Development
#1

The latest changes merged into plugin_template make the functional tests execute from inside the container that is running Pulp. This makes it more similar to how developers run tests in their environments.

Before you apply the latest plugin_template to your plugin, you shoud update the aiohttp_fixtures_origin value to 127.0.0.1 in your template_config.yaml. If your tests don’t use pytest fixtures, this is not necessary. However, if you choose to use them in the future, you will need to make this change.

2 Likes
#2

It seems there have been some silent assumptions with this change that we need to discuss.

Prior to this change, all functional tests were required to be able to run against a Pulp API served at whatever URL that the test runner could reach. This implied that pulp-smash was used to configure how to reach the Pulp under test though the api. But also some methods to execute code in the shell or python environment of that Pulp were provided. Let us refer to this setup as a remote test run.
The change announced and executed here is a change to greatly simplify the way we run tests by running them inside the environment in which Pulp itself is running, a local test run.

To me this implies immediately that the requirement for tests to be fit for remote test runs would be dropped immediately and completely. But I heard that we may still want to be able to run most of the tests in remote test runs. A marker for tests that need to run local was thus introduced.
My claim is that without regularly testing it and investing a lot of dev-hours, we cannot maintain the use of this marker in a quality that will give a good QE experience. Resorting to guess-work will give the worst possible experience. Also telling a new contributor “Yes, that test you wrote looks nice and it is green, even in the CI. BUT…” is something i really want to avoid.

So my proposal to be straight forward about this change:
Let’s embrace the fact that the tests run locally as a great simplification for the test suite. Going forward, let’s further simplify the tests by gradually moving all the pytest facilities in pulp-smash over to Pulpcore (with the opinionated configuration for localhost). Replace all the execute-code-on-server interfaces with easier (hopefully faster) implementations that need not bother using remote shell access. Eventually remove the test dependency on pulp-smash altogether. That last step should specifically be helpful, as pulp-smash is not versioned and changes there are always risky.

FWIW, the CLI test suite is still running from outside the CI container on plugins that use it. We could use parts of those as remote against a cluster smoke tests.

1 Like
Retirement of Pulp Smash
#3

The two aspects I’m reading in your proposal (brief recap below in bullet form) all sound good to me. +1

  • Assume that all tests are run against the same system where Pulp is running. This provides direct filesystem access for assertion, import/export, and pulpcore-manager calls
  • Moving the pytest plugin pieces to pulpcore and the eventual retirement of pulp-smash altogether

If I’ve missed an aspect please let me know. The CLI test suite I think of as a more specialized component so I think it can run however it wants to (aka a remote test run is fine).

The downside is that we cannot run the test suite on places like k8s environments as easily (maybe?) but I think that’s ok. Ultimately, to me, the test suite is to assert the correctness of the source tree for either Pulp development, or downstream patching/packaging. I believe this simplification brings much more benefits than downsides.

One practical question: How can we communicate the retirement of pulp-smash facilities and a timeline for that? One possible idea is to introduce deprecations in those pulp-smash facilities, and let the deprecation CI check alert plugin developers…

#4

At some point we may even add logging assertions.

As far as pytest fixtures goes, we can simply duplicate them. Pytest will chose whatever it finds first. I don’t see any plugin involvement there. For all the other pulp-smash imports, we will probably keep pulp-smash in the same unversioned state around forever. So i do not see any required plugin involvement at any given point in time until they decide to stop importing from pulp-smash.

1 Like
#5

I would like to retire pulp-smash.

4 Likes
Pulp/Galaxy integration meeting
Pulpcore Meeting Minutes
#6

The general feedback I’ve heard from discussion of this topic in the pulpcore and galaxy_ng integration meetings is that we want to allow those using pulp-smash from PyPI or from source checkouts to continue doing so. We do want to stop making changes to pulp-smash though (which is the retirement part).

To do this is very simple as a two-step process.

First, archive it on github, which would cause github to put a banner up that says it’s no longer accepting changes. It would show like this one: https://github.com/pulp/pulpcore-plugin

Second, let folks port off of it at their own pace.

1 Like
#7

To add to this, we will not yank pulp-smash from pypi until after pulp2 goes EOL

2 Likes