RFC better local reproducible tests

[david-caro]

Use tools like tox and docker-compose to achieve maximum test run reproducible results, to allow locally testing and debugging the issues.

This is implemented on pr #65

[jirikuncar]

Note that we used to have both tox and docker-compose in individual packages in the past. I'm in favor in returning tox as long as we don't duplicate testing logic.

The only thing one has to run in Invenio software repositories is ./run-tests.sh.

cc @inveniosoftware/triagers

[david-caro]

sohuld that be using the reproducible docker-compose or the lighter tox? Maybe allow passing an option for it?

[lnielsen]

As @jirikuncar said we originally had both. However, they were rarely used and thus weren't always working. I think they weren't used because they are simply too slow. Personally I very rarely run the entire test matrix locally. I prefer simply pushing a branch to github and let Travis do the job.

In case Travis report a problem for one of the builds that doesn't match my local install, I usually find it pretty fast to create a virtualenv and install the package. Also, you usually only need to run one or two particular tests over and over again, until the problem is solved.

So long story short, IMHO I don't think it's worthwhile the effort to add these.

[david-caro]

I've found myself several times in the last month trying to reproduce issues (on inspire and on the invenio-workflows) and it was quite painful to actually get the environment correctly as it's running on travis. Starting with different python versions, passing by different database versions, and finishing with the long list of untold dependencies needed to actually install some of the python packages into any virtualenv.

With this docker+tox setup, I have a fully working test environment it just the time it takes to download the docker image + the pip dependencies.
If you want to use a virtualenv and mess around, you can source .tox/${suite}/bin/activate and you are there. And if you want to live-debug a test failure, you can run tox -e $suite -- --pbd and it will pop up a pdb console on test failure (I suggest installing pdbpp before too, it's way nicer console, similar to ipython).

[david-caro]

This is very interesting for integration tests too, where you have multiple agents (postgres, elasticsearch, celery, rabbitmq, worker node, ...). Setting up all that environment in your own laptop it's hard, and having to maintain/sync it every time way harder.

[jirikuncar]

This is very interesting for integration tests too, ...

Sure. We are providing Docker recipe for overlays.

I think they weren't used because they are simply too slow.

👍

[david-caro]

Running the invenio-workflows basic (pytest on sqlite) recreating the venv/docker container once the deps are cached:

• on tox: 0m26.087s
• plain venv: 0m31.963s
• docker: 1m6.445s (it does not actually use the pip cache, yet at least)

Once you have run them once, you can enter the virtualenv of any of them:

• tox: source .tox/$suite/bin/activate
• plain venv: workon $venvname or source path/to/venv/bin/activa
• docker: docker-compose -f docker-compose.test.yaml run --rm test-$suite bash, and once there, source .tox/$suite/bin/activate

And once there, running the tests on any of the envs is technically the exact same process, so the same exact speed (just run py.test... or ./run_tests.sh or whatever you want to run there).

[david-caro]

And on travis, the full suite takes ~3min fully recreating the environment with docker (including the databases) and running tox on it:

https://travis-ci.org/inveniosoftware/invenio-workflows/builds/149122031

[david-caro]

Where a previous run without tox nor docker took >4min:

https://travis-ci.org/inveniosoftware/invenio-workflows/builds/76413761