Integration tests strategy

[jmini]

Current setup:

The current setup is based on the fact that maven is starting a very old image of GitLab (version 12.9.2-ce.0) and is then executing some integration tests against this instance of GitLab.

Since I am maintaining this library I do not see any value in those tests.

They are slow and very flaky.
Contributors try to extend existing integration tests and sometimes those are failing because the GitLab version is outdated.

Testcontainers attempt:

There is an issue #925 and a pull request #935 where some work was made to replace the current setup based on maven with something based on Testcontainers.

This is almost finished but I never took the time to finish it.

Even if this would allow to update GitLab, I fear to face the same issue: it takes time to start GitLab inside a container and the test might continue to be flaky.

Decision:

• I will disable the current integration tests (they are too slow and too flaky) now.
• I will work on more reliable solution

New solution:

This should be based on a mock server (I am currently working with https://www.mock-server.com), in order to have fast and reproducible HTTP responses during our integration tests.

The maintenance of the stubs (the responses corresponding to given http requests) is a real challenge over time.
I do not want to write them manually and I would prefer capturing them at development time, by running the tests locally against a real GitLab instance (probably started locally, with or without docker).
This way we would always have a script available to renew or extend the mocked responses.

But in the CI we would have very fast and reproducible HTTP responses. Running the test suite would not require any docker or any gitlab installation.

This issue is open to collect feedback.

[LeJeanbono]

Even if this would allow to update GitLab, I fear to face the same issue: it takes time to start GitLab inside a container and the test might continue to be flaky.

With TC you can wait for readyness: https://java.testcontainers.org/features/startup_and_waits/

[jmini]

Yes, I think this is working in the branch pr_935 where I continued the work started in #935

---

I see two problems with test containers:

• The time it takes to have GitLab ready inside a docker installation. You can run this script https://github.com/jmini/gitlab-experiments/blob/main/testcontainers/RunGitlab.java to get a feeling.
• The need to have docker locally

Right now the running the tests is very quick. Even on GitLab CI the job takes now less than a minute now that I made the switch to use gradle.

The only advantage would be that it is probably easier than the solution with capture and replay.

---

At the end what we want in this project:

• know when our model is no longer up-to-date with GitLab
• know when somebody is breaking existing use-cases
• be quick(er) with the maintenance (when GitLab is modifying the REST API)

---

I think a "capture and replay" mechanism would really help with this. In an other context I worked on a prototype here: https://github.com/jmini/quarkus-experiments/tree/main/mockserver-record-replay

I now would explore if the concept presented in that project would work in a bigger project with so many API calls.