What options are there to balance sharding?

[MattiJarvinen]

We run all of our test suites in shards and parallel but some suites take a lot more time than others.

Most of our runners complete around 15 min but some take over twice as much. Does --ordering file also affect sharding so we could get long running suites in separate shards? What other solutions we could try?

We have around 600 suites and 40 runners.

[joonaskuisma]

Hi @MattiJarvinen !

With the --ordering file, you can freely group suites or tests using curly braces {} to define their execution order.

For example, an order.txt file could look like this:

{
--suite Suite.First 15 Min
--suite Suite.Second 15 Min
}
--suite Suite.30 Min

In this case, the 30-minute test runs in its own subprocess, while the two 15-minute tests run together in the same subprocess.

Thus, executing the command:

pabot --ordering order.txt --processes 2 .

would take 30 minutes to complete. However, the final result of a 30-minute duration can also be achieved without using curly braces, as long as the longest tests are placed at the beginning of the order.txt file. For example:

--suite Suite.30 Min
--suite Suite.First 15 Min
--suite Suite.Second 15 Min

In this case, all tests are executed in their own subprocesses, but once the executor is freed up, it will immediately start processing the next test in line. This way, the total execution time will still be 30 minutes.