TFactor: bulkerify GEMV (both MC and GPU)

[albestro]

In this PR we try to increase parallelisation of GEMV step of TFactor, which is by construction serial (all results goes into the same block tile), by using workspaces for storing partial results and then reducing them before the final TRMV step.

Algorithmically, the main change is that stepGEMV loop has been replaced with a single stepGEMVAll, and the concept has been applied in a similar way for both backends, MC and GPU:

• MC: pika::bulk splits input tiles over multiple task, each one stores the partial results in their workspace and just one at the end does the reduction.
• GPU: similarly to the CPU, the work is forked over different pika tasks, each one computing a partial result on a different GPU stream. These tasks then join into a single task which performs the reduction.

In order to implement this solution, workspaces for intermediate results have been added (there is another option which does not require any additional workspace nor reduction, and that we might explore for MC in another PR).

TODO:

• This work is based on TFactor: simplification and cleanup #1213
• check with @msimberg if we can do something different instead of the "fork" (e.g. a bulk for the GPU but each pika thread should get a different cuda stream)
• define better how many workspaces are needed (currently a random choice)
• profile and test performance of this solution
• Verify if there is anything to be "ported" from computeTFactor parallelise computation with bulk (MC Local and Distributed) #798
• workers default value should be backend dependent

Close #798.

EDIT: since this is conceptually similar to #798 and it is going to be closed as soon as this gets merged, I migrated here the doc fixes happened there.

[albestro]

cscs-ci run

just to check if there is any major problems

[rasolca]

LGTM.
Please name new functions in snake case, and rename existing internal functions if possible.

[rasolca]

All Ts are allocated at scheduling. Better reuse it.

[msimberg]

Only a few questions, nothing blocking.

[albestro]

cscs-ci run

[albestro]

Converted to draft in order to prevent merging, since I'm still doing some checks.

[albestro]

cscs-ci run

[albestro]

Apart the fixes committed between the two last CI runs, which fix for sure some problems, one major difference between the CI runs is that the oldest one ran on Eiger while the latest ran on Daint (because of the rebase on master).

The oldest one on Eiger had some failures that are concerning, specifically all the CPU UNIT_RANK_6, which failed due to a timeout on Eigensolver/GenEigensolver MC Distributed test (example)

I will try to investigate why that's the case and why it haven't happened on Daint.

[msimberg]

Very naive question: could you briefly explain why a vector of tile senders is more appropriate here instead of e.g. a round robin thing of tiles or a panel?

[albestro]

If I get it correctly, #1214 (comment) is on topic. The main idea is that in this way the caller can manage the workspace by re-using it over multiple calls.

[msimberg]

Sorry if I'm not getting the point: can the caller not manage workspaces if the tiles are in a panel?

[msimberg]

Answering my own question after a call with @albestro: It is using a round-robin panel for the tile senders, the senders are just accessed by the caller of computeTFactor. From the call we concluded that it may be worth passing the panel itself to computeTFactor and let it access the senders directly. This may be useful to avoid some splitTiles and unnecessary sender accesses that are currently happening in the caller(s) of computeTFactor. It remains to be seen if it's actually an improvement (in organization and/or performance). Thanks @albestro for the explanation and for looking into this!

[albestro]

Random thought: just saw this comment and wondering if we should have a look and see if we can "merge" this workspace into the other one? @rasolca