Benefit of grid parallelization and workflow engines

[hironori-kondo]

Hi all,

quacc has been a wonderful tool for me, but as I've begun to scale up the number of materials I calculate at once, I'm beginning to think more about the benefits of grid parallelization (as implemented, for example, by @tomdemeyere for Espresso).

The beauty of workflow engines, as I understand it, is that they do the troublesome work of figuring out which calculations are dependent, then fit the jobs into your available resources like a time-ordered game of Tetris. You thus get to decrease your wall-clock time by consuming more resources to compute more things at once.

Given many materials for which to run the same calculations (within some maximum reasonable allocation), however, what is the benefit of this approach? It seems that this game of Tetris would inevitably have some gaps in usage. Why would it not be quicker, for example, to run more materials in parallel (where each calculation job therein is less parallel/takes longer, in order to fit more materials)? This seems (in my mind) to minimize downtime/overhead; while each material takes longer to finish, they all finish roughly together instead of being broken up and stacked awkwardly. I am, of course, assuming that your allocation is roughly commensurate with a divisor of the number of materials you have, but that doesn't seem so terrible to figure out.

One benefit I see of splitting up large jobs (especially phonon jobs) is that you don't need to plan around jobs that require significantly more memory than others, but am I missing some crucial understanding here? Wiser guidance would be much appreciated.

[tomdemeyere]

Hi @hironori-kondo

Let say that you have 30 materials which require each 30 representations and you have 30 nodes.

In that case, yes it might be better to just run the 30 tasks with normal phonon job, at the same time by attributing them 1 node each.

Now let say that you have 900 nodes. Without the grid parallelisation there would be no way to efficiently parallelise over all these nodes.

Basically the grid technique becomes interesting when conventional parallelisation techniques would not allow you to parallelise efficiently over your available computational ressources.

Honestly even in the first case, even if you use grid, the timing would not be much slower. But yes, if you have 10000 materials and 2 nodes, don't use grid.

Does that answer your question?

[hironori-kondo]

Yes, that does! Thank you very much!