Implementation of Horovod backend in Mesh TensorFlow

[EiffL]

This issue is to track the developments needed to finalize and validate the Mesh TensorFlow implementation relying on horovod for the backend. This overarching goal will encapsulate several smaller issues.

Goal

By the end of the hackweek, submit a Pull Request to https://github.com/tensorflow/mesh with our new implemenation for GPU clusters

Participants

The main participants to this task are:

• @tobias-liaudat
• @b-remy

Tasks

• : Identify reason for deadlocks: bug with mft.reshape mesh#4
• : Fix current implementation bugs: Checking/Optimizing Horovod implementation collectives mesh#2
• : Implement variables: Adding support for hvd LaidOutVariable  mesh#1
• : profiling a distributed training task (for instance the demo mnist example)
• : optimizing

Progress made on these subtasks can be reported here.

[EiffL]

And we have identified another issue here, I'm adding it tot the list of things we need to resolve: DifferentiableUniverseInitiative/mesh#4

[EiffL]

We have managed to mostly solve this the two first points of this issue, by the following:

• Fixing the random ops in a distributed setting
• Switching to this fork of horovod which doesn't seem to cause any deadlocks Concurrently running collective operations on process subsets [TensorFlow] horovod/horovod#2839