improve speed of data transfer and ingestion

[anne-urai]

Currently, after finishing training in the late afternoon (~4pm) and transferring all data to the server PC (~6pm), it takes until fairly late the next day to display the animal's behavior in the IBL navigator. This means that when a behavioral criterion is reached, the user mostly won't find out until the animal has already been trained the day after. The additional lag also makes it tricky to identify and fix behavioral problems quickly.

I remember there has been talk of changing the upload to FlatIron, so that small behavioral files are uploaded quickly. In an ideal world, all upload and ingestion would happen overnight so that the experimenter has access to yesterday's data through the Navigator the next morning.

[oliche]

So there was a few improvements already:

• data gets registered as soon as it gets on the server. Behaviour extraction is fast and has high priority.
• small files are uploaded separately from large files.

Here are some of the issues:

• so far we run jobs sequentially, so if you have a large ephys job already running when the data gets copied, the training extraction will be delayed
• Globus tasks run asynchronously, so the transfer status is checked only once a day. There is a lot to gain here by running threads or daemons that keep track of Globus transfers and set the database status as soon as the transfers are complete.
• send a nudge to DJ as soon as transfers are complete too ?