1. VAME Workflow
⚠️ Check out also the published VAME Workflow Guide! Including more hands-on recommendations and tricks HERE.
Thank you for working with VAME! We are excited to explain its functionality and give some guiding on how to analyse your data with our newest update of VAME v.1.0-alpha. The new version introduces a lot of new functions, has a much cleaner implementation and fixes some previous bugs. This wiki page will cover the VAME workflow, which consists of four main steps and optional steps to analyse your data.
- Set up your project, get your data into the right format and create a training dataset for the VAME deep learning model
- Train a variational autoencoder which is parameterized with recurrent neural network to embed behavioural dynamics
- Evaluate the trained model based on its reconstruction capabilities
- Segment behavioural motifs/poses/states from the input time series
- Optional: Create motif videos to get insights about the fine grained poses
- Optional: Investigate the hierarchical order of your behavioural states by detecting communities in the resulting markov chain
- Optional: Create community videos to get more insights about behaviour on a hierarchical scale
- Optional: Visualization and projection of latent vectors onto a 2D plane via UMAP
- Optional: Use the generative model (reconstruction decoder) to sample from the learned data distribution, reconstruct random real samples or visualize the cluster centre for validation
- Optional: Create a video of an egocentrically aligned animal + path through the community space (similar to our gif on github)
- Some remarks about considerations of obtaining behavioural data can be found at the end of this wiki
The following will cover all the steps and optional choices above in detail. Note that we also have a workflow script with short explanations and from where you can call the function after installing VAME. We hope that VAME helps to accelerate your research!
Throughout VAME we assume that you either use DeepLabCut (DLC) for pose estimation or that your resulting x,y-coordinates are stored in a .csv file similar to the one of DLC.
VAME starts by initializing a new project. This will create a folder with your project name + the current date. Within this folder four sub-folder will be created (data, model, results and videos) and a config.yaml file.
config = vame.init_new_project(project='Your-VAME-Project', videos=['/directory/to/your/video-1','/directory/to/your/video-2','...'], working_directory='/YOUR/WORKING/DIRECTORY/', videotype='.mp4')
The function takes in four arguments, the first is your project name as a string, the second takes a list of the exact directory of your videos as string and the third specifies your working directory i.e. where your project folder gets created. The last one specififes if you are using .mp4 or .avi videos.
Note: You need to give a full path to a video like '/directory/to/your/video-1.mp4', otherwise the config.yaml is not correctly initialized.
After creation you need to move your pose estimation .csv (e.g. video-1.csv) into the '/YOUR/WORKING/DIRECTORY/Your-VAME-Project-Mar22-2021/videos/pose_estimation/ folder.
Once you created your project you can easily address your config file by specifying the path to it
config = '/YOUR/WORKING/DIRECTORY/Your-VAME-Project-Mar22-2021/config.yaml'
The config.yaml file is essential for your VAME project as you can set all the necessary parameter in here. If you want to know more about the hyperparameter in here check out our little guide.
From here you are all set and can continue either aligning your data egocentrically or, if your data is by design egocentric (head-fixed experiment with a stationary camera), you can transform the pose .csv to a .npy array ready for analysing with VAME.
This function comes in handy to help to egocentrically align the behavioural data if it was for example recorded in an open field arena by one fixed camera from the bottom or top view.
vame.egocentric_alignment(config, pose_ref_index=[0,5], crop_size=(300,300), use_video=False, video_format='.mp4', check_video=False)
The takes in the string to the config file and the pose_ref_index which is a list of reference coordinate indices for alignment. As an example we take the labeling from the VAME paper. Here we put ther markers like the following:
0: snout, 1: forehand_left, 2: forehand_right, 3: hindleft, 4: hindright, 5: tail
This results into a DLC .csv file with 18 entries:
snout_x, snout_y, snout_accuracy, forehand_left_x, forehand_right_y, ... , tail_x, tail_y, tail_accuracy
For the reference coordinate we chose to align the animal along the axis from snout to tail which leavesus with [0,5] as reference coordinates. Please make sure you do this in a similar fashion, otherwise it can not be guaranteed that the data is egocentrically aligned and you will get results which are hard to interpret.
The argument crop_size defines the range of the markers or with other words, the pixel area from the video which contains the animal. As long as you get a stable looking timeseries with no cut-offs you do not need to change this one. The other arguments are just meant for debugging and quickly checking that the animal data is egocentrically aligned in form of a video. Please note that by setting use_video to True the function will take much longer to run.
If your experiment is by design egocentrical (e.g. head-fixed experiment on treadmill etc) you can use the following helper function to convert your .csv to a .npy array.
vame.csv_to_numpy(config, datapath='\\PATH\\TO\\YOUR\\.CSV')
Here, you only need to specify the path where your .csv is stored.
After your data is prepared and stored in the right format in the data folder you can call this function to create a training dataset along with a test dataset. The split fraction is defined in the config.yaml under the hyperparameter test_fraction.
vame.create_trainset(config)
After finishing the initialization of your project and the arrangement of the data we can finally start to train the VAME variational autoencoder which is parameterized by a recurrent neural network as encoder, decoder and prediction decoder.
vame.train_model(config)
For this you need to open your config.yaml and set the following hyperparameter if they differ for your project:
model_name: VAME_MODEL (you can set any name you want and also train different models by giving different names)
num_features: 12 (This is the number of x-y-coordinates from your virtual marker)
time_window: 30 (Defines how many timesteps the RNN considers for embedding a data point)
zdims: 30 (Specifies the number of dimension the data will be embedded in. We suggest 30 for a similar dataset as ours)
NOTE: As this is the deep learning part of VAME a GPU will be an advantage in terms of computing time. VAME checks automatically for a GPU and prints out which one it is using.
After the training has finished you can evaluate your model performance by inspecting the loss curves, but more importantly but inspecting its reconstruction capabilities (five top plots in second Figure). We also plot the prediction capabilities (five bottom plots in second figure) but these can be somewhat neglected as it the prediction decoder serves as a regularization for the dynamics.
vame.evaluate_model(config)
After successfully training the VAME model we can finally start to segment the behavioural time series into motifs via:
vame.pose_segmentation(config)
Please set the following config.yaml parameter:
n_cluster: 30 (how many k-Means states should be clustered)
individual_parameterization: False (If True, every animal gets their own parameterization/clustering via k-Means)
A note for the latter parameter, this might be useful in cases where you have different conditions and want to parameterize only certain individuals or groups with the same k-Means clustering. Later on, the community approach can help to identify similar behavioural regions from different groups.
An important note is that the resulting motif labels are starting at the time_window / 2 frame. This means if you chose time_window = 30, your first motif label is the 15th frame of your video.
Once you run all the main steps, you can finally start getting insights about your behaviour. The most obvious way is to look at a video for each found motif.
vame.motif_videos(config, videoType='.mp4') - videoType='.mp4' refers to your original video, the output will be an .avi
These will be most likely short behavioural poses with distinct dynamics. To observe the motif in there full dynamical range we introduce next our hierarchical community detection script. Once you obtained communities of behavior i.e. walk, rear, turn etc. you can also create videos of communities, see below
Welcome to our community detection script. This is, to be fair, not as automatic and intuitive. Lets dive into it and figure out its mechanisms
vame.community(config, show_umap=False, cut_tree=None)
The second argument is a helper to visualize the detected communities on a UMAP. We will first consider the behavior of the third argument cut_tree.
cut_tree can be set to either None or an integer, usually between 1-3. Setting it to an integer will automatically finish the process, however, it might be that some motifs are missing and therefore it is recommended to set it to 'None'.
Setting the argument to None will invoke some inline commands. Here, a tree structure will be plotted (its recommended to not have inline plots) and a question will appear on the console:
Where do you want to cut the Tree? 0/1/2/3/... - The following Figure visualizes an example Tree with cut lines (Note, these are not visible in the actual plot):
From here you can choose where to set the cut line. The connected motifs below the cut line will now form a community. For example, if
you choose cut_tree=2, the communities are the following: [10, 2 12, 5, 0], [11, 3, 14, 8], [1, 6, 13], [9, 4, 7]
This list pops up in your console and you can inspect if every motif is listed:
Are all motifs in the list? (yes/no/restart) Choosing yes will finish the process, choosing restart will let you choose a new cutline and
choosing no will let you insert motif numbers manually either inside a community list or at the end, appending a new list.
Extend list or add in the end? (ext/end) By choosing either ext (extend a given list) or end (append new list), the next question is:
Which motif number? Here you add a missing motif from the tree as integer number
Case ext:
At which position in the list? (pythonic indexing starts at 0) Choose the list starting with zero from above which you wish to extend. The new list will be printed. The script asks you again Are all motifs in the list? (yes/no/restart). If more then one are missing, continue with no, if everything seems correct now continue with yes or restart the process.
Case end:
Which motif number? Type in the motif number you wish to append as a list. The new lit gets printed and you can either confirm that everything seems to be right now or keep adding motifs until all the motifs are correct.
We know this script is a bit experimental and hope to make this workflow more user-friendly in the future. Till then, enjoy the 80's style game ;)
With this function you can create videos of from your community labels to observe how the behaviour in your experiment is hierarchically connected.
vame.motif_videos(config, videoType='.mp4') - videoType='.mp4' refers to your original video, the output will be an .avi
OPTIONAL: vame.visualization() (Visualization and projection of latent vectors onto a 2D plane via UMAP)
We finally added a function to visualize your embedded latent space within a 2D UMAP projection.
vame.visualization(config, label="motif") options: label: None, "motif", "community"
- Option None: Create an embedding of your latent vectors with no labels applied
- Option "motif": Create an embedding of your latent vectors with motif labels applied
- Option "community": Create an embedding of your latent vectors with community labels applied
Note that you can set the specific parameter for the UMAP creation within the config.yaml.
OPTIONAL: vame.generative_model() (Generate realistic behavioral samples, reconstruct samples or reconstruct cluster centre)
VAME learns a smooth spatiotemporal latent space or manifold, from which it is possible to generate new samples from the overall distribution or individual motif clusters, reconstruct original samples or even visualize the cluster center as a pose time series to validate cluster.
vame.generative_model(config, mode="centers") - options: mode: "sampling", "reconstruction", "centers", "motifs"
- Option "sampling: Generate random, unseen samples from the overall latent distribution
- Option "reconstruction": Reconstruct real samples
- Option "centers": Visualize cluster center as pose time series
- Option "motifs": Generate random, unseen samples from individual clusters (e.g. for validation purposes)
vame.gif(config, pose_ref_index=[0,5], subtract_background=True, start=0, length=500, max_lag=30, label='community', file_format='.mp4', crop_size=(300,300))
Side view -> self egocentric alignment or intrinsic information use, multiple animal camera angle problems etc