combining the ATLAS pipeline with different examples at different times

[omrctnr]

Hello everyone,

As far as I looked for, I don't know whether a similar topic has been written before. If yes, I am really sorry...

In our study, we have an enormous number of samples that come at different times. We want to make metagenome analysis gradually rather than as a whole, which is not reasonable due to the physical sufficient of the system.

Do you have a suggestion for us?

Thank you,
Omer

[jmtsuji]

@omrctnr In ATLAS, the qc and assembly modules can be run independently for all samples. The binning module requires your samples to be organized into BinGroups in the samples.tsv file, and samples in the same BinGroup will have their reads mapped to each other to guide genome binning. Lastly, the genomes and Genecatalog workflows require you to have all of your samples in the same batch.

In your case, then, you could run the qc and assembly modules for your samples as they arrive. Then, after all your samples arrive, you could move/copy all the sample folders into the same directory, merge the samples.tsv files into a single file, and finish the rest of ATLAS (binning, genomes, and Genecatalog). I've done this before, and it worked, although I might have needed to modify a couple files to get the combined ATLAS run to start (I do not remember now). For example, one issue you might run into is the stats folder -- you might need to combine the qc and assembly stats files manually.

To check if the merged ATLAS directory will run properly, after merging everything, you can run ATLAS with the -n flag (which shows all the steps that will be run, without executing them) to confirm that ATLAS will start at the point you expect in its workflow.

You could test this concept on a small sample set (e.g., 2 samples) and see if it works. Hope this helps.

[SilasK]

Ass @jmtsuji explained almost everithing in atlas is parallelized. Except the genecatalog and the genomes annotation.. for witch the quantification is done.

Even the binning can be used stepwise if you use single-sample binning. The Genecatalog, especially the eggNOG takes a lot of time. Maybe I would run atlas run genomes . So you get the quantification based on MAGs. Your results should improve the more samples you add. Before rerunning simply rename or delete the genome folder and rerun atlas. all samples that are already analized yhould't be re-assembled for example.

However do a test there might be some small files that could urge the pipeline to restart from the beginning. If you tell me which wan I can try to circumvent that.

[SilasK]

Is there a database of genomes/MAGs for the vaginal metagenome? You could quantify based on them. Couldn't you?