List of improvements for big datasets

[grst]

• speed up group_abundance
• alignment dist as CLI that reada and writes anndata -> to be able to submit this easily to a cluster
• parallelize chain pairing
• increase chunk size in multiprocessing (less IOwait for pickling... maybe can reduce the amount of data sent to each worker)
• sensible defaults for graph layout. How would force atlase work?
• increase point size and set edges to False, when plotting many clonotypes
• optimize/parallelize construction and reduction of coord dictionary. (superseeded by Refactor CDR3-network construction.  #191)
• further optimize matrix housekeeping in tcr_neighbors (superseeded by Refactor CDR3-network construction.  #191)

[grst]

In GitLab by @grst on Apr 1, 2020, 14:43

[grst]

In GitLab by @grst on Apr 6, 2020, 08:26

changed title from List of improvements for {-many cell-}s to List of improvements for {+big dataset+}s

[grst]

In GitLab by @grst on Apr 6, 2020, 08:26

changed the description

[grst]

Maybe this can be abused to compute Levenshtein distance in linear time:
https://github.com/wolfgarbe/SymSpell

and/or to prefilter which alignments to compute

[grst]

Consider storing clonotype x clonotypes matrices rather than cell x cell matrices.
This could save a lot of space & computation time
The clonotype network plot would then be updated to show a single dot per
clonotype (potentially connected into clusters), where the size represents the number of cells.

[grst]

I tried to compute edit distance with symspellpy, but is actually 3x slower than the current levenshtein implementation. The reason for this is probably that the Levenshtein package is a highly optimized C library, while symspellpy is a pure python port of the SymSpell library.
Would need a Python wrapper for C SymSpell for max performance.

[grst]

Maybe WFA provides even better alignment performances than parasail?
https://github.com/smarco/WFA
Need to try it out at some point. In any case, we would need a python wrapper for that library first.

[grst]

mmseqs2 should do a great job for this.
Just tried it for some bacterial sequences (many-against-many). It's super fast and I don't see why we couldn't apply it here.

[grst]

superseded by #190 (faster chain_pairing) and #230.