Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: replace all bbtool wrappers with one bbtool wrapper for all subcommands #1322

Merged
merged 119 commits into from
Nov 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
119 commits
Select commit Hold shift + click to select a range
a52e2e4
added bbtool gneric
SilasK May 2, 2023
e214a7a
less specific conda software
SilasK May 2, 2023
8e519d2
passed linting
SilasK May 2, 2023
81b1dff
added test
SilasK May 2, 2023
28a302e
formating
SilasK May 2, 2023
a8411be
undo wrong tests
SilasK May 2, 2023
7d0b8e4
no rich text
SilasK May 2, 2023
1d0d8e4
less outputfiles as they were not understood
SilasK May 2, 2023
77aa526
no space
SilasK May 2, 2023
0ec3ebc
snakemake has defines not out
SilasK May 2, 2023
0c7690e
dryrun worked
SilasK May 2, 2023
291b67c
typo
SilasK May 3, 2023
228faae
inferr log_files as seperate function
SilasK May 3, 2023
0f14e50
formating
SilasK May 3, 2023
3735f55
import _log_shell_redirect from script
SilasK May 3, 2023
599d1d9
list of keys
SilasK May 3, 2023
c50cd66
renamed functions
SilasK May 3, 2023
78bc7c4
fix
SilasK May 3, 2023
2d74a00
missing return
SilasK May 3, 2023
dadf217
log stderr in anycase
SilasK May 3, 2023
f258ea3
threads not t
SilasK May 3, 2023
47642bc
don't define threads
SilasK May 3, 2023
5ed9eb6
no pipe
SilasK May 3, 2023
6d6b8d3
inverted stderr and stdout
SilasK May 3, 2023
c142c7f
less memory
SilasK May 3, 2023
f4317b5
correct hasattr(key, log)
SilasK May 3, 2023
4956c7d
invert log files so they will be printed
SilasK May 3, 2023
42c0c26
not sam but out
SilasK May 3, 2023
9e6068a
paired=True
SilasK May 3, 2023
ea5dc07
black wrapper
SilasK May 3, 2023
1bbe3e0
formating
SilasK May 3, 2023
f623195
Merge branch 'master' into bbtools
SilasK May 3, 2023
ded8886
move to main folder
SilasK Oct 10, 2023
19ee88a
use python from main process
SilasK Oct 10, 2023
25c3106
add list of scripts suported
SilasK Oct 10, 2023
182cb7f
added threads
SilasK Oct 10, 2023
162d0fb
combined tests
SilasK Oct 10, 2023
0327950
updated test
SilasK Oct 10, 2023
9e26427
update
SilasK Oct 10, 2023
be9e01f
Merge branch 'master' into bbtools
SilasK Oct 10, 2023
c6517ce
formating
SilasK Oct 10, 2023
cd198d4
remove other wrappers
SilasK Oct 10, 2023
a8de480
check test
SilasK Oct 10, 2023
bd1787e
don't use sample iun name
SilasK Oct 10, 2023
4fc2b88
fix command
SilasK Oct 10, 2023
7feb5cd
fix test
SilasK Oct 10, 2023
693ed62
strange error for ecape character
SilasK Oct 10, 2023
79742bf
this was only a warning
SilasK Oct 10, 2023
19ff843
removed spade
SilasK Oct 10, 2023
142d2cb
update of script to allow unnamed arguments
SilasK Oct 10, 2023
d0a4c20
unnamed
SilasK Oct 10, 2023
a971aa9
debug logg
SilasK Oct 10, 2023
54b5889
fix test
SilasK Oct 10, 2023
ad3b275
parse_special_keys_as
SilasK Oct 10, 2023
75fa0de
format
SilasK Oct 10, 2023
086da74
spelling
SilasK Oct 10, 2023
c9e7692
fix
SilasK Oct 10, 2023
292c121
add extra
SilasK Oct 10, 2023
80ccf2d
add check to extra
SilasK Oct 10, 2023
200bf9d
need change in bbduck
SilasK Oct 10, 2023
bc3c9c6
fix names for bbduck
SilasK Oct 10, 2023
d284497
black
SilasK Oct 10, 2023
dfe895a
wrapper for no keyword
SilasK Oct 10, 2023
317eed7
fix
SilasK Oct 10, 2023
b38b8c6
fix single unamed arg
SilasK Oct 10, 2023
9965303
enforce string
SilasK Oct 10, 2023
ba8f46e
fixed error with single input
SilasK Oct 10, 2023
7cfff9c
black
SilasK Oct 10, 2023
e93e6b1
fix test for tadpole
SilasK Oct 10, 2023
2fca68e
is this the end?
SilasK Oct 10, 2023
7838f32
wrong mode
SilasK Oct 10, 2023
8f4f73e
allow tadpole to use extra in input
SilasK Oct 10, 2023
63bcb42
formating
SilasK Oct 10, 2023
0f328f1
a final fix.
SilasK Oct 11, 2023
e919daf
Update bio/bbtools/meta.yaml
SilasK Oct 11, 2023
22cfcd5
Update bio/bbtools/meta.yaml
SilasK Oct 11, 2023
3afe61d
Update meta.yaml
SilasK Oct 11, 2023
cb3e289
index genome before random access.
SilasK Oct 12, 2023
87a0dc4
Update bio/bbtools/test/Snakefile
SilasK Oct 24, 2023
55a7457
Update bio/bbtools/test/Snakefile
SilasK Oct 24, 2023
88a6082
Merge branch 'master' into bbtools
SilasK Oct 24, 2023
cbf5219
Merge branch 'master' into bbtools
SilasK Oct 31, 2023
8898edb
don't write index multiple times
SilasK Nov 8, 2023
51718bb
Merge branch 'master' into bbtools
SilasK Nov 8, 2023
fd72fa7
random reads doesn't allow nodisk
SilasK Nov 8, 2023
284d217
specify path for genome index
SilasK Nov 9, 2023
544860c
WIP use expand
SilasK Nov 9, 2023
90833ce
bump version
SilasK Nov 9, 2023
e554f22
wip use bbduck in pe
SilasK Nov 9, 2023
7ebc702
reformated wrapper
SilasK Nov 9, 2023
6670da6
re-implement get_java_opts
SilasK Nov 10, 2023
1224ace
missing return
SilasK Nov 10, 2023
41a077e
log why not correct memory is set
SilasK Nov 10, 2023
7a94db5
set default resources for bbtols
SilasK Nov 10, 2023
0049b1a
default resources before -- key
SilasK Nov 10, 2023
ddcda9e
specify resources in snakefile
SilasK Nov 11, 2023
6e7339d
Merge branch 'master' into bbtools
SilasK Nov 13, 2023
755491e
Merge branch 'bbtools' into bbtools2
SilasK Nov 13, 2023
e34340e
formating
SilasK Nov 13, 2023
64f1efd
simplified test
SilasK Nov 13, 2023
f9e52e2
tested works
SilasK Nov 13, 2023
b0bd233
index genome
SilasK Nov 13, 2023
7e71403
updated wrapper
SilasK Nov 13, 2023
c4faf0e
updated outm is now also paired
SilasK Nov 13, 2023
e457189
formating
SilasK Nov 13, 2023
37d0bf2
remove my script
SilasK Nov 13, 2023
05b766c
use regex to extract memory options
SilasK Nov 13, 2023
dd3263a
fix java opts memory
SilasK Nov 13, 2023
f35928a
make linting pass
SilasK Nov 13, 2023
873de9c
Update bio/bbtools/meta.yaml
SilasK Nov 13, 2023
a652bd7
Update bio/bbtools/meta.yaml
SilasK Nov 13, 2023
40d883c
updated meta
SilasK Nov 13, 2023
b33c376
move create_reads down as it is less important
SilasK Nov 13, 2023
e808afe
centralized imports
SilasK Nov 13, 2023
e2a0f63
Merge branch 'master' into bbtools
SilasK Nov 13, 2023
c03687e
rename private method add todo comments.
SilasK Nov 14, 2023
9dc9686
comment for include
SilasK Nov 14, 2023
a64f5ea
I hope this passes linting
SilasK Nov 14, 2023
c9afb41
blakc
SilasK Nov 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 0 additions & 18 deletions bio/bbtools/bbduk/meta.yaml

This file was deleted.

38 changes: 0 additions & 38 deletions bio/bbtools/bbduk/test/Snakefile

This file was deleted.

4 changes: 0 additions & 4 deletions bio/bbtools/bbduk/test/reads/pe/a.1.fastq

This file was deleted.

4 changes: 0 additions & 4 deletions bio/bbtools/bbduk/test/reads/pe/a.2.fastq

This file was deleted.

4 changes: 0 additions & 4 deletions bio/bbtools/bbduk/test/reads/se/a.fastq

This file was deleted.

50 changes: 0 additions & 50 deletions bio/bbtools/bbduk/wrapper.py

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ channels:
- nodefaults
dependencies:
- bbmap =39.01
- snakemake-wrapper-utils =0.5.3
- snakemake-wrapper-utils =0.6.2
- pigz=2
7 changes: 0 additions & 7 deletions bio/bbtools/loglog/environment.yaml

This file was deleted.

12 changes: 0 additions & 12 deletions bio/bbtools/loglog/meta.yaml

This file was deleted.

26 changes: 0 additions & 26 deletions bio/bbtools/loglog/test/Snakefile

This file was deleted.

4 changes: 0 additions & 4 deletions bio/bbtools/loglog/test/reads/pe/a.1.fastq

This file was deleted.

4 changes: 0 additions & 4 deletions bio/bbtools/loglog/test/reads/pe/a.2.fastq

This file was deleted.

4 changes: 0 additions & 4 deletions bio/bbtools/loglog/test/reads/se/a.fastq

This file was deleted.

26 changes: 0 additions & 26 deletions bio/bbtools/loglog/wrapper.py

This file was deleted.

138 changes: 138 additions & 0 deletions bio/bbtools/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
name: Generic
url: https://jgi.doe.gov/data-and-tools/software-tools/bbtools/bb-tools-user-guide/
description: |
Generic wrapper for BBTools. One pattern to run them all.
authors:
- Silas Kieser
input:
- input: All keywords in input are passed to the bbtool as key=value pairs. A special case is 'input', which is translated to 'in'. If exactly two files are provides for 'input' the wrapper parses them as in1, in2.
- flag: The keyword 'flag' is ignored and allows to specify input files that are not processed by the bbtool that is used.
output:
- out: If exactly two files are provides for 'out' the wrapper parses them as out1, out2.
- outm: Same as 'out'
- outu: Same as 'out'
- flag: The keyword 'flag' is not passed to the command.
params:
- command: Required parameter defining the command to be used e.g. 'bbmap.sh'
- extra: additional program arguments. All other parameters are passed to the bbtool as key=value pairs
notes: |
This wrapper allows to run any of the bbtools. The command is defined by the 'command' parameter, which is required.
**All keywords in input, output, params are passed as key value pairs to the command!**
Take care that they are valid for the bbtool.

As it is not possible to define 'in' as a keyword, the keyword 'input' is used instead. Allowed aliases are 'sample' and 'reads'.
### Paired input/output files

If exactly two files are provides for 'input' the wrapper parses them as in1, in2.
The same holds for the output keywords 'out', 'outm', 'outu'.
For all parameters, if more than two files are provided, the wrapper parses them as key=value1,value2,value3...

### Logging
The wrapper makes a detailed log of how he parse the parameters. If you want to use the log of a bbtool,
e.g. for parsing how many reads where processed, you have to specify a 'stdout', and 'stderr' log file.
The wrapper will then write only to the stderr-log file.

### List of all the tools in the bbtools suite 39.01

In theory all of them are sported by this wrapper, but we didn't test them all.
Scripts with different input/output might not be supported by this wrapper.
If you find one that is not yet supported, please feel free to adjust this wrapper accordingly and include a test case.

SilasK marked this conversation as resolved.
Show resolved Hide resolved
- bbmap.sh
- removehuman.sh
- removehuman2.sh
- mapnt.sh
- mapPacBio.sh
- bbmapskimmer.sh
- bbsplit.sh
- bbwrap.sh
- pileup.sh
- summarizescafstats.sh
- filterbycoverage.sh
- mergeOTUs.sh
- bbest.sh
- postfilter.sh
- bbduk.sh
- bbduk2.sh
- seal.sh
- summarizeseal.sh
- loglog.sh
- kmercountexact.sh
- bbnorm.sh
- ecc.sh
- khist.sh
- bbcountunique.sh
- commonkmers.sh
- kmercoverage.sh
- callpeaks.sh
- tadpole.sh
- tadwrapper.sh
- kcompress.sh
- stats.sh
- statswrapper.sh
- countgc.sh
- fungalrelease.sh
- filterbytaxa.sh
- gi2taxid.sh
- gitable.sh
- sortbytaxa.sh
- splitbytaxa.sh
- taxonomy.sh
- taxtree.sh
- reducesilva.sh
- synthmda.sh
- crosscontaminate.sh
- decontaminate.sh
- crossblock.sh
- dedupe.sh
- dedupe2.sh
- dedupebymapping.sh
- clumpify.sh
- bbmerge.sh
- bbmerge-auto.sh
- bbmergegapped.sh
- randomreads.sh
- bbfakereads.sh
- gradesam.sh
- samtoroc.sh
- addadapters.sh
- grademerge.sh
- printtime.sh
- msa.sh
- cutprimers.sh
- idmatrix.sh
- matrixtocolumns.sh
- countbarcodes.sh
- filterbarcodes.sh
- mergebarcodes.sh
- removebadbarcodes.sh
- demuxbyname.sh
- filterbysequence.sh
- filterbyname.sh
- filtersubs.sh
- getreads.sh
- estherfilter.sh
- bbqc.sh
- rqcfilter.sh
- shred.sh
- fuse.sh
- shuffle.sh
- calcmem.sh
- textfile.sh
- countsharedlines.sh
- filterlines.sh
- a_sample_mt.sh
- bbmask.sh
- calctruequality.sh
- makechimeras.sh
- phylip2fasta.sh
- readlength.sh
- reformat.sh
- removesmartbell.sh
- rename.sh
- repair.sh
- bbsplitpairs.sh
- splitnextera.sh
- splitsam.sh
- testformat.sh
- translate6frames.sh
7 changes: 0 additions & 7 deletions bio/bbtools/tadpole/environment.yaml

This file was deleted.

17 changes: 0 additions & 17 deletions bio/bbtools/tadpole/meta.yaml

This file was deleted.

Loading
Loading