Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bacpop-202 Assign fallback to refs to db #49

Merged
merged 29 commits into from
Dec 6, 2024
Merged

Bacpop-202 Assign fallback to refs to db #49

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
014ec0b
feat: get 'working' version
absternator Nov 21, 2024
6dc261a
feat: get working version cleaned
absternator Nov 21, 2024
4396446
fix: remove redundant deletion of full_assign directory in get_cluste…
absternator Nov 21, 2024
569c289
feat: get refactored version working
absternator Nov 22, 2024
d764af7
feat: use pandas for data manipulation
absternator Nov 22, 2024
7b1ae88
feat: remove bad include files.txt
absternator Nov 22, 2024
5a4a806
feat: add dataclass config to reduce params
absternator Nov 22, 2024
ec2dbce
feat: enhance documentation and type hints in file handling functions
absternator Nov 25, 2024
d0ea736
pycode style fixes
absternator Nov 25, 2024
3fcf23f
tests: get all unit tests up for ref fallback
absternator Nov 27, 2024
53f3e32
fix doc blocs
absternator Nov 27, 2024
38cde8a
fix: doc blocs again
absternator Nov 27, 2024
4b4dea4
fix: improve docstring formatting in update_external_clusters_csv fun…
absternator Nov 27, 2024
ed29fd3
fix: correct docstring formatting in update_external_clusters_csv fun…
absternator Nov 27, 2024
8398fec
fix: correct parameter docstring formatting in update_external_cluste…
absternator Nov 27, 2024
a78e25b
fix: correct parameter and return docstring formatting across multipl…
absternator Nov 27, 2024
d458f8b
fix: correct parameter docstring formatting in update_external_cluste…
absternator Nov 27, 2024
2b237f3
feat: updates for new netowrk code
absternator Nov 27, 2024
5eec95a
update conflict logic for include files
absternator Nov 29, 2024
59117ca
Update beebop/assignClusters.py
absternator Nov 29, 2024
be0e070
Update beebop/assignClusters.py
absternator Nov 29, 2024
2b11e2d
Update beebop/assignClusters.py
absternator Nov 29, 2024
0480f18
pr changes for cleanup
absternator Nov 29, 2024
b6e8555
fix lint
absternator Dec 2, 2024
1214b41
Add miscellaneous section to README with analysis process diagram
absternator Dec 2, 2024
972c302
Update README diagrams section and remove unused parameter from filte…
absternator Dec 5, 2024
2dd4194
Merge branch 'bacpop-186-v9-db-support' of https://github.com/bacpop/…
absternator Dec 5, 2024
9c596de
update diagram as per pr comments
absternator Dec 5, 2024
3e9d0d9
update drawio
absternator Dec 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions beebop/app.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
from redis import Redis
import redis.exceptions as redis_exceptions
from rq import Queue
from rq.job import Job
from rq.job import Job, Dependency
import os
from io import BytesIO
import zipfile
Expand Down Expand Up @@ -169,7 +169,7 @@ def get_species_config() -> json:
"""
all_species_args = vars(get_args().species)
species_config = {
species: get_species_kmers(args.dbname)
species: get_species_kmers(args.refdb)
for species, args in all_species_args.items()
}
return jsonify(response_success(species_config))
Expand Down Expand Up @@ -248,8 +248,13 @@ def run_poppunk_internal(sketches: dict,
400,
)

db_fs = DatabaseFileStore(
f"{dbs_location}/{species_args.dbname}",
# pass in both full and refs to assign
ref_db_fs = DatabaseFileStore(
f"{dbs_location}/{species_args.refdb}",
species_args.external_clusters_file,
)
full_db_fs = DatabaseFileStore(
f"{dbs_location}/{species_args.fulldb}",
species_args.external_clusters_file,
)

Expand Down Expand Up @@ -279,7 +284,8 @@ def run_poppunk_internal(sketches: dict,
hashes_list,
p_hash,
fs,
db_fs,
ref_db_fs,
full_db_fs,
args,
species,
**queue_kwargs)
Expand All @@ -290,7 +296,7 @@ def run_poppunk_internal(sketches: dict,
job_network = q.enqueue(visualise.network,
args=(p_hash,
fs,
db_fs,
full_db_fs,
args,
name_mapping,
species),
Expand All @@ -302,13 +308,13 @@ def run_poppunk_internal(sketches: dict,
job_microreact = q.enqueue(visualise.microreact,
args=(p_hash,
fs,
db_fs,
full_db_fs,
args,
name_mapping,
species,
redis_host,
queue_kwargs),
depends_on=job_network, **queue_kwargs)
depends_on=Dependency([job_assign, job_network], allow_failure=True), **queue_kwargs)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised you need to specify dependency on job_assign here, when job_network already depends on job_assign..? What difference does allow_failure make?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the reason we specify job assign as well is because we need the clusters info which is a dependency... thus if network fails then microreact will run and can still get cluster info as dependency,,,, allow_failiure means we still run if the previous jobs fails

redis.hset("beebop:hash:job:microreact", p_hash, job_microreact.id)
return jsonify(
response_success(
Expand Down
Loading
Loading