Adding parallelization via sbatch #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mcmcgrath13
reviewed
Aug 27, 2020
| ); | ||
| let force = if options.force { "-F" } else { "" }; | ||
|
|
||
| let memory = option_env!("SBATCH_MEM_PER_NODE"); |
There was a problem hiding this comment.
would it be worth making a util that does this pattern since it's used quite a bit?
There was a problem hiding this comment.
I have a feeling there's a rust-ism that I'm missing here. Maybe I need to employ the ? operator and some other magic?
There was a problem hiding this comment.
https://dev-notes.eu/2020/05/Set-Default-Values-in-Rust/
Apparently I can do option_env!("FOO").unwrap_or("my value");?
|
@broarr do you remember if this worked? I have a vague memory of you mentioning it didn't... but I wanted to see if this is something we should looking into picking up and exploring again, or do we close and stay sequential or handle paralellization outside of Rust (batch box instead). Any knowledge dump would help us. Thanks! |
|
Yeah, I think you're right. IIRC there was something with the cache
directories conflicting or something? There's another repository with
python code that I used first, might be easier to manipulate than old rust
code.
I don't see any problems with wrapping this in a bash script to parallelize
it. Did we ever get a singularity registry on OSCAR? That might simplify
some of these problems, and I think it's something we could use across the
board (especially with Jordan's next flow singularity container stuff)
…On Wed, Oct 19, 2022 at 2:52 PM Maria Isabel Restrepo < ***@***.***> wrote:
@broarr <https://github.com/broarr> do you remember if this worked? I
have a vague memory of you mentioning it didn't... but I wanted to see if
this is something we should looking into picking up and exploring again, or
do we close and stay sequential or handle paralellization outside of Rust
(batch box instead). Any knowledge dump would help us. Thanks!
—
Reply to this email directly, view it on GitHub
<#2 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALZRKTRK7EB4D3VTIWT4YQLWEA7P5ANCNFSM4QNB2DTA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
<http://ccv.brown.edu/> Bradford N. Roarr <https://github.com/broarr>
<https://directory.brown.edu/uuid/0adf6ab4-8d56-410a-8ac6-4b4ca136a534>
Senior Research Software Engineer
Data & Visualization
Center for Computation and Visualization
Brown University | OIT
On Wed, Oct 19, 2022 at 2:52 PM Maria Isabel Restrepo < ***@***.***> wrote:
@broarr <https://github.com/broarr> do you remember if this worked? I
have a vague memory of you mentioning it didn't... but I wanted to see if
this is something we should looking into picking up and exploring again, or
do we close and stay sequential or handle paralellization outside of Rust
(batch box instead). Any knowledge dump would help us. Thanks!
—
Reply to this email directly, view it on GitHub
<#2 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALZRKTRK7EB4D3VTIWT4YQLWEA7P5ANCNFSM4QNB2DTA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
<http://ccv.brown.edu/> Bradford N. Roarr <https://github.com/broarr>
<https://directory.brown.edu/uuid/0adf6ab4-8d56-410a-8ac6-4b4ca136a534>
Senior Research Software Engineer
Data & Visualization
Center for Computation and Visualization
Brown University | OIT
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.