-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
9 changed files
with
18 additions
and
18 deletions.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -416,12 +416,11 @@ <h2 id="4-check-size-of-conda-environment-tar-archive">4. Check Size of Conda En | |
<p>The tar archive, <code>env-name.tar.gz</code>, created in the previous step will be used as input for | ||
subsequent job submission. As with all job input files, you should check the size of this | ||
Conda environment file. <strong>If >100MB in size, you should NOT transfer the tar ball using | ||
<code>transfer_input_files</code></strong>. Instead, you should plan to use either CHTC’s web proxy, SQUID or | ||
<code>transfer_input_files</code></strong>. Instead, you should plan to use the | ||
large data filesystem Staging. Please contact a research computing facilitators at | ||
[email protected] to determine the best option for your jobs.</p> | ||
|
||
<p>More information is available at <a href="file-avail-squid.html">File Availability with Squid Web Proxy</a> | ||
and <a href="file-avail-largedata.html">Managing Large Data in HTC Jobs</a>.</p> | ||
<p>More information is available at <a href="file-avail-largedata.html">Managing Large Data in HTC Jobs</a>.</p> | ||
|
||
<h2 id="5-create-a-job-executable">5. Create a Job Executable</h2> | ||
|
||
|
@@ -459,9 +458,8 @@ <h2 id="6-submit-jobs">6. Submit Jobs</h2> | |
<li>Remember to transfer your Python script and the environment <code>tar.gz</code> file via | ||
<code>transfer_input_files</code>. | ||
<strong>Since the <code>tar.gz</code> file will almost certainly be larger than 100MB, | ||
please <a href="mailto:[email protected]">email</a> us about different tools for | ||
delivering the installation to your jobs, | ||
likely our <a href="file-avail-squid.html">SQUID web proxy</a>.</strong></li> | ||
please place it in your <code>/staging</code> directory and use the instructions in | ||
(file-avail-largedata.html) to transfer the <code>tar.gz</code> file instead.</strong></li> | ||
</ul> | ||
|
||
<!-- | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -492,7 +492,7 @@ <h4 id="1-compile-mpi-code">1. Compile MPI Code</h4> | |
<p>If your MPI program is especially large (more than 100 MB, compiled), or | ||
if it can <em>only</em> run from the exact location to which it was installed, | ||
you may also need to take advantage of CHTC's shared software location | ||
or our public web proxy called Squid. Email CHTC's Research Computing | ||
or large data staging system. Email CHTC's Research Computing | ||
Facilitators at <a href="mailto:[email protected]">[email protected]</a> if this is the case.</p> | ||
|
||
<p><span name="script"></span></p> | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -452,9 +452,9 @@ <h2 id="save-installed-packages-for-later-jobs">Save Installed Packages For Late | |
submit server (specifically where ever you were located when you submitted | ||
the interactive build job). A copy of <code>packages.tar.gz</code> will be present. <strong>Be | ||
sure to check the size of the project tarball before proceeding to subsequent job | ||
submissions.</strong> If the file is >100MB please contact us at <a href="mailto:[email protected]">[email protected]</a> so | ||
that we can get you setup with access to our SQUID web proxy. More details | ||
are available on our SQUID guide: <a href="file-avail-squid.html">File Availability with SQUID</a></p> | ||
submissions.</strong> If the file is >100MB please use your staging directory instead. | ||
More details are available in our large data staging guide: | ||
<a href="file-avail-largedata.html">Use Large Input and Output Files Via Staging</a></p> | ||
|
||
<pre class="term"><code>[alice@submit]$ ls | ||
build.sub julia-#.#.#-linux-x86_64.tar.gz julia-build.log | ||
|
@@ -552,8 +552,8 @@ <h2 id="create-htcondor-submit-file">Create HTCondor Submit File</h2> | |
<pre><code class="language-{.sub}">transfer_input_files = julia-#.#.#-linux-x86_64.tar.gz, script.jl, my-project.tar.gz | ||
</code></pre> | ||
|
||
<p>For project tar.gz files that are larger than 100MB, email a facilitator about | ||
using SQUID.</p> | ||
<p>For project tar.gz files that are larger than 100MB, use your <code>/staging</code> directory. | ||
For more information, see our <a href="file-avail-largedata.html">guide on staging large data</a>.</p> | ||
|
||
<p>Modify the CPU/memory request lines to match what is needed by the job. | ||
Test a few jobs for disk space/memory usage in order to make sure your | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -318,7 +318,7 @@ <h1 id="2-job-qualifications"><strong>2. Job Qualifications</strong></h1> | |
<li> | ||
<p><strong>Moderate Data Sizes</strong>: We can support input file sizes of up to | ||
20 GB per file per job. This covers input files that would normally be | ||
transferred out of a <code>/home</code> directory or use SQUID, in addition to larger | ||
transferred out of a <code>/home</code> directory or use staging, in addition to larger | ||
files up to 20GB. Outputs per job can be of similar sizes. If your input or | ||
output files are larger than 1GB, or you have any other questions about | ||
handling data on resources beyond CHTC, please <a href="mailto:[email protected]">contact us</a>!</p> | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters