Skip to content

Commit

Permalink
Merge Split Branch
Browse files Browse the repository at this point in the history
  • Loading branch information
actions-user committed May 9, 2024
2 parents cb84dc3 + bf3c84a commit 35c0197
Show file tree
Hide file tree
Showing 9 changed files with 18 additions and 18 deletions.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -416,12 +416,11 @@ <h2 id="4-check-size-of-conda-environment-tar-archive">4. Check Size of Conda En
<p>The tar archive, <code>env-name.tar.gz</code>, created in the previous step will be used as input for
subsequent job submission. As with all job input files, you should check the size of this
Conda environment file. <strong>If &gt;100MB in size, you should NOT transfer the tar ball using
<code>transfer_input_files</code></strong>. Instead, you should plan to use either CHTC’s web proxy, SQUID or
<code>transfer_input_files</code></strong>. Instead, you should plan to use the
large data filesystem Staging. Please contact a research computing facilitators at
[email protected] to determine the best option for your jobs.</p>

<p>More information is available at <a href="file-avail-squid.html">File Availability with Squid Web Proxy</a>
and <a href="file-avail-largedata.html">Managing Large Data in HTC Jobs</a>.</p>
<p>More information is available at <a href="file-avail-largedata.html">Managing Large Data in HTC Jobs</a>.</p>

<h2 id="5-create-a-job-executable">5. Create a Job Executable</h2>

Expand Down Expand Up @@ -459,9 +458,8 @@ <h2 id="6-submit-jobs">6. Submit Jobs</h2>
<li>Remember to transfer your Python script and the environment <code>tar.gz</code> file via
<code>transfer_input_files</code>.
<strong>Since the <code>tar.gz</code> file will almost certainly be larger than 100MB,
please <a href="mailto:[email protected]">email</a> us about different tools for
delivering the installation to your jobs,
likely our <a href="file-avail-squid.html">SQUID web proxy</a>.</strong></li>
please place it in your <code>/staging</code> directory and use the instructions in
(file-avail-largedata.html) to transfer the <code>tar.gz</code> file instead.</strong></li>
</ul>

<!--
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ <h1 id="overview">Overview</h1>

<p>Typically, software in CHTC jobs is installed or compiled locally by
individual users and then brought along to each job, either using the
default file transfer or our SQUID web server. However, another option
default file transfer or the staging system. However, another option
is to use a <em>container</em> system, where the software is installed in a
<em>container image</em>. Using a container to handle software can be
advantageous if the software installation 1) has many dependencies, 2)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -492,7 +492,7 @@ <h4 id="1-compile-mpi-code">1. Compile MPI Code</h4>
<p>If your MPI program is especially large (more than 100 MB, compiled), or
if it can <em>only</em> run from the exact location to which it was installed,
you may also need to take advantage of CHTC's shared software location
or our public web proxy called Squid. Email CHTC's Research Computing
or large data staging system. Email CHTC's Research Computing
Facilitators at <a href="mailto:[email protected]">[email protected]</a> if this is the case.</p>

<p><span name="script"></span></p>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -452,9 +452,9 @@ <h2 id="save-installed-packages-for-later-jobs">Save Installed Packages For Late
submit server (specifically where ever you were located when you submitted
the interactive build job). A copy of <code>packages.tar.gz</code> will be present. <strong>Be
sure to check the size of the project tarball before proceeding to subsequent job
submissions.</strong> If the file is &gt;100MB please contact us at <a href="mailto:[email protected]">[email protected]</a> so
that we can get you setup with access to our SQUID web proxy. More details
are available on our SQUID guide: <a href="file-avail-squid.html">File Availability with SQUID</a></p>
submissions.</strong> If the file is &gt;100MB please use your staging directory instead.
More details are available in our large data staging guide:
<a href="file-avail-largedata.html">Use Large Input and Output Files Via Staging</a></p>

<pre class="term"><code>[alice@submit]$ ls
build.sub julia-#.#.#-linux-x86_64.tar.gz julia-build.log
Expand Down Expand Up @@ -552,8 +552,8 @@ <h2 id="create-htcondor-submit-file">Create HTCondor Submit File</h2>
<pre><code class="language-{.sub}">transfer_input_files = julia-#.#.#-linux-x86_64.tar.gz, script.jl, my-project.tar.gz
</code></pre>

<p>For project tar.gz files that are larger than 100MB, email a facilitator about
using SQUID.</p>
<p>For project tar.gz files that are larger than 100MB, use your <code>/staging</code> directory.
For more information, see our <a href="file-avail-largedata.html">guide on staging large data</a>.</p>

<p>Modify the CPU/memory request lines to match what is needed by the job.
Test a few jobs for disk space/memory usage in order to make sure your
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,9 @@ <h1 class="uw-mini-bar mb-4">
Building R in HTC
</h1>

<h1 id="overview">Overview</h1>
<p><strong>This guide is deprecated. Please follow the instructions <a href="software-overview-htc.html#r">here</a> instead.</strong></p>

<h1 id="overview">Overview</h1>

<p><strong>This guide will provide instructions for compiling base R from source code and is intended
for compiling versions of R that are currently not provided via the CHTC Squid Web Proxy</strong></p>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,7 @@ <h1 id="2-job-qualifications"><strong>2. Job Qualifications</strong></h1>
<li>
<p><strong>Moderate Data Sizes</strong>: We can support input file sizes of up to
20 GB per file per job. This covers input files that would normally be
transferred out of a <code>/home</code> directory or use SQUID, in addition to larger
transferred out of a <code>/home</code> directory or use staging, in addition to larger
files up to 20GB. Outputs per job can be of similar sizes. If your input or
output files are larger than 1GB, or you have any other questions about
handling data on resources beyond CHTC, please <a href="mailto:[email protected]">contact us</a>!</p>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,7 @@ <h2 id="overview">Overview</h2>

<p>Typically, software in CHTC jobs is installed or compiled locally by
individual users and then brought along to each job, either using the
default file transfer or our SQUID web server. However, another option
default file transfer or our staging system. However, another option
is to use a <em>container</em> system, where the software is installed in a
<em>container image</em>. CHTC (and the OS Pool) have capabilities to access and
start containers and run jobs inside them. One container option
Expand Down

0 comments on commit 35c0197

Please sign in to comment.