-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
10 changed files
with
221 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| Archiving Data on Kronos | ||
| ======================== | ||
|
|
||
| `Kronos <https://docs.olcf.ornl.gov/data/#kronos-nearline-archival-storage-system>`_ | ||
| is the mass storage system at OLCF. Each user has a directory of the form: | ||
|
|
||
| .. code:: bash | ||
| /nl/kronos/olcf/<projectID>/users/<userID> | ||
| and data can be transferred there using standard Unix commands. | ||
|
|
||
| .. note:: | ||
|
|
||
| You need to be logged into ``dtn.olcf.ornl.gov`` to access kronos. It is | ||
| not visible directly from Frontier or Andes. | ||
|
|
||
| A submission / shell script pair that automates the transfer of data is available in | ||
| `workflow/job_scripts/hpss <https://github.com/AMReX-Astro/workflow/tree/main/job_scripts/hpss>`_ as: | ||
|
|
||
| * ``olcf_kronos.submit`` : the slurm submission script | ||
| * ``kronos_process.sh`` : a BASH script that finds output and automates the archiving. | ||
|
|
||
| You submit the job from the directory containing the plotfiles you wish to archive. | ||
| It will then: | ||
|
|
||
| * tar up the diagnostic files, inputs, and other metadata into a file with the | ||
| date-stamp in the file name and copy that to kronos | ||
|
|
||
| * find all of the plotfiles and tar them directly to kronos. If the tar is successful, | ||
| it will move the plotfile into a ``plotfiles/`` subdirectory and add a ``.processed`` | ||
| file so the script knows it was archived already. | ||
|
|
||
| * find the checkpoint files matching a pattern (currently defaults to every 5000 steps) | ||
| and archive those in the same fashion, moving them to a ``checkfiles/`` subdirectory | ||
| once archived. | ||
|
|
||
| * loop, looking for new output files | ||
|
|
||
| By default, it will not transfer the last file, in case it is actively being written to. | ||
|
|
||
| .. tip:: | ||
|
|
||
| The ``olcf_kronos_once.submit`` can be used to just transfer without the loop | ||
| waiting for new files. | ||
|
|
||
|
|
||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,4 +9,5 @@ Working at OLCF | |
| olcf-compilers | ||
| olcf-workflow | ||
| olcf-jupyter | ||
| olcf-kronos | ||
| olcf-andes | ||
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
|
|
||
|
|
||
| <!DOCTYPE html> | ||
| <html class="writer-html5" lang="en" data-content_root="./"> | ||
| <head> | ||
| <meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" /> | ||
|
|
||
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> | ||
| <title>Archiving Data on Kronos — AMReX-Astro 1.0 documentation</title> | ||
| <link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" /> | ||
| <link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" /> | ||
| <link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" /> | ||
| <link rel="stylesheet" type="text/css" href="_static/theme_overrides.css?v=5694489b" /> | ||
|
|
||
|
|
||
| <script src="_static/jquery.js?v=5d32c60e"></script> | ||
| <script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script> | ||
| <script src="_static/documentation_options.js?v=f2a433a1"></script> | ||
| <script src="_static/doctools.js?v=9bcbadda"></script> | ||
| <script src="_static/sphinx_highlight.js?v=dc90522c"></script> | ||
| <script src="_static/clipboard.min.js?v=a7894cd8"></script> | ||
| <script src="_static/copybutton.js?v=f281be69"></script> | ||
| <script src="_static/js/theme.js"></script> | ||
| <link rel="index" title="Index" href="genindex.html" /> | ||
| <link rel="search" title="Search" href="search.html" /> | ||
| <link rel="next" title="Batch Visualization on Andes" href="olcf-andes.html" /> | ||
| <link rel="prev" title="Running Jupyter Remotely from OLCF" href="olcf-jupyter.html" /> | ||
| </head> | ||
|
|
||
| <body class="wy-body-for-nav"> | ||
| <div class="wy-grid-for-nav"> | ||
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> | ||
| <div class="wy-side-scroll"> | ||
| <div class="wy-side-nav-search" > | ||
|
|
||
|
|
||
|
|
||
| <a href="index.html" class="icon icon-home"> | ||
| AMReX-Astro | ||
| </a> | ||
| <div role="search"> | ||
| <form id="rtd-search-form" class="wy-form" action="search.html" method="get"> | ||
| <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" /> | ||
| <input type="hidden" name="check_keywords" value="yes" /> | ||
| <input type="hidden" name="area" value="default" /> | ||
| </form> | ||
| </div> | ||
| </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu"> | ||
| <p class="caption" role="heading"><span class="caption-text">AMReX Astro basics</span></p> | ||
| <ul class="current"> | ||
| <li class="toctree-l1"><a class="reference internal" href="alcf.html">Working at ALCF</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="nersc.html">Working at NERSC</a></li> | ||
| <li class="toctree-l1 current"><a class="reference internal" href="olcf.html">Working at OLCF</a><ul class="current"> | ||
| <li class="toctree-l2"><a class="reference internal" href="olcf-compilers.html">Compiling at OLCF</a></li> | ||
| <li class="toctree-l2"><a class="reference internal" href="olcf-workflow.html">Managing Jobs at OLCF</a></li> | ||
| <li class="toctree-l2"><a class="reference internal" href="olcf-jupyter.html">Running Jupyter Remotely from OLCF</a></li> | ||
| <li class="toctree-l2 current"><a class="current reference internal" href="#">Archiving Data on Kronos</a></li> | ||
| <li class="toctree-l2"><a class="reference internal" href="olcf-andes.html">Batch Visualization on Andes</a></li> | ||
| </ul> | ||
| </li> | ||
| <li class="toctree-l1"><a class="reference internal" href="iacs.html">Working at IACS</a></li> | ||
| <li class="toctree-l1"><a class="reference internal" href="workstations.html">Linux Workstations</a></li> | ||
| </ul> | ||
|
|
||
| </div> | ||
| </div> | ||
| </nav> | ||
|
|
||
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" > | ||
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> | ||
| <a href="index.html">AMReX-Astro</a> | ||
| </nav> | ||
|
|
||
| <div class="wy-nav-content"> | ||
| <div class="rst-content"> | ||
| <div role="navigation" aria-label="Page navigation"> | ||
| <ul class="wy-breadcrumbs"> | ||
| <li><a href="index.html" class="icon icon-home" aria-label="Home"></a></li> | ||
| <li class="breadcrumb-item"><a href="olcf.html">Working at OLCF</a></li> | ||
| <li class="breadcrumb-item active">Archiving Data on Kronos</li> | ||
| <li class="wy-breadcrumbs-aside"> | ||
| <a href="_sources/olcf-kronos.rst.txt" rel="nofollow"> View page source</a> | ||
| </li> | ||
| </ul> | ||
| <hr/> | ||
| </div> | ||
| <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> | ||
| <div itemprop="articleBody"> | ||
|
|
||
| <section id="archiving-data-on-kronos"> | ||
| <h1>Archiving Data on Kronos<a class="headerlink" href="#archiving-data-on-kronos" title="Link to this heading"></a></h1> | ||
| <p><a class="reference external" href="https://docs.olcf.ornl.gov/data/#kronos-nearline-archival-storage-system">Kronos</a> | ||
| is the mass storage system at OLCF. Each user has a directory of the form:</p> | ||
| <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>/nl/kronos/olcf/<projectID>/users/<userID> | ||
| </pre></div> | ||
| </div> | ||
| <p>and data can be transferred there using standard Unix commands.</p> | ||
| <div class="admonition note"> | ||
| <p class="admonition-title">Note</p> | ||
| <p>You need to be logged into <code class="docutils literal notranslate"><span class="pre">dtn.olcf.ornl.gov</span></code> to access kronos. It is | ||
| not visible directly from Frontier or Andes.</p> | ||
| </div> | ||
| <p>A submission / shell script pair that automates the transfer of data is available in | ||
| <a class="reference external" href="https://github.com/AMReX-Astro/workflow/tree/main/job_scripts/hpss">workflow/job_scripts/hpss</a> as:</p> | ||
| <ul class="simple"> | ||
| <li><p><code class="docutils literal notranslate"><span class="pre">olcf_kronos.submit</span></code> : the slurm submission script</p></li> | ||
| <li><p><code class="docutils literal notranslate"><span class="pre">kronos_process.sh</span></code> : a BASH script that finds output and automates the archiving.</p></li> | ||
| </ul> | ||
| <p>You submit the job from the directory containing the plotfiles you wish to archive. | ||
| It will then:</p> | ||
| <ul class="simple"> | ||
| <li><p>tar up the diagnostic files, inputs, and other metadata into a file with the | ||
| date-stamp in the file name and copy that to kronos</p></li> | ||
| <li><p>find all of the plotfiles and tar them directly to kronos. If the tar is successful, | ||
| it will move the plotfile into a <code class="docutils literal notranslate"><span class="pre">plotfiles/</span></code> subdirectory and add a <code class="docutils literal notranslate"><span class="pre">.processed</span></code> | ||
| file so the script knows it was archived already.</p></li> | ||
| <li><p>find the checkpoint files matching a pattern (currently defaults to every 5000 steps) | ||
| and archive those in the same fashion, moving them to a <code class="docutils literal notranslate"><span class="pre">checkfiles/</span></code> subdirectory | ||
| once archived.</p></li> | ||
| <li><p>loop, looking for new output files</p></li> | ||
| </ul> | ||
| <p>By default, it will not transfer the last file, in case it is actively being written to.</p> | ||
| <div class="admonition tip"> | ||
| <p class="admonition-title">Tip</p> | ||
| <p>The <code class="docutils literal notranslate"><span class="pre">olcf_kronos_once.submit</span></code> can be used to just transfer without the loop | ||
| waiting for new files.</p> | ||
| </div> | ||
| </section> | ||
|
|
||
|
|
||
| </div> | ||
| </div> | ||
| <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer"> | ||
| <a href="olcf-jupyter.html" class="btn btn-neutral float-left" title="Running Jupyter Remotely from OLCF" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a> | ||
| <a href="olcf-andes.html" class="btn btn-neutral float-right" title="Batch Visualization on Andes" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a> | ||
| </div> | ||
|
|
||
| <hr/> | ||
|
|
||
| <div role="contentinfo"> | ||
| <p>© Copyright 2018-2024, AMReX-Astro development tem.</p> | ||
| </div> | ||
|
|
||
| Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a | ||
| <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a> | ||
| provided by <a href="https://readthedocs.org">Read the Docs</a>. | ||
|
|
||
|
|
||
| </footer> | ||
| </div> | ||
| </div> | ||
| </section> | ||
| </div> | ||
| <script> | ||
| jQuery(function () { | ||
| SphinxRtdTheme.Navigation.enable(true); | ||
| }); | ||
| </script> | ||
|
|
||
| </body> | ||
| </html> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.