Merge pull request #174 from ChristinaLK/docker-update

Docker update

documentation/htc_workloads/using_software/containers-docker.md

@@ -23,6 +23,9 @@ Docker image from the Docker Hub.
 
 ## Building Your Own Docker Image
 
+If you already have an existing Docker container image, skip
+to [Preparing Docker Containers for HTCondor Jobs](#preparing-docker-containers-for-htcondor-jobs). Otherwise, continue reading. 
+
 ### Identify Components
 
 What software do you want to install? Make sure that you have either the source 
@@ -85,7 +88,6 @@ is `alice` and I created an image with the NCBI `blast` tool, I might use this n
 
     $ docker build -t alice/NCBI-blast .
 
-
 #### Editing an Image Interactively
 
 You can also build an image interactively, without a Dockerfile. First, get 
@@ -121,7 +123,23 @@ You can also use the session's hash as found in the command prompt (`740b9db736a
 in the above example) in place of the docker session name. 
 
 
-## Converting from Docker to Apptainer/Singularity SIF format
+## Preparing Docker Containers for HTCondor Jobs
+
+Once you have a Docker container image, whether created by you or found 
+on DockerHub, you should convert it to the "sif" image format for 
+the best experience on the OSpool. 
+
+### Convert Docker containers on Docker Hub or online
+
+If the Docker container you want to use is online, on a site like Docker Hub, you can 
+log in to your Access Point and run a single command to convert it to a `.sif` image: 
+
+	$ apptainer build my-container.sif docker://owner/repository:tag
+
+Where the path at the end of the command is customized to be the container image
+you want to use. 
+
+### Convert Docker containers on your computer
 
 If you have built a Docker image on your own host, you can save it as a 
 tar file and then convert it to an Apptainer/Singularity SIF image. First
@@ -140,66 +158,11 @@ Apptainer to convert it to a SIF image:
 
     $ apptainer build my-container.sif docker-archive://my-container.tar
 
-You may now use the image in your job as described in the
-[Apptainer/Singularity Guide](../containers-singularity)
-
-
-## Syncronize from Docker Hub to /cvmfs (deprecated)
-
-OSG provide a mechanism for synchronizing images from Docker Hub to
-the global /cvmfs filesystem. The method is still available, but the
-preferred approach is now to save/import the image as desribed in the
-previous section.
-
-### Register Container with the OSG CVMFS Repository
-
-When you have found an image on Docker Hub,
-it needs to be submitted to the OSG Singularity repository
-(`/cvmfs/singularity.opensciencegrid.org/`), which also hosts the
-OSG-provided default images.
-
-To get your images included, please create a git pull request with the
-container identifier in `docker_images.txt` in the
-[cvmfs-singularity-sync repository](https://github.com/opensciencegrid/cvmfs-singularity-sync),
-or contact
-[[email protected]](mailto:[email protected])
-and we can help you.
-
-Once your submission has been accepted, it will be automatically
-converted to a Singularity image and pushed to the OSG Singularity
-repository (see further above). Note: some common Dockerfile features,
-like ENV and ENTRYPOINT, are ignored when the Docker image is converted
-to a Singularity image.
-
-Once your container has been added to CVMFS, if you update your original
-Docker image, new versions pushed to Docker Hub will automatically be
-detected and the version on the OSG (in the CVMFS filesystem) will be
-updated accordingly.
-
-### Using /cvmfs based Images in Jobs
-
-Once your Docker image is pushed to the OSG Singularity repository and converted to a Singularity/Apptainer image, a synchronized copy of the Singularity/Apptainer image will be distributed by CVMFS and available under
-`/cvmfs/singularity.opensciencegrid.org/` which is cached and available
-to the execution nodes.
-
-To run a job with a Docker image, use the `+SingularityImage` to
-specify the image the job should be using. Example:
-
-    +SingularityImage = "/cvmfs/singularity.opensciencegrid.org/htc/rocky:9"
-
-    <other usual submit file lines>
-    queue
-
-Another example would be if your Docker Hub username is `alice` and you
-created a container called `ncbi-blast`, and tag `v1`, added to the OSG
-Singularity repository, your submit file will include:
-
-    +SingularityImage = "/cvmfs/singularity.opensciencegrid.org/alice/ncbi-blast:v1"
-
-    <other usual submit file lines>
-    queue
-
+## Using Containers in HTCondor Jobs
 
+After converting the Docker image to a sif format, you can use the 
+image in your job as described in the
+[Apptainer/Singularity Guide](../containers-singularity#using-singularity-or-apptainer-images-in-an-htcondor-job). 
 
 ## Special Cases
 

documentation/htc_workloads/using_software/containers-singularity.md

@@ -10,7 +10,30 @@ in the Open Science Pool. You can use your own custom container to run
 jobs in the Open Science Pool. This guide describes how to create your
 own Apptainer/Singularity container "image" (the blueprint for the container).
 
-## Identify Components
+## Do You Need to Build a Container? 
+
+If there is an existing Docker container or Apptainer/Singularity container with 
+the software you need, you can proceed with using these options to submit a job. 
+* [See OSPool-provided containers here](../available-containers-list/)
+* [Using an existing Docker container](../containers-docker/)
+* [Using an existing Apptainer/Singularity container](#using-singularity-or-apptainer-images-in-an-htcondor-job)
+
+If you can't find a good option among existing containers, you may need to 
+build your own. See [this section of the guide](#building-your-own-apptainer-singularity-container) for more information. 
+
+## OSG-Provided Apptainer/Singularity Images
+
+The OSG Team maintains a set of images that are already in the OSG
+Apptainer/Singularity repository. [A list of ready-to-use containers can be found on this page](../available-containers-list/).
+
+If the software you need isn't already supported in a listed container,
+you can create your own container or use any container image in Docker Hub.
+
+How to explore these containers is shown [below](#exploring-apptainer-singularity-images-on-the-access-points). 
+
+## Building Your Own Apptainer/Singularity Container
+
+### Identify Components
 
 What software do you want to install? Make sure that you have either the source
 code or a command that can be used to install it through Linux (like `apt-get` or
@@ -22,15 +45,7 @@ as your starting point. See the available containers on Docker Hub here:
 [OSG Docker Containers](https://hub.docker.com/u/opensciencegrid)
 The best candidates for you will be containers that have "osgvo" in the name.
 
-## OSG-Provided Apptainer/Singularity Images
-
-The OSG Team maintains a set of images that are already in the OSG
-Apptainer/Singularity repository. [A list of ready-to-use containers can be found on this page](../available-containers-list/).
-
-If the software you need isn't already supported in a listed container,
-you can create your own container or any container image in Docker Hub.
-
-## Apptainer/Singularity Build
+### Apptainer/Singularity Build
 
 If you are building an image for the first time, the temporary cache directory of the apptainer image needs to be defined. The following commands define the cache location of the apptainer image to be built. Please run the commands in the terminal of your access point.
 
@@ -128,8 +143,7 @@ Once your build spec is ready, you can "build" the container image by running th
 Once the image is built, test it on an OSG-managed access point,
 and use it in your HTCondor jobs.
 
-
-### Exploring Apptainer/Singularity Images on the Access Points
+## Exploring Apptainer/Singularity Images on the Access Points
 
 Just like it is important to test your codes and jobs at a small scale,
 you should make sure that your Apptainer/Singularity container is working correctly before using it in jobs. One way
@@ -142,31 +156,42 @@ to how containers are started for jobs, is:
 
     apptainer shell my-container.sif
 
+If you want to test an existing container produced by OSG Staff, use the 
+full path provided in [this guide](../available-containers-list/). 
+
 This example will give you an interactive shell. You can explore the
 container and test your code with your own inputs from your `/home`
 directory, which is automatically mounted (but note - $HOME will not be
 available to your jobs later). Once you are down exploring, exit the
-container by running `exit` or with `CTRL+D`
+container by running `exit` or with `CTRL+D`. 
 
+## Using Singularity or Apptainer Images in an HTCondor Job
 
-## Pre-existing .sif Apptainer/Singularity Images
+Once you have a ".sif" container image file with all your needed software, 
+you can use this file as part of an HTCondor job. 
+
+### Upload the Container Image to the OSDF
 
 The image will be resused for
 each job, and thus the preferred transfer method is [OSDF](../../managing_data/osdf/).
-Store the .sif file under the "protected" area on your access point
-(see table [here](../../managing_data/overview/)), and then use the OSDF
+Store the .sif file under your personal data area on your access point
+(see table [here](../../managing_data/osdf/#where-to-put-your-files)). 
+
+### Use the Container in an HTCondor Job
+
+Once the image is placed in your OSDF space, you can use an OSDF 
 url directly in the `+SingularityImage` attribute. Note that you can not
 use shell variable expansion in the submit file - be sure to replace the
 username with your actual OSPool username. Example:
 
-    +SingularityImage = "osdf:///ospool/protected/USERNAME/my-custom-image-v1.sif"
+    +SingularityImage = "osdf:///ospool/apXX/data/USERNAME/my-custom-image-v1.sif"
 
     <other usual submit file lines>
     queue
 
-Be aware that OSDF aggressively caches the image based on file naming.
+**Be aware that OSDF aggressively caches the image based on file naming.
 If you need to do quick changes, please use versioning of the .sif file
-so that the caches see a "new" name. In this example, replacing
+so that the caches see a "new" name.** In this example, replacing
 `my-custom-image-v1.sif` with new content will probably mean that some
 nodes get the old version and some nodes the new version. Prevent this
 by creating a new file named with v2.