Skip to content

Commit

Permalink
Feature/doc update1 (#134)
Browse files Browse the repository at this point in the history
* Updated documentation to reflect new features
  • Loading branch information
garyluu authored and agduncan94 committed Oct 30, 2017
1 parent 6b8113d commit 0868dd2
Show file tree
Hide file tree
Showing 23 changed files with 84 additions and 69 deletions.
8 changes: 6 additions & 2 deletions src/app/container/add-tag/add-tag.component.ts
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,12 @@ export class AddTagComponent implements OnInit, AfterViewChecked {
this.unsavedVersion.cwl_path = this.tool.default_cwl_path;
this.unsavedVersion.wdl_path = this.tool.default_wdl_path;
this.unsavedVersion.dockerfile_path = this.tool.default_dockerfile_path;
this.unsavedTestCWLFile = this.tool.defaultCWLTestParameterFile;
this.unsavedTestWDLFile = this.tool.defaultWDLTestParameterFile;
if (this.tool.defaultCWLTestParameterFile) {
this.unsavedTestCWLFile = this.tool.defaultCWLTestParameterFile;
}
if (this.tool.defaultWDLTestParameterFile) {
this.unsavedTestWDLFile = this.tool.defaultWDLTestParameterFile;
}
}
});
}
Expand Down
2 changes: 1 addition & 1 deletion src/app/docs/docs.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ export class DocsService {
'Getting Started with Dockstore'),
new Doc('docker_registries',
`This page talks about the various Docker registries supported by Dockstore,
and the best practices for using these registries.`,
and the best practices for using these registries`,
'Docker Registries') ,
new Doc('public_private_tools',
'This page talks about the difference between public and private Dockstore tools.',
Expand Down
2 changes: 1 addition & 1 deletion src/app/docs/main/main.component.html
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ <h3>Introduction</h3>
datasets.
We also provide functionality to register, display, and share workflows.</p>

<p>If you are new to Dockstore, please continue to read or start with our <a routerLink="/onboarding" routerLinkActive="active">tutorial</a></p>
<p>If you are new to Dockstore, please continue to read or start with our <a routerLink="/onboarding" routerLinkActive="active">tutorial</a>.</p>

<h3>Table of Contents</h3>
<ol>
Expand Down
2 changes: 1 addition & 1 deletion src/assets/docs/about.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Dockstore also attempts to work with new and alternative languages/standards suc

## Built with Quay.io and GitHub

Docker repositories, like [DockerHub](https://hub.docker.com/), [Quay.io](https://quay.io/) and [GitLab](https://gitlab.com), and source control repositories like [GitHub](http://github.com), [BitBucket](https://bitbucket.org/), and [GitLab](https://gitlab.com), provide much of the infrastructure we need. They allow users build, publish, and share both public and private Docker images. However, the services lack standardized ways of describing how to invoke tools contained within Docker containers. The CWL standard has defined a way to define the inputs, parameterizations, and outputs of tools using a YAML-formatted file. Together, these resources provide the necessary tools to share analytical tools in a highly portable way, a key concern for the scientific community.
Docker repositories, like [Docker Hub](https://hub.docker.com/), [Quay.io](https://quay.io/) and [GitLab](https://gitlab.com), and source control repositories like [GitHub](http://github.com), [Bitbucket](https://bitbucket.org/), and [GitLab](https://gitlab.com), provide much of the infrastructure we need. They allow users build, publish, and share both public and private Docker images. However, the services lack standardized ways of describing how to invoke tools contained within Docker containers. The CWL standard has defined a way to define the inputs, parameterizations, and outputs of tools using a YAML-formatted file. Together, these resources provide the necessary tools to share analytical tools in a highly portable way, a key concern for the scientific community.

![Overview](../assets/images/docs/dockstore_logos.png)

Expand Down
8 changes: 4 additions & 4 deletions src/assets/docs/advanced-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

As a convenience, the dockstore command-line can perform file provisioning for inputs and outputs.

File provisioning for some protocols like http and ftp is built in while other protocols are handled by plugins as documented [here](https://github.com/ga4gh/dockstore/tree/develop/dockstore-file-plugin-parent).
File provisioning for some protocols like HTTP and FTP is built in while other protocols are handled by plugins as documented [here](https://github.com/ga4gh/dockstore/tree/develop/dockstore-file-plugin-parent).

To illustrate, for this [tool](https://dockstore.org/containers/quay.io/collaboratory/dockstore-tool-bamstats) we provide a couple of parameter files that can be used to parameterize a run of bamstats.

Expand All @@ -23,7 +23,7 @@ In the following JSON file, this file indicates for a CWL run that the input fil
}
```

The Dockstore command-line allows you to specify that the input file can be at a http(s) location, an ftp location, an AWS S3 location, a [synapse id](http://python-docs.synapse.org/#accessing-data), or an [ICGC storage id](http://docs.icgc.org/cloud/guide/#cloud-guide) in place of that path. For example the following indicates that the input file will be downloaded under http.
The Dockstore command-line allows you to specify that the input file can be at a HTTP(S) location, an FTP location, an AWS S3 location, a [synapse id](http://python-docs.synapse.org/#accessing-data), or an [ICGC storage id](http://docs.icgc.org/cloud/guide/#cloud-guide) in place of that path. For example the following indicates that the input file will be downloaded under HTTP.

```
{
Expand Down Expand Up @@ -85,9 +85,9 @@ By default, Dockstore will attempt to download files up to three times. Control

## Running CWL-runner with extra tags

When running a CWL tool, you may want to add additional parameters/flags to the cwl-runner command. You can do this by updating your dockstore config file (~/.dockstore/config).
When running a CWL tool, you may want to add additional parameters/flags to the cwl-runner command. You can do this by updating your dockstore config file (`~/.dockstore/config`).

As an example, adding the following line to your config file will stop cwl-runner from cleaning up, make it run in debug mode, and set the outdir to '/new/outputdir'
As an example, adding the following line to your config file will stop cwl-runner from cleaning up, make it run in debug mode, and set the outdir to `/new/outputdir`

```
cwltool-extra-parameters: --debug, --leave-container, --leave-tmpdir, --outdir /new/outputdir
Expand Down
2 changes: 1 addition & 1 deletion src/assets/docs/blog.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# News and Events

## October X, 2017 - Dockstore 1.3.0
## October 30, 2017 - Dockstore 1.3.0

We've just released the next major version of Dockstore!

Expand Down
2 changes: 1 addition & 1 deletion src/assets/docs/docker_registries.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Docker Registries

There are a number of alternative Docker Registries aside from Docker Hub and Quay.io. These registries are a combination of public registries and private registries.
There are a number of alternative Docker Registries aside from Quay.io, Docker Hub and GitLab. These registries are a combination of public registries and private registries.

## Public vs Private Docker Registries

Expand Down
29 changes: 14 additions & 15 deletions src/assets/docs/getting-started-with-cwl.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,19 +152,19 @@ outputs:
doc: "A zip file that contains the HTML report and various graphics."
```

Finally, the outputs section defines the output files. In this case it says in the current working directory there will be a file called `bamstats_report.zip`. When running this tool with CWL tools the file will be copied out of the container to a location you specify in your parameter JSON file. We'll walk though an example in the next section.
Finally, the outputs section defines the output files. In this case, it says in the current working directory there will be a file called `bamstats_report.zip`. When running this tool with CWL tools the file will be copied out of the container to a location you specify in your parameter JSON file. We'll walk though an example in the next section.

Finally, the `baseCommand` is the actual command that will be executed, in this case it's the wrapper script I wrote for bamstats.
Finally, the `baseCommand` is the actual command that will be executed. In this case, it's the wrapper script I wrote for bamstats.

```
baseCommand: ["bash", "/usr/local/bin/bamstats"]
```

The [CWL standard](http://www.commonwl.org/) is continuing to evolve and hopefully we will see new features, like support for [EDAM ontology](http://edamontology.org/page) terms, in future releases. In the mean time the [Gitter chat](https://gitter.im/common-workflow-language/common-workflow-language) is an active community to help drive the development of CWL in positive directions and we recommend tool authors make their voices heard.
The [CWL standard](http://www.commonwl.org/) is continuing to evolve and hopefully we will see new features, like support for [EDAM ontology](http://edamontology.org/page) terms, in future releases. In the mean time, the [Gitter chat](https://gitter.im/common-workflow-language/common-workflow-language) is an active community to help drive the development of CWL in positive directions and we recommend tool authors make their voices heard.

## Testing Locally

So at this point you've created a Docker-based tool and have described how to call that tool using CWL. Let's test running the BAMStats using the Dockstore command line and descriptor rather than just directly calling it via Docker. This will test that the CWL correctly describes how to run your tool.
So at this point, you've created a Docker-based tool and have described how to call that tool using CWL. Let's test running the BAMStats using the Dockstore command line and descriptor rather than just directly calling it via Docker. This will test that the CWL correctly describes how to run your tool.

First thing I'll do is create a completely local dataset and JSON parameterization file:

Expand Down Expand Up @@ -265,13 +265,13 @@ cwltool --non-strict --enable-net --outdir /home/ubuntu/gitroot/dockstore-tool-b
**Tip:** the `dockstore` CLI automatically create a `datastore` directory in the current working directory where you execute the command and uses it for inputs/outputs. It can get quite large depending on the tool/inputs/outputs being used. Plan accordingly e.g. execute the dockstore CLI in a directory located on a partition with sufficient storage.

## Adding a Test Parameter File
We are able register the above input parameterization of the tool into Dockstore so that users can see and test an example with our tool. Users can manually add test parameter files for a given tool tag or workflow version through both the command line and the versions tab in the UI.
We are able to register the above input parameterization of the tool into Dockstore so that users can see and test an example with our tool. Users can manually add test parameter files for a given tool tag or workflow version through both the command line and the versions tab in the UI.

**Tip:** Make sure that any required input files are given as publically accessible URLs so that a user can run the example successfully.

## Releasing on GitHub

At this point we've successfully created our tool in Docker, tested it, written a CWL that describes how to run it, and tested running this via the Dockstore command line. All of this work has been done locally, so if we encounter problems along the way its fast to perform debug cycles, fixing problems as we go. At this point we're confident that the tool is ready to share with others and bug free. It's time to release `1.25-6_1.1`
At this point, we've successfully created our tool in Docker, tested it, written a CWL that describes how to run it, and tested running this via the Dockstore command line. All of this work has been done locally, so if we encounter problems along the way its fast to perform debug cycles, fixing problems as we go. At this point, we're confident that the tool is ready to share with others and bug free. It's time to release `1.25-6_1.1`

Releasing will tag your GitHub repository with a version tag so you always can get back to this particular release. I'm going to use the tag `1.25-6_1.1` which you can see referenced in my Docker image tag and also my CWL file. Note that if you're following the tutorial using a forked version of the bamstats repo, your organization name should be different. GitHub makes it very easy to release:

Expand All @@ -283,15 +283,15 @@ I click on "releases" in my forked version of the GitHub project [page](https://

# Building on Quay.io

Now that you've perfected the `Dockerfile`, have built the image on your local host, and have tested running the Docker container and tool packaged inside and have released this version on GitHub, it's time to push the image to a place where others can use it. For this you can use DockerHub or GitLab but we prefer [Quay.io](http://quay.io) since it integrates really nicely with Dockstore.
Now that you've perfected the `Dockerfile`, have built the image on your local host, have tested running the Docker container and tool packaged inside, and have released this version on GitHub, it's time to push the image to a place where others can use it. For this you can use Docker Hub or GitLab but we prefer [Quay.io](http://quay.io) since it integrates really nicely with Dockstore.

You can manually `docker push` the image you have already built but the most reliable and transparent thing you can do is link your GitHub repository (and the Dockerfile contained within) to Quay. This will cause Quay to automatically build the Docker image every time there is a change.
You can manually `docker push` the image you have already built but the most reliable and transparent thing you can do is link your GitHub repository (and the Dockerfile contained within) to Quay.io. This will cause Quay to automatically build the Docker image every time there is a change.

Log onto Quay now and setup a new repository (click the "+" icon).
Log onto Quay.io now and setup a new repository (click the "+" icon).

![New Quay Repo](../assets/images/docs/quay_new_repo.png)

For your sanity, you should match the name to what you were using previously, so in this case it's my username then the same repo name as in GitHub `denis-yuen/dockstore-tool-bamstats`. Also, Dockstore will only work with `Public` repositories currently. Notice I'm selecting "Link to a GitHub Repository Push", this is because we want Quay to automatically build our Docker image every time we update the repository on GitHub. Very slick!
For your sanity, you should match the name to what you were using previously. So in this case, it's my username then the same repo name as in GitHub `denis-yuen/dockstore-tool-bamstats`. Also, Dockstore will only work with `Public` repositories currently. Notice I'm selecting "Link to a GitHub Repository Push", this is because we want Quay to automatically build our Docker image every time we update the repository on GitHub. Very slick!

![Build Trigger](../assets/images/docs/build_all.png)

Expand All @@ -305,14 +305,13 @@ So every time you do a commit to your GitHub repo Quay automatically builds and

It will also ask you where your Dockerfile is located and where your build context is (normally the root).

At this point you can confirm your settings and "Create Trigger" followed by "Run Trigger Now" to actually perform the initial build of the Docker images. You'll need to click on the little gear icon next to your build trigger to accomplish this.
At this point, you can confirm your settings and "Create Trigger" followed by "Run Trigger Now" to actually perform the initial build of the Docker images. You'll need to click on the little gear icon next to your build trigger to accomplish this.

![Manual Trigger](../assets/images/docs/manual_trigger.png)

Manually trigger it with a version name of `1.25-6_1.1` for this tutorial. Normally, I let the build trigger build a new tag for each new release on GitHub. "latest" on Quay.io is built any time I check-in on any branch. This can be useful for development but is discouraged in favour of a tagged version number for formal releases of your tool.

Build it for `1.25-6_1.1` for this tutorial. Typically, I build for each release and develop aka latest is built next time I check-in on that branch.

In my example I should see a `1.25-6_1.1` listed for this Quay Docker repository:
In my example, I should see a `1.25-6_1.1` listed for this Quay.io Docker repository:

![Build Tags](../assets/images/docs/build_tags.png)

Expand All @@ -321,7 +320,7 @@ And I do, so this Docker image has been built successfully by Quay and is ready

## Describe Your Tool in WDL

It is also possible to describe tools via the [WDL language](https://github.com/broadinstitute/wdl). A tool can either be described in CWL-only or can be described with both WDL and CWL.
It is also possible to describe tools via the [WDL language](https://github.com/openwdl/wdl). A tool can either be described in CWL-only or can be described with both WDL and CWL.

In WDL, a tool can also be described as a one task WDL workflow.

Expand Down
Loading

0 comments on commit 0868dd2

Please sign in to comment.