Merge pull request #96 from databio/dev

Release for v0.9

README.md

@@ -3,5 +3,6 @@
 # Pypiper
 
 [![Documentation Status](https://readthedocs.org/projects/pypiper/badge/?version=latest)](http://pypiper.readthedocs.org/en/latest/?badge=latest)
+[![Build Status](https://travis-ci.org/databio/pypiper.svg?branch=master)](https://travis-ci.org/databio/pypiper)
 
 A lightweight python toolkit for gluing together restartable, robust command line pipelines. The best place to learn more is at the [documentation hosted at Read the Docs](http://pypiper.readthedocs.org/).

doc/source/changelog.rst

@@ -1,6 +1,19 @@
 Changelog
 ******************************
 
+- **v0.9.0** (*2018-11-19*):
+
+    - Use ``psutil`` to track aggregate memory usage for processes that spawn
+    children. This results in accurate memory records for these processes.
+
+    - Individual commands in a string of commands connected by shell pipes are
+    now treated as individual commands, and and monitored individually for
+    time and memory, and if a single component, fails, the entire string will
+    fail. Previously, only the final return command was recorded, as in ``bash``.
+
+    - Various other small improvements (like waiting checking for dynamic recover
+    flags)
+
 
 - **v0.8.1** (*2018-09-20*):
 

doc/source/contributing.rst

@@ -0,0 +1,10 @@
+
+Contributing
+=========================
+
+We welcome contributions in the form of pull requests.
+
+If proposing changes to package source code, please run the test suite in `python2` and `python3` by running ``pytest`` or ``python setup.py test`` from within the repository root.
+
+If using ``pytest`` directly, we suggest first activating the appropriate Python version's virtual environment and running ``pip install --ugprade ./``.
+Otherwise, simply specify the appropriate Python version, i.e. ``python2 setup.py test`` or ``python3 setup.py test``.

doc/source/develop-arguments.rst

@@ -51,7 +51,7 @@ Individual arguments that are understood and used by pypiper:
 - ``-C, --config``: Pypiper pipeline config yaml file.
 
 Individual arguments just provided for convenience and standardization:
-
+- ``-S, --sample-name``: name of the sample
 - ``-I, --input``: primary input file (e.g. read1)
 - ``-I2, --input2``: secondary input file (e.g. read2)
 - ``-O, --output-parent``: parent folder for pipeline results (the pipeline will use this as the parent directory for a folder named ``sample-name``)
@@ -100,4 +100,4 @@ Examples
     parser = pypiper.add_pypiper_args(parser, args=["genome"])
 
     # add some groups and some individual arguments
-    parser = pypiper.add_pypiper_args(parser, args=["genome"], groups=["looper", "ngs"])
+    parser = pypiper.add_pypiper_args(parser, args=["genome"], groups=["looper", "ngs"])

doc/source/functions.rst

@@ -2,7 +2,7 @@
 Basic functions
 =========================
 
-Pypiper is simple, but powerful. You only need 3 functions to get started. ``PipelineManager`` can do:
+Pypiper is simple, but powerful. You really only need to know about 3 functions to get started. ``PipelineManager`` can do:
 
 .. currentmodule:: pypiper.manager.PipelineManager
 .. autosummary:: 
@@ -11,7 +11,7 @@ Pypiper is simple, but powerful. You only need 3 functions to get started. ``Pip
 	stop_pipeline
 
 
-With that you can create a simple pipeline. You can click on each function to view the in-depth documentation for that function. There are quite a few optional parameters to the ``run`` function, which is where most of Pypiper's power comes from
+With those 3 functions, you can create a simple pipeline. Click on each function to view its in-depth documentation. There are quite a few optional parameters to the ``run`` function, which is where most of Pypiper's power comes from.
 
 When you've mastered the basics and are ready to get more powerful, add in a few new (optional) commands that make debugging and development easier:
 

doc/source/hello-world.rst

@@ -3,7 +3,7 @@
 |logo| Installing and Hello World
 ==============================
 
-Release versions are hosted on  `pypi (under the name piper) <https://pypi.org/project/piper/>`_, and posted on the GitHub `releases page <https://github.com/epigen/pypiper/releases>`_. Install directly using pip:
+Release versions are hosted on  `pypi (under the name piper) <https://pypi.org/project/piper/>`_, and posted on the GitHub `releases page <https://github.com/databio/pypiper/releases>`_. Install directly using pip:
 
 .. code-block:: bash
 
@@ -24,7 +24,7 @@ Now, to test pypiper, follow the commands in the ``Hello, Pypiper!`` tutorial: j
 	pip install --user piper
 
 	# download hello_pypiper.py
-	wget https://raw.githubusercontent.com/epigen/pypiper/master/example_pipelines/hello_pypiper.py
+	wget https://raw.githubusercontent.com/databio/pypiper/master/example_pipelines/hello_pypiper.py
 	
 	# Run it:
 	python hello_pypiper.py

doc/source/index.rst

@@ -54,7 +54,8 @@ Contents
    faq.rst
    changelog.rst
    support.rst
-   GitHub <http://github.com/epigen/pypiper>
+   contributing.rst
+   GitHub <http://github.com/databio/pypiper>
 
 
 

doc/source/ngstk.rst

@@ -2,7 +2,7 @@
 NGSTk
 =========================
 
-An optional feature of pypiper is the accompanying toolkits, such as the next-gen sequencing toolkit, `NGSTk <https://github.com/epigen/pypiper/blob/master/pypiper/ngstk.py>`_, which simply provides some convenient helper functions to create common commands, like converting from file formats (*e.g.* bam to fastq), merging files (*e.g.* merge_bams), counting reads, etc. These make it faster to design bioinformatics pipelines in Pypiper, but are entirely optional.
+An optional feature of pypiper is the accompanying toolkits, such as the next-gen sequencing toolkit, `NGSTk <https://github.com/databio/pypiper/blob/master/pypiper/ngstk.py>`_, which simply provides some convenient helper functions to create common commands, like converting from file formats (*e.g.* bam to fastq), merging files (*e.g.* merge_bams), counting reads, etc. These make it faster to design bioinformatics pipelines in Pypiper, but are entirely optional.
 
 Example:
 
@@ -18,6 +18,6 @@ Example:
 	ngstk.index_bam("sample.bam")
 
 
-A list of available functions can be found in the :doc:`API <api>` or in the source code for `NGSTk <https://github.com/epigen/pypiper/blob/master/pypiper/ngstk.py>`_.
+A list of available functions can be found in the :doc:`API <api>` or in the source code for `NGSTk <https://github.com/databio/pypiper/blob/master/pypiper/ngstk.py>`_.
 
 Contributions of additional toolkits or functions in an existing toolkit are welcome.

doc/source/support.rst

@@ -2,10 +2,4 @@
 Support
 =========================
 
-Bug Reports
-*************
-If you find a bug or want request a feature, open an issue at https://github.com/epigen/pypiper/issues.
-
-Contributing
-*************
-We welcome contributions in the form of pull requests.
+If you find a bug or want request a feature, open an issue at https://github.com/databio/pypiper/issues.

doc/source/tutorials-advanced.rst

@@ -3,7 +3,7 @@ Advanced tutorial
 
 Here we have a more advanced bioinformatics pipeline that adds some new concepts. This is a simple script that takes an input file and returns the file size and the number of sequencing reads in that file. This example uses a function from from the built-in :doc:`NGSTk toolkit <ngstk>`. In particular, this toolkit contains a few handy functions that make it easy for a pipeline to accept inputs of various types. So, this pipeline can count the number of reads from files in ``BAM`` format, or ``fastq`` format, or ``fastq.gz`` format. You can also use the same functions from NGSTk to develop a pipeline to do more complicated things, and handle input of any of these types.
 
-First, grab this pipeline. Download `count_reads.py <https://github.com/epigen/pypiper/blob/master/example_pipelines/count_reads.py>`_, make it executable (``chmod 755 count_reads.py``), and then run it with ``./count_reads.py``). 
+First, grab this pipeline. Download `count_reads.py <https://github.com/databio/pypiper/blob/master/example_pipelines/count_reads.py>`_, make it executable (``chmod 755 count_reads.py``), and then run it with ``./count_reads.py``). 
 
 You can grab a few small data files in the `microtest repository <https://github.com/epigen/microtest/tree/master/config>`_. Run a few of these files like this:
 

doc/source/tutorials-basic.rst

@@ -1,7 +1,7 @@
 Basic tutorial
 *****************
 
-Now, download `basic.py <https://github.com/epigen/pypiper/blob/master/example_pipelines/basic.py>`_ and run it with `python basic.py` (or, better yet, make it executable (`chmod 755 basic.py`) and then run it directly with `./basic.py`). This example is a documented vignette; so just read it and run it to get an idea of how things work.
+Now, download `basic.py <https://github.com/databio/pypiper/blob/master/example_pipelines/basic.py>`_ and run it with `python basic.py` (or, better yet, make it executable (`chmod 755 basic.py`) and then run it directly with `./basic.py`). This example is a documented vignette; so just read it and run it to get an idea of how things work.
 
 .. literalinclude:: ../../example_pipelines/basic.py
 

pypiper/_version.py

@@ -1 +1 @@
-__version__ = "0.8.1"
+__version__ = "0.9.0"

pypiper/folder_context.py

@@ -0,0 +1,35 @@
+""" Context manager for temporarily changing folder. """
+
+import os
+
+
+__author__ = "Vince Reuter"
+__email__ = "[email protected]"
+
+
+
+class FolderContext(object):
+    """ Context manager for temporarily changing directory. """
+
+    def __init__(self, folder):
+        """
+        Store the previous working path to restore upon exit.
+
+        :param str folder: Path to set as new working directory
+        """
+        if not os.path.isdir(folder):
+            raise ValueError(
+                "Requested temp entry to non-folder: {}".format(folder))
+        self._prevdir = os.getcwd()
+        self._currdir = folder
+
+    def __enter__(self):
+        """ Make the working directory switch. """
+        os.chdir(self._currdir)
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        """ Switch back to the previous working directory. """
+        if not os.path.isdir(self._prevdir):
+            raise RuntimeError("Return path is no longer a directory: {}".
+                               format(self._prevdir))
+        os.chdir(self._prevdir)