Add documentation #60
Conversation
almussel
commented
Nov 2, 2016
•
almussel
commented
- Documentation on installing toil-lib
- Documentation on developing on Toil and toil-lib (not done yet)
- Spark documentation moved over from Toil
There was a problem hiding this comment.
This is more cursory than I'd like. At the least, I think we should also give a general description of the various packages/modules (tools contains job functions for the various -omics tools we run, files contains helper functions for dealing with local files, blah blah). I'd also explain the philosophy behind why this project is split from toil-scripts. These don't need to be voluminous—someone who wants to understand how to use the methods in toil_lib.files can look at the inline docs—but there should be enough information so that someone who is new to toil_lib can navigate around the project quickly.
|
|
||
| :: | ||
|
|
||
| make develop |
There was a problem hiding this comment.
IIRC, toil-lib make develop will not install toil. I would note this and link back to the toil install docs.
There was a problem hiding this comment.
Yes, this is key. Further, how does one develop against both toil and toil-lib simultaneously? I assume you create one virtual env and checkout toil, build it, then checkout toil lib and similarly build it, and then you're good to go?
There was a problem hiding this comment.
I know my question may be stupid, but I doubt I'll be the only one who gets confused by this.
| ------------------ | ||
|
|
||
| For developers and people interested in building the project from source the | ||
| following explains how to setup virtualenv to create an environment to use toil-lib |
There was a problem hiding this comment.
Instead of this paragraph, I would say that our make flow requires a virtualenv to be installed, why that is, and then how to install virtualenv. virtualenv doesn't get installed by default with python, so I wouldn't assume that people know about it.
almussel
force-pushed
the
documentation
branch
from
50cb412
to
3e034b0
Compare
| if you are running on a preemptable cluster: in Spark and HDFS, failed workers can | ||
| be restarted, but a failed master will cause all workers in the cluster to | ||
| dissociate. If the cluster does not restart, this causes a deadlock. | ||
|
|
There was a problem hiding this comment.
This might be tricky, but it would be good to link or at least tell people to see the toil documentation on checkpoints:
http://toil.readthedocs.io/en/releases-3.3.x/developing.html#checkpoints
| instructions can be found [here](https://docs.docker.com/engine/installation/). | ||
| For examples of how to connect to a Toil-based Spark cluster, see the Spark | ||
| unit tests: | ||
|
|
There was a problem hiding this comment.
This is good, but do we need a separate section on Docker in general? Possibly this is for @fnothaft or @jvivian to do, but we should have a good documentation section on docker which describes how to run pipelines with docker and how to use the functions in: https://github.com/BD2KGenomics/toil-lib/blob/master/src/toil_lib/programs.py
?
There was a problem hiding this comment.
I'm in favor. I'm familiar with about half of the functions in toil_lib/programs.py, so I could get it started but would likely need some help.
|
|
||
| :: | ||
| export PYTHONPATH=${SPARK_HOME}/python:${SPARK_HOME}/python/lib/py4j-0.10.1-src.zip:${PYTHONPATH} | ||
|
|
There was a problem hiding this comment.
Should the "py4j-0.10-1-src.zip" be that specific? Is this not specific version?
There was a problem hiding this comment.
spark.rst is being moved over from the Toil docs. @fnothaft is there a reason for this specific link?
almussel
force-pushed
the
documentation
branch
from
3e034b0
to
b512c64
Compare
|
@benedictpaten I've addressed your comments. |
