README

FASTX-Toolkit
=============

*******************************************************************
*                                                                 *
* FASTX TOOLKIT is unmaintained software.                         *
* No new features have been added since 2010.                     *
*                                                                 *
* There are many better alternatives for low-level FASTQ/FASTA    *
* manipulation. Use at your own risk.                             *
*                                                                 *
*******************************************************************


Short Summary
===============

The FASTX-Toolkit is a collection of command line tools for Short-Reads 
FASTA/FASTQ files preprocessing.



More Details
==============

Next-Generation sequencing machines usually produce FASTA or FASTQ files, 
containing multiple short-reads sequences (possibly with quality information).

The main processing of such FASTA/FASTQ files is mapping (aka aligning)
the sequences to reference genomes or other databases using specialized
programs. 

Example of such mapping programs are:
Blat (http://www.kentinformatics.com/index.asp), 
SHRiMP (http://compbio.cs.toronto.edu/shrimp),
LastZ (http://www.bx.psu.edu/miller_lab),
MAQ (http://maq.sourceforge.net/)
And many many others.

However, 
It is sometimes more productive to preprocess the FASTA/FASTQ files before 
mapping the sequences to the genome - manipulating the sequences to 
produce better mapping results.

The FASTX-Toolkit tools perform some of these preprocessing tasks.



Available Tools
===============

FASTQ-to-FASTA - Converts a FASTQ file to a FASTA file..

FASTQ-Statistics - scans a FASTQ file, and produces some statistics about the 
	quality and the sequences in the file.
	
FASTQ-Quality-BoxPlot, and
FASTQ-Nucleotides-Distribution - Generates charts based on the statistics 
	generated by FASTQ-Statistics. These charts can be used to quickly
	see the quality of the sequenced library.
	
FASTQ-Quality-Converter - Converts from ASCII to numeric quality scores.

FASTQ-Quality-Filter - removes low-quality sequences from FASTQ files.

FASTX-Artifacts-Filter - removes some sequencing artifacts from FASTA/Q files.

FASTX-Barcode-Splitter - A common practice is to sequence multiple biological
	samples in the same library (marking each sample using a dedicated 
	barcode). The resulting FASTA/Q file contains intermixed sequences 
	from those samples. This tool separates FASTA/Q files into several 
	individual files, based on the barcodes.
	
FASTX-Clipper - Adapters (aka Linkers) are added to the library (before 
	sequencing), and should be removed from the resulting FASTA/Q file.
	This tool removes (clips) adapters.
	
FASTA-Clipping-Histogram - After clipping a FASTA file, this tool generates a
	chart showing the length of the clipped sequences.
	
FASTX-Reverse-Complement - Produces a reverse-complement of FASTA/Q file.
	If a FASTQ file is given, the quality scores are also reversed.
	
FASTX-Trimmer - Extract sub-seqeunces from FASTA/Q file. Two examples are:
	Removing barcodes from the 5'-end of all sequences in a FASTQ file;
	Cutting 7 nucleotides from the 3'-end of all sequences in a FASTA file.



Galaxy
======

Galaxy (https://usegalaxy.org) is web-based framework for computational biology.

While the programs in the FASTX-Toolkit are command-line based, the package 
include the necessary files to integrate the tools into a Galaxy server,
Allowing users to execute this tools from their web-browser.

If you run your own local mirror of a Galaxy server, you can integrate the
FASTX-Toolkit into your Galaxy server.



Software Requirements
=====================

1. GCC is required to compile most tools.

2. FASTA-Clipping-Histogram tool requires Perl, the "PerlIO::gzip",
   "GD::Graph::bars" modules.
   
   Installing the perl modules can be accomplised by running:

   $ sudo cpan 'PerlIO::gzip'
   $ sudo cpan 'GD::Graph::bars'
   
3. FASTX-Barcode-Splitter requires the GNU Sed program.
   
4. FASTQ-Quality-Boxplot and FASTQ-Nucleotides-Distribution requires the
   'gnuplot' program.


Installation
============

When downloading the git repository from github, use the following:

   $ git clone https://github.com/agordon/fastx_toolkit
   $ cd fastx_toolkit
   $ ./reconf
   $ ./configure
   $ make

When downloading a released version archive:

   $ wget https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2
   $ tar -xjvf fastx_toolkit-0.0.14.tar.bz2
   $ cd fastx_toolkit-0.0.14
   $ ./configure
   $ make

The available releases are here:
   https://github.com/agordon/fastx_toolkit/releases

To install the tools, run (as root):

  $ sudo make install

This will install the tools into /usr/local/bin.
To install the tools to a different location, change the 'configure' step to:

  $ ./configure --prefix=/DESTINATION/DIRECTORY


The libgtextutils package is required to build fastx-toolkit,
see https://github.com/agordon/libgtextutils/ .


Command Line Usage
==================

Most tools support "-h" argument to show a short help screen.
Better documentation is not available at this moment.
Some more details and examples are available in the <help> section
of the XML tool files (in the 'galaxy' subdirectory).
  
 
Galaxy Installation
===================

Galaxy Installation should be done manually, and requires technical
understading of the Galaxy framework.

1. build and install the command line tools (as described above).

2. Make backup of your galaxy installation (better safe than sorry).

3. Run the 'install_galaxy_files.sh' script, 
   and specify the galaxy root directory.
   This script copies the files from the 'galaxy' sub-directory into
   your galaxy mirror directory.
   
4. Manually add the content of ./galaxy/fastx_toolkit_conf.xml file,
   into your Galaxy's tool_conf.xml
   
5. Edit [YOUR-GALAXY]/tool-data/fastx_clipper_sequences.txt file,
   And add your custom adapters/linkers.
   
6. Modify the "fastx_barcode_splitter_galaxy_wrapper.sh" as explained
   Below (see section "Special configuration for Barcode-Splitter").

7. Restart Galaxy.

Always make backup of your galaxy server files before trying to install 
the FASTX-Toolkit. 



Galaxy Testing
==============

The following tools support Galaxy's functional testing:
(Run from Galaxy's main directory)
  $ sh run_functional_tests.sh -id cshl_fastq_qual_conv
  $ sh run_functional_tests.sh -id cshl_fastq_to_fasta
  $ sh run_functional_tests.sh -id cshl_fastq_qual_stat
  $ sh run_functional_tests.sh -id cshl_fastx_trimmer
  $ sh run_functional_tests.sh -id cshl_fastx_reverse_complement
  $ sh run_functional_tests.sh -id cshl_fastx_artifacts_filter
  $ sh run_functional_tests.sh -id cshl_fasta_collapser
  $ sh run_functional_tests.sh -id cshl_fastx_clipper
 

Special configuration for Barcode-Splitter
==========================================

When running the barcode-splitter tool from the command line you specify a 
prefix direcotry - the output files will be written to that directory (similar
to GNU's split program usage).

Running the barcode-splittter inside galaxy requires a special hack beacuse
(I don't know how to|Galaxy can't) create a variable number of output datasets.
The number of required output files is determined by the tool only AFTER reading 
the barcodes description file.

The Galaxy-version of Barcode-Splitter works like this:
1. A FASTA/FASTQ file, and a Barcode description file are fed to the tool.
2. The tool produces a single output dataset (inside galaxy). This output
   is an HTML file, containing links to the split FASTA files.
3. Users can use the links to get the split FASTA files.
   (Since Galaxy's 'upload data' tool accepts URLs, this is not a real problem).
   
4. As the galaxy administrator, you'll have to edit 
   'fastx_barcode_splitter_galaxy_wrapper.sh' script and change BASEPATH and 
   PUBLICURL to point to a publicly accesibly path on your server.
   
Example:

fastx_barcode_splitter_galaxy_wrapper.sh contains:

   BASEPATH="/media/sdb1/galaxy/barcode_splits/"
   PUBLICURL="http://tango.cshl.edu/barcode_splits/"

When a user runs the barcode splitter tool, the FASTA files will be generated in 
"/media/sdb1/galaxy/barcode_splits/".  
The URL "http://tango.cshl.edu/barcode_splits" is set (in an apache server) to
serve files from "/media/sdb1/galaxy/barcode_splits/", with the following 
configuration:

    Alias /barcode_splits "/media/sdb1/galaxy/barcode_splits/"
    <Directory "/media/sdb1/galaxy/barcode_splits/">
        AllowOverride None
        Order allow,deny
        Allow from all
    </Directory>




Licenses
========

FASTX-Toolkit is distributed under the Affero GPL version 3 or later (AGPLv3),

EXCEPT

All files under the 'galaxy' sub-directory are distributed under the
same license as Galaxy itself (which is an MIT-style license).


While IANAL, these licenses basically mean that:
1. You're free to use FASTX-toolkit,

2. You're free to integrate FASTX-toolkit in your Galaxy mirror server 
   (or any other server).
   
3. You're free to modify the files under 'galaxy',
   without making your modifications public.
   
4. If you modify the FASTX-toolkit tools, and make those modifications 
   publicly available (either as downloadable tools, part of another product),
   or as a web-based server - you must make the modified source code freely 
   available (free as in speech).
   
See the COPYING file for the full Affero GPL.
See the GALAXY-LICENSE file for galaxy's license.

Please remember: 
  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.


=============
Please send all comments, suggestions, bug reports (or better yet - bug fixes)
to assafgordon@gmail.com .