Notebooks/01-Mapping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Table of Contents\n",
    "* [Iterative vs fragment-based mapping](#Iterative-vs-fragment-based-mapping)\n",
    "\t* [Advantages of iterative mapping](#Advantages-of-iterative-mapping)\n",
    "\t* [Advantages of fragment-based mapping](#Advantages-of-fragment-based-mapping)\n",
    "* [Mapping](#Mapping)\n",
    "\t* [Iterative mapping](#Iterative-mapping)\n",
    "\t* [Fragment-based mapping](#Fragment-based-mapping)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Iterative vs fragment-based mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "source": [
    "Iterative mapping first proposed by <a name=\"ref-1\"/>[(Imakaev et al., 2012)](#cite-Imakaev2012a), allows to map usually a high number of reads. However other methodologies, less \"brute-force\" can be used to take into account the chimeric nature of Hi-C reads.\n",
    "\n",
    "A simple alternative is to allow split mapping, just as with RNA-seq data.\n",
    "\n",
    "Another way consists in _pre-truncating_ <a name=\"ref-1\"/>[(Ay and Noble, 2015)](#cite-Ay2015a) reads that contain a ligation site and map only the longest part of the read <a name=\"ref-2\"/>[(Wingett et al., 2015)](#cite-Wingett2015).\n",
    "\n",
    "Finally, an intermediate approach, _fragment-based_, consists in mapping full length reads first, and than splitting unmapped reads at the ligation sites <a name=\"ref-1\"/>[(Serra, Ba\\`{u, Filion and Marti-Renom, 2016)](#cite-Serra2016)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Advantages of iterative mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "- It's the only solution when no restriction enzyme has been used (i.e. micro-C)\n",
    "- Can be faster when few windows (2 or 3) are used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Advantages of fragment-based mapping "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "- Generally faster\n",
    "- Safer: mapped reads are generally larger than 25-30 nm (the largest window used in iterative mapping). Less reads are mapped, but the difference is usually canceled or reversed when looking for \"valid-pairs\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "__ _Note:_ __ We use __GEM__ <a name=\"ref-1\"/>[(Marco-Sola, Sammeth, Guig\\'{o and Ribeca, 2012)](#cite-Marco-Sola2012), performance are very similar to Bowtie2, perhaps a bit better. \n",
    "\n",
    "_For now TADbit is only compatible with GEM._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from pytadbit.mapping.full_mapper import full_mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The full mapping function can be used to perform either iterative or fragment-based mapping, or a combination of both."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Iterative mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Here an example of use as iterative mapping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "r_enz = 'MboI'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "! mkdir -p results/iterativ/$r_enz\n",
    "! mkdir -p results/iterativ/$r_enz/01_mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing FASTQ file\n",
      "  - conversion to MAP format\n",
      "  - trimming reads 1-25\n",
      "Mapping reads in window 1-25...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_A67p5X\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_A67p5X -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_A67p5X_full_1-25\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_A67p5X\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_A67p5X_full_1-25.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-30\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_A67p5X_filt_1-25.map\n",
      "Mapping reads in window 1-30...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_hXqWFT\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_hXqWFT -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_hXqWFT_full_1-30\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_hXqWFT\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_hXqWFT_full_1-30.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-35\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_hXqWFT_filt_1-30.map\n",
      "Mapping reads in window 1-35...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_dUwVst\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_dUwVst -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_dUwVst_full_1-35\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_dUwVst\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_dUwVst_full_1-35.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-40\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_dUwVst_filt_1-35.map\n",
      "Mapping reads in window 1-40...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_Q_Vgmu\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_Q_Vgmu -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_Q_Vgmu_full_1-40\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_Q_Vgmu\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_Q_Vgmu_full_1-40.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-45\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_Q_Vgmu_filt_1-40.map\n",
      "Mapping reads in window 1-45...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_ORtV47\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_ORtV47 -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_ORtV47_full_1-45\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_ORtV47\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_ORtV47_full_1-45.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-50\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_ORtV47_filt_1-45.map\n",
      "Mapping reads in window 1-50...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_j3SUJW\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_j3SUJW -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_j3SUJW_full_1-50\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_j3SUJW\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_j3SUJW_full_1-50.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-55\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_j3SUJW_filt_1-50.map\n",
      "Mapping reads in window 1-55...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_qp6ptt\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_qp6ptt -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_qp6ptt_full_1-55\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_qp6ptt\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_qp6ptt_full_1-55.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-60\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_qp6ptt_filt_1-55.map\n",
      "Mapping reads in window 1-60...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_MBbXEq\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_MBbXEq -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_MBbXEq_full_1-60\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_MBbXEq\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_MBbXEq_full_1-60.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-65\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_MBbXEq_filt_1-60.map\n",
      "Mapping reads in window 1-65...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_bI4lHh\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_bI4lHh -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_bI4lHh_full_1-65\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_bI4lHh\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_bI4lHh_full_1-65.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-70\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_bI4lHh_filt_1-65.map\n",
      "Mapping reads in window 1-70...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_M9AUiy\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_M9AUiy -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_M9AUiy_full_1-70\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_M9AUiy\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_M9AUiy_full_1-70.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-75\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_M9AUiy_filt_1-70.map\n",
      "Mapping reads in window 1-75...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_1HR068\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_1HR068 -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_1HR068_full_1-75\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_1HR068\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_1HR068_full_1-75.map\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['results/iterativ/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-25.map',\n",
       " 'results/iterativ/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-30.map',\n",
       " 'results/iterativ/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-35.map',\n",
       " 'results/iterativ/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-40.map',\n",
       " 'results/iterativ/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-45.map',\n",
       " 'results/iterativ/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-50.map',\n",
       " 'results/iterativ/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-55.map',\n",
       " 'results/iterativ/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-60.map',\n",
       " 'results/iterativ/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-65.map',\n",
       " 'results/iterativ/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-70.map',\n",
       " 'results/iterativ/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-75.map']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for the first side of the reads\n",
    "full_mapping(gem_index_path='/media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem',\n",
    "             out_map_dir='results/iterativ/{0}/01_mapping/mapped_{0}_r1/'.format(r_enz),\n",
    "             fastq_path='/media/storage/FASTQs/K562_%s_1.fastq' % (r_enz),\n",
    "             r_enz='hindIII', frag_map=False, clean=True, nthreads=20,\n",
    "             windows=((1,25),(1,30),(1,35),(1,40),(1,45),(1,50),(1,55),(1,60),(1,65),(1,70),(1,75)), \n",
    "             temp_dir='results/iterativ/{0}/01_mapping/mapped_{0}_r1_tmp/'.format(r_enz))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "And for the second side of the read:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing FASTQ file\n",
      "  - conversion to MAP format\n",
      "  - trimming reads 1-25\n",
      "Mapping reads in window 1-25...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_F9N2ee\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_F9N2ee -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_F9N2ee_full_1-25\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_F9N2ee\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_F9N2ee_full_1-25.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-30\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_F9N2ee_filt_1-25.map\n",
      "Mapping reads in window 1-30...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_yGZnsy\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_yGZnsy -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_yGZnsy_full_1-30\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_yGZnsy\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_yGZnsy_full_1-30.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-35\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_yGZnsy_filt_1-30.map\n",
      "Mapping reads in window 1-35...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_KEYxTn\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_KEYxTn -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_KEYxTn_full_1-35\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_KEYxTn\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_KEYxTn_full_1-35.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-40\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_KEYxTn_filt_1-35.map\n",
      "Mapping reads in window 1-40...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_nk9UkN\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_nk9UkN -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_nk9UkN_full_1-40\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_nk9UkN\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_nk9UkN_full_1-40.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-45\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_nk9UkN_filt_1-40.map\n",
      "Mapping reads in window 1-45...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_bCMD1N\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_bCMD1N -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_bCMD1N_full_1-45\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_bCMD1N\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_bCMD1N_full_1-45.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-50\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_bCMD1N_filt_1-45.map\n",
      "Mapping reads in window 1-50...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_S2625S\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_S2625S -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_S2625S_full_1-50\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_S2625S\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_S2625S_full_1-50.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-55\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_S2625S_filt_1-50.map\n",
      "Mapping reads in window 1-55...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_b09ELu\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_b09ELu -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_b09ELu_full_1-55\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_b09ELu\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_b09ELu_full_1-55.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-60\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_b09ELu_filt_1-55.map\n",
      "Mapping reads in window 1-60...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_aTLSwi\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_aTLSwi -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_aTLSwi_full_1-60\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_aTLSwi\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_aTLSwi_full_1-60.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-65\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_aTLSwi_filt_1-60.map\n",
      "Mapping reads in window 1-65...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_3JeChj\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_3JeChj -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_3JeChj_full_1-65\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_3JeChj\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_3JeChj_full_1-65.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-70\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_3JeChj_filt_1-65.map\n",
      "Mapping reads in window 1-70...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_9gkcVR\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_9gkcVR -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_9gkcVR_full_1-70\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_9gkcVR\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_9gkcVR_full_1-70.map\n",
      "Preparing MAP file\n",
      "  - trimming reads 1-75\n",
      "   x removing original input /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_9gkcVR_filt_1-70.map\n",
      "Mapping reads in window 1-75...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_nigfxR\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_nigfxR -o /home/student/Course/3DAROC/Notebooks/results/iterativ/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_nigfxR_full_1-75\n"
     ]
    }
   ],
   "source": [
    "# for the second side of the reads\n",
    "full_mapping(gem_index_path='/media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem',\n",
    "             out_map_dir='results/iterativ/{0}/01_mapping/mapped_{0}_r2/'.format(r_enz),\n",
    "             fastq_path='/media/storage/FASTQs/K562_%s_2.fastq' % (r_enz),\n",
    "             r_enz=r_enz, frag_map=False,  clean=True, nthreads=20,\n",
    "             windows=((1,25),(1,30),(1,35),(1,40),(1,45),(1,50),(1,55),(1,60),(1,65),(1,70),(1,75)),\n",
    "             temp_dir='results/iterativ/{0}/01_mapping/mapped_{0}_r2_tmp/'.format(r_enz))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Fragment-based mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "With fragment based mapping it would be:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "! mkdir -p results/fragment/$r_enz\n",
    "! mkdir -p results/fragment/$r_enz/01_mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/student/.miniconda2/lib/python2.7/site-packages/pytadbit/mapping/full_mapper.py:390: UserWarning: WARNING: only 62 Gb left on tmp_dir: /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp\n",
      "\n",
      "  warn('WARNING: only %d Gb left on tmp_dir: %s\\n' % (fspace, temp_dir))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing FASTQ file\n",
      "  - conversion to MAP format\n",
      "Mapping reads in window 1-end...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_mUfCKk\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_mUfCKk -o /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_mUfCKk_full_1-end\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_mUfCKk\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_mUfCKk_full_1-end.map\n",
      "  - splitting into restriction enzyme (RE) fragments using ligation sites\n",
      "  - ligation sites are replaced by RE sites to match the reference genome\n",
      "    * enzyme: MboI, ligation site: GATCGATC, RE site: GATC\n",
      "Preparing MAP file\n",
      "   x removing pre-GEM input /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_mUfCKk_filt_1-end.map\n",
      "Mapping fragments of remaining reads...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_OIA1pF\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_OIA1pF -o /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_OIA1pF_frag_1-end\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_OIA1pF\n",
      "   x removing failed to map /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_mUfCKk_fail.map\n",
      "   x removing tmp mapped /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r1_tmp/K562_MboI_1_OIA1pF_frag_1-end.map\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['results/fragment/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_full_1-end.map',\n",
       " 'results/fragment/MboI/01_mapping/mapped_MboI_r1/K562_MboI_1_frag_1-end.map']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for the first side of the reads \n",
    "full_mapping(gem_index_path='/media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem',\n",
    "             out_map_dir='results/fragment/{0}/01_mapping/mapped_{0}_r1/'.format(r_enz),\n",
    "             fastq_path='/media/storage/FASTQs/K562_%s_1.fastq' % (r_enz),\n",
    "             r_enz=r_enz, frag_map=True, clean=True, nthreads=20, \n",
    "             temp_dir='results/fragment/{0}/01_mapping/mapped_{0}_r1_tmp/'.format(r_enz))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/student/.miniconda2/lib/python2.7/site-packages/pytadbit/mapping/full_mapper.py:390: UserWarning: WARNING: only 33 Gb left on tmp_dir: /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp\n",
      "\n",
      "  warn('WARNING: only %d Gb left on tmp_dir: %s\\n' % (fspace, temp_dir))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing FASTQ file\n",
      "  - conversion to MAP format\n",
      "Mapping reads in window 1-end...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_DPXqE5\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_DPXqE5 -o /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_DPXqE5_full_1-end\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_DPXqE5\n",
      "   x removing map /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_DPXqE5_full_1-end.map\n",
      "  - splitting into restriction enzyme (RE) fragments using ligation sites\n",
      "  - ligation sites are replaced by RE sites to match the reference genome\n",
      "    * enzyme: MboI, ligation site: GATCGATC, RE site: GATC\n",
      "Preparing MAP file\n",
      "   x removing pre-GEM input /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_DPXqE5_filt_1-end.map\n",
      "Mapping fragments of remaining reads...\n",
      "TO GEM /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_LywLiM\n",
      "/usr/local/bin/gem-mapper -I /media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem -q offset-33 -m 0.04 -s 0 --allow-incomplete-strata 0.00 --granularity 10000 --max-decoded-matches 1 --min-decoded-strata 0 --min-insert-size 0 --max-insert-size 0 --min-matched-bases 0.8 --gem-quality-threshold 26 --max-big-indel-length 15 --mismatch-alphabet ACGT -E 0.30 --max-extendable-matches 20 --max-extensions-per-match 1 -e 0.04 -T 20 -i /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_LywLiM -o /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_LywLiM_frag_1-end\n",
      "Parsing result...\n",
      "   x removing GEM input /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_LywLiM\n",
      "   x removing failed to map /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_DPXqE5_fail.map\n",
      "   x removing tmp mapped /home/student/Course/3DAROC/Notebooks/results/fragment/MboI/01_mapping/mapped_MboI_r2_tmp/K562_MboI_2_LywLiM_frag_1-end.map\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['results/fragment/MboI/01_mapping/mapped_MboI_r2/K562_MboI_2_full_1-end.map',\n",
       " 'results/fragment/MboI/01_mapping/mapped_MboI_r2/K562_MboI_2_frag_1-end.map']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for the second side of the reads\n",
    "full_mapping(gem_index_path='/media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem',\n",
    "             out_map_dir='results/fragment/{0}/01_mapping/mapped_{0}_r2/'.format(r_enz),\n",
    "             fastq_path='/media/storage/FASTQs/K562_%s_2.fastq' % (r_enz),\n",
    "             r_enz=r_enz, frag_map=True, clean=True, nthreads=20, \n",
    "             temp_dir='results/fragment/{0}/01_mapping/mapped_{0}_r2_tmp/'.format(r_enz))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "<!--bibtex\n",
    "@article{Imakaev2012a,\n",
    "abstract = {Extracting biologically meaningful information from chromosomal interactions obtained with genome-wide chromosome conformation capture (3C) analyses requires the elimination of systematic biases. We present a computational pipeline that integrates a strategy to map sequencing reads with a data-driven method for iterative correction of biases, yielding genome-wide maps of relative contact probabilities. We validate this ICE (iterative correction and eigenvector decomposition) technique on published data obtained by the high-throughput 3C method Hi-C, and we demonstrate that eigenvector decomposition of the obtained maps provides insights into local chromatin states, global patterns of chromosomal interactions, and the conserved organization of human and mouse chromosomes.},\n",
    "author = {Imakaev, Maxim V and Fudenberg, Geoffrey and McCord, Rachel Patton and Naumova, Natalia and Goloborodko, Anton and Lajoie, Bryan R and Dekker, Job and Mirny, Leonid A},\n",
    "doi = {10.1038/nmeth.2148},\n",
    "file = {:home/fransua/.local/share/data/Mendeley Ltd./Mendeley Desktop/Downloaded/Imakaev et al. - 2012 - Iterative correction of Hi-C data reveals hallmarks of chromosome organization.pdf:pdf},\n",
    "issn = {1548-7105},\n",
    "journal = {Nature methods},\n",
    "keywords = {Hi-C},\n",
    "mendeley-groups = {stats/Hi-C,Research articles},\n",
    "mendeley-tags = {Hi-C},\n",
    "month = {oct},\n",
    "number = {10},\n",
    "pages = {999--1003},\n",
    "pmid = {22941365},\n",
    "title = {{Iterative correction of Hi-C data reveals hallmarks of chromosome organization.}},\n",
    "url = {http://www.ncbi.nlm.nih.gov/pubmed/22941365},\n",
    "volume = {9},\n",
    "year = {2012}\n",
    "}\n",
    "@article{Ay2015a,\n",
    "author = {Ay, Ferhat and Noble, William Stafford},\n",
    "doi = {10.1186/s13059-015-0745-7},\n",
    "file = {:home/fransua/.local/share/data/Mendeley Ltd./Mendeley Desktop/Downloaded/Ay, Noble - 2015 - Analysis methods for studying the 3D architecture of the genome.pdf:pdf},\n",
    "issn = {1474-760X},\n",
    "journal = {Genome Biology},\n",
    "keywords = {Chromatin conformation capture,Genome architecture,Hi-C,Three-dimensional genome,Three-dimensional modeling,chromatin,conformation capture,genome architecture,in many other applications,ranging from genome assem-,review,three-dimensional genome,three-dimensional modeling},\n",
    "mendeley-groups = {Research articles},\n",
    "mendeley-tags = {Hi-C,review},\n",
    "number = {1},\n",
    "pages = {183},\n",
    "publisher = {Genome Biology},\n",
    "title = {{Analysis methods for studying the 3D architecture of the genome}},\n",
    "url = {http://genomebiology.com/2015/16/1/183},\n",
    "volume = {16},\n",
    "year = {2015}\n",
    "}\n",
    "@article{Wingett2015,\n",
    "abstract = {HiCUP is a pipeline for processing sequence data generated by Hi-C and Capture Hi-C (CHi-C) experiments, which are techniques used to investigate three-dimensional genomic organisation. The pipeline maps data to a specified reference genome and removes artefacts that would otherwise hinder subsequent analysis. HiCUP also produces an easy-to-interpret yet detailed quality control (QC) report that assists in refining experimental protocols for future studies. The software is freely available and has already been used for processing Hi-C and CHi-C data in several recently published peer-reviewed studies.},\n",
    "author = {Wingett, Steven and Ewels, Philip and Furlan-Magaril, Mayra and Nagano, Takashi and Schoenfelder, Stefan and Fraser, Peter and Andrews, Simon},\n",
    "doi = {10.12688/f1000research.7334.1},\n",
    "file = {:home/fransua/Downloads/f1000research-4-7903.pdf:pdf},\n",
    "issn = {2046-1402},\n",
    "journal = {F1000Research},\n",
    "mendeley-groups = {Computer programs/Hi-C/Hi-C processing},\n",
    "pages = {1310},\n",
    "pmid = {26835000},\n",
    "title = {{HiCUP: pipeline for mapping and processing Hi-C data.}},\n",
    "url = {http://f1000research.com/articles/4-1310/v1},\n",
    "volume = {4},\n",
    "year = {2015}\n",
    "}\n",
    "@article{Serra2016,\n",
    "abstract = {The sequence of a genome is insufficient to understand all genomic processes carried out in the cell nucleus. To achieve this, the knowledge of its three- dimensional architecture is necessary. Advances in genomic technologies and the development of new analytical methods, such as Chromosome Conformation Capture (3C) and its derivatives, now permit to investigate the spatial organization of genomes. However, inferring structures from raw contact data is a tedious process for shortage of available tools. Here we present TADbit, a computational framework to analyze and model the chromatin fiber in three dimensions. To illustrate the use of TADbit, we automatically modeled 50 genomic domains from the fly genome revealing differential structural features of the previously defined chromatin colors, establishing a link between the conformation of the genome and the local chromatin composition. More generally, TADbit allows to obtain three-dimensional models ready for visualization from 3C-based experiments and to characterize their relation to gene expression and epigenetic states. TADbit is open-source and available for download from http://www.3DGenomes.org.},\n",
    "author = {Serra, Fran{\\c{c}}ois and Ba{\\`{u}}, Davide and Filion, Guillaume and Marti-Renom, Marc A.},\n",
    "doi = {10.1101/036764},\n",
    "file = {:home/fransua/.local/share/data/Mendeley Ltd./Mendeley Desktop/Downloaded/Serra et al. - 2016 - Structural features of the fly chromatin colors revealed by automatic three-dimensional modeling.pdf:pdf},\n",
    "journal = {bioRxiv},\n",
    "keywords = {3d,Hi-C,capture,genome architecture,genome reconstruction chromosome conformation,modeling,optimization,resampling,restraint-based,three-dimensional genome reconstruction},\n",
    "mendeley-groups = {Research articles,projects/WIREs{\\_}review/modeling{\\_}tools,Computer programs/Hi-C/Hi-C modeling},\n",
    "mendeley-tags = {Hi-C,modeling,optimization,resampling},\n",
    "pages = {1--29},\n",
    "title = {{Structural features of the fly chromatin colors revealed by automatic three-dimensional modeling.}},\n",
    "url = {http://biorxiv.org/content/early/2016/01/15/036764},\n",
    "year = {2016}\n",
    "}\n",
    "@misc{Marco-Sola2012,\n",
    "abstract = {Because of ever-increasing throughput requirements of sequencing data, most existing short-read aligners have been designed to focus on speed at the expense of accuracy. The Genome Multitool (GEM) mapper can leverage string matching by filtration to search the alignment space more efficiently, simultaneously delivering precision (performing fully tunable exhaustive searches that return all existing matches, including gapped ones) and speed (being several times faster than comparable state-of-the-art tools).},\n",
    "author = {Marco-Sola, Santiago and Sammeth, Michael and Guig{\\'{o}}, Roderic and Ribeca, Paolo},\n",
    "booktitle = {Nature Methods},\n",
    "doi = {10.1038/nmeth.2221},\n",
    "isbn = {1548-7105 (Electronic)$\\backslash$r1548-7091 (Linking)},\n",
    "issn = {1548-7091},\n",
    "mendeley-groups = {Research articles},\n",
    "pmid = {23103880},\n",
    "title = {{The GEM mapper: fast, accurate and versatile alignment by filtration}},\n",
    "year = {2012}\n",
    "}\n",
    "\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# References\n",
    "\n",
    "<a name=\"cite-Imakaev2012a\"/><sup>[^](#ref-1) </sup>Imakaev, Maxim V and Fudenberg, Geoffrey and McCord, Rachel Patton and Naumova, Natalia and Goloborodko, Anton and Lajoie, Bryan R and Dekker, Job and Mirny, Leonid A. 2012. _Iterative correction of Hi-C data reveals hallmarks of chromosome organization._. [URL](http://www.ncbi.nlm.nih.gov/pubmed/22941365)\n",
    "\n",
    "<a name=\"cite-Ay2015a\"/><sup>[^](#ref-2) </sup>Ay, Ferhat and Noble, William Stafford. 2015. _Analysis methods for studying the 3D architecture of the genome_. [URL](http://genomebiology.com/2015/16/1/183)\n",
    "\n",
    "<a name=\"cite-Wingett2015\"/><sup>[^](#ref-3) </sup>Wingett, Steven and Ewels, Philip and Furlan-Magaril, Mayra and Nagano, Takashi and Schoenfelder, Stefan and Fraser, Peter and Andrews, Simon. 2015. _HiCUP: pipeline for mapping and processing Hi-C data._. [URL](http://f1000research.com/articles/4-1310/v1)\n",
    "\n",
    "<a name=\"cite-Serra2016\"/><sup>[^](#ref-4) </sup>Serra, Fran\\c{cois and Ba\\`{u, Davide and Filion, Guillaume and Marti-Renom, Marc A.. 2016. _Structural features of the fly chromatin colors revealed by automatic three-dimensional modeling._. [URL](http://biorxiv.org/content/early/2016/01/15/036764)\n",
    "\n",
    "<a name=\"cite-Marco-Sola2012\"/><sup>[^](#ref-5) </sup>Marco-Sola, Santiago and Sammeth, Michael and Guig\\'{o, Roderic and Ribeca, Paolo. 2012. _The GEM mapper: fast, accurate and versatile alignment by filtration_.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}