bamql_queries.7

.\" Authors: Paul Boutros and Lab Members
.TH bamql_queries 7 "Dec 2014" "1.0" "MISCELLANEOUS"
BAMQL filters SAM or BAM files using a simple query language that is more expressive than the
.B view
options available in
.BR samtools (1).

The language consists of logical connectives, specifically and, or, not, and a ternary if, and predicates that match properties of the read. Expressions may be grouped using parentheses.

.SH LOGICAL CONNECTIVES
The logical connectives are presented in order from lowest precedence to highest precedence.

.SS VARIABLES
\fBlet\fR name1 \fB=\fR expr1 [\fB,\fR name2 \fB=\fR expr2 ...] \fBin\fR body_expr

Computes the value of \fIexprN\fR and assigns it to \fInameN\fR which can then be used in \fIbody_expr\fR. The name may contain only alphanumeric characters or underscore.

\fBbind\fR str_expr \fBusing /\fRregex\fB/\fR[\fBi\fR]\fB in\fR body_expr

Compute the value of \fIstr_expr\fR, which must be a string, and then matches it against \fIregex\fR. If \fBi\fR is included, the match will be case-insensitive. If the expression matches, \fIbody_expr\fR, which must be Boolean, will be evaluated. Any named subpatterns, as per
.BR pcrepattern (3),
are available as variables. If the patterns name ends in \fB_d\fR, \fB_i\fR, \fB_c\fR, it will be converted to a floating point value, an integer value, or the first character as an integer.

For example, \fBbind read_group using /C3BUK\\.(?<x_i>\\d+)/ in x_i > 2\fR will match any read group that starts with \fBC3BUK.\fR followed by a number greater than 2.

.SS LOOPS
\fBall\fR name \fB=\fR expr1 \fB,\fR expr2 [ \fB,\fR expr3 ...] \fBin\fR body_expr

Evaluates \fIbody_expr\fR with \fIname\fR set to each of \fIexpr1\fR, \fIexpr2\fR, and so on, and is satisfied if it is satisfied for every value.

\fBany\fR name \fB=\fR expr1 \fB,\fR expr2 [ \fB,\fR expr3 ...] \fBin\fR body_expr

Evaluates \fIbody_expr\fR with \fIname\fR set to each of \fIexpr1\fR, \fIexpr2\fR, and so on, and is satisfied if it is satisfied for some value.

.SS TERNARY IF
cond \fBthen\fR then_expression \fBelse\fR else_expression

If \fIcond\fR is satisfied, this expression will only be satisfied if \fIthen_expression\fR is satisfied. If \fIcond\fR is not satisfied, then this expression will only be satisfied if \fIelse_expression\fR is satisfied.

.SS DISJUNCTION (OR)
expr \fB|\fR expr

Is satisfied if at least one operand is satisfied.

.SS EXCLUSIVE DISJUNCTION (XOR)
expr \fB^\fR expr

Is satisfied if exactly one operand is satisfied, but not both.

.SS CONJUNCTION (AND)
expr \fB&\fR expr

Is satisfied only if both operands are satisfied.

.SS MATERIAL CONDITONAL (IMPLIES)
antecedent \fB->\fR assertion

Is satisfied if either both the \fIantecedent\fR and \fIassertion\fR are statisfied or the \fIantecedent\fR is not satisfied. Consider the following:

.B position(500) -> nt_exact(500, G)

This will reject any reads where position 500 is not \fBG\fR, but keep reads which either have a \fBG\fR at position 500 or do not overlap with position 500.

.SS COMPARISON AND MATCHING
expr \fB==\fR expr
.br
expr \fB!=\fR expr
.br
expr \fB<\fR expr
.br
expr \fB>\fR expr
.br
expr \fB<=\fR expr
.br
expr \fB=>\fR expr

Compares values in the usual way. Note that only values of the same type can be compared. For strings, lexicographical order is used for less/greater than comparisons.

expr \fB~ /\fRregex\fB/\fR[\fBi\fR]
.br
expr \fB: \fRglob

Does string matching usin a PCRE regular expression or a glob. See
.BR pcrepattern (3)
or
.BR glob (7)
for details. If \fBi\fR is included, the match will be case-insensitive.

haystack \fB\\\fR needle

Check that the integer \fIhaystack\fR contains all the bits set in \fIneedle\fR. This is equivalent to the C/Java/C#/Python/PERL expression \fB(haystack & needle) == needle\fR.

.SS NEGATION (NOT)
\fB!\fR expr

Is satisfied if \fIexpr\fR is not satisfied.

.SS LITERALS
\fB'\fRx

The integer value of character \fIx\fR.

Numeric floating point and integer values are also supported.

.SH PREDICATES
The predicates are grouped based on function.

.SS BAM FLAGS
These predicates match the sequences in the BAM file. For more details, see
.BR sam (5).

.B paired?

The read is paired in sequencing.

.B proper_pair?

The read is mapped in a proper pair; that is, the aligner thought the insert size implied by the paired-end reads seemed correct.

.B unmapped?

The query sequence itself is unmapped.

.B mate_unmapped?

The mate is unmapped.

.B mapped_to_reverse?

The read is mapped to the reverse strand.

.B mate_mapped_to_reverse?

The read's mate is mapped to the reverse strand.

\fBraw_flag(\fRnumber\fB)\fR

If numeric BAM flags have been burned into your brain, you can check them directly by specifying \fInumber\fR.

.B read1?

The read is the first read in a pair.

.B read2?

The read is the second read in a pair.

.B secondary?

The alignment is not primary.

.B failed_qc?

The read fails platform/vendor quality checks.

.B duplicate?

The read is either a PCR or an optical duplicate. That is, another read with the same sequence occurs at exactly the same position in the reference genome.

.B supplementary?

The alignment is supplementary.

.B flags

Returns the read flag as an integer.

.SS MAPPING INFORMATION

\fBbegin\fR

Returns the start position of the read on the chromosome, if it is mapped.

\fBchr(\fRglob\fB)\fR

Matches the chromosome name to which the read is mapped. Chromosome names should not start with \fBchr\fR, as that will be automatically checked. Moreover, some human chromosome have differing names, so both are checked. The known rules are:

X == 23
.br
Y == 24
.br
MT == M == 25


Also, case is ignored. Additionally, the chromosome is matched using wildcards from 
.BR glob (7).

\fBchr_name\fR

Returns the chromosome name (stripped of \fIchr\fR) as a string. This can be used with other comparisons, but lacks all the equivalent chromosome magic that \fBchr\fR provides.

\fBmapping_quality(\fRprobability\fB)\fR

Matches the read if the probability of error is less than \fIprobability\fR. The mapping quality is approximated in the SAM format, so this will be imperfect. For clarity, setting the probability to 0 will be so stringent as to reject all reads, while setting it to 1 will be so liberal as to accept all reads.

\fBend\fR

Returns the final position of the read on the chromosome, if it is mapped.

\fBinsert_reversed\fR

Satisfied if the insert size indicates that it is flipped relative to the reference. If the insert is in the direction of the reference or the insert size is not provided, this returns false.

\fBinsert_size\fR

Returns the size of the region between a pair of mapped reads if the aligner has been able to determine it.

\fBmate_chr(\fRglob\fB)\fR

This works identically to \fBchr\fR, but on the chromosome of the mate pair, if one exists. If the mate is unmapped, this returns false.

\fBmate_begin\fR

Returns the start position of the mate pair on the chromosome, if it is mapped. Note that \fBmate_end\fR, while logical, cannot exist because the necessary information is not available.

\fBmate_chr_name\fR

Returns the chromosome name of the mate pair, if one exists, (stripped of \fIchr\fR) as a string. This can be used with other comparisons, but lacks all the equivalent chromosome magic that \fBmate_chr\fR provides.

\fBsplit_pair?\fR

Checks if both the reads in a mate pair are mapped, but to different chromosomes.

.SS OTHER READ INFORMATION
\fBread_group\fR

Returns the read group.

\fBaux_dbl(\fRcode\fB)\fR
.br
\fBaux_int(\fRcode\fB)\fR
.br
\fBaux_str(\fRcode\fB)\fR

Gets a piece of auxiliary data, if specified in the input. The \fIcode\fR is the two symbol identifier for the auxiliary format. The result will be a float point number, integral number, or string for \fBaux_dbl\fR, \fBaux_int\fR, and \fBaux_str\fR, respectively. BAM files also have a character type which may be read as an integer.

.SS POSITION
All of the position operations are inclusive: that means they take any reads with nucleotides in the desired range. This means that the start or end of a read can extend beyond the desired positions. BAM files allow reads to have position information while still being marked as unmapped. This operations ignore the official mapping status, and work solely on the position information. If this is undesirable, combine with \fB& !unmapped?\fR. Occasionally, the aligner produces reads which have a position, but no detailed mapping information (\fIi.e.\fR, no CIGAR string). In this case, the end position of the read is assumed to be mapped with no insertions or deletions.

\fBafter(\fRposition\fB)\fR

Matches all sequences that cover the specified position or any higher position (more \fIq\fR-ward on the chromosome).

\fBbefore(\fRposition\fB)\fR

Matches all sequences that cover the specified position or any lower position (more \fIp\fR-ward on the chromosome).

\fBposition(\fRstart\fB,\fR end\fB)\fR

Matches all sequences that cover the range of position from \fIstart\fR to \fIend\fR.

.SS SEQUENCE
\fBnt(\fRposition\fB,\fR n\fB)\fR

Matches a read has nucleotide \fIn\fR at the provided position, relative to the chromosome. The nucleotide can be any IUPAC-style base (ACGTU, KMYR, BDHV, and N). The match is degenerate; that is, if the nucleotide specified is N, any base will match. It will reject unmapped reads and reads which do not contain the required position.

\fBnt_exact(\fRposition\fB,\fR n\fB)\fR

Matches a read has nucleotide \fIn\fR at the provided position, relative to the chromosome. The nucleotide can be any IUPAC-style base (ACGTU, KMYR, BDHV, and N). The match is exact; that is, if the nucleotide specified is N, the base in the read must be N too. It will reject unmapped reads and reads which do not contain the required position.

.SS MISCELLANEOUS

\fBbed(\fRfile\fB)\fR

Reads a BED-formatted file and creates an expression that is satisfied if the read interesects any of the segments in the file. Note that this must load the BED file into memory, so do not use large BED files.

\fBheader\fR

Returns the header string of the read.

\fBmax(\fRexpr1\fB, \fRexpr2\fB, \fR...\fB)\fR

Returns the largest value among the supplied arguments, which must be of the same type. If strings, this is the last string, lexicographically.

\fBmin(\fRexpr1\fB, \fRexpr2\fB, \fR...\fB)\fR

Returns the smallest value among the supplied arguments, which must be of the same type. If strings, this is the first string, lexicographically.

.B true

Always satisfied.

.B false

Never satisfied.

\fBrandom(\fRprobability\fB)\fR

This chooses a uniform pseudo-random variable and is satisfied with frequency \fIprobability\fR. This can be used to provide a random sub-sample of reads, keeping the proportion of total reads specified as the probability. The probability must be between 0 and 1 and can be specified using scientific notation. The random number chosen is selected using
.BR drand48 (3)
if one is inclined to care about such things.

.SH EXAMPLES

Match sequences on chromosome 7 which are from the read group labelled \fBRUN3\fR:

.B "chr(7) & read_group : RUN3"

Sub-sample mitochondrial reads and all the reads that have matched to chromosomes starting with \fBug\fR.

.B "chr(M) & random(0.2) | chr(ug*)"

.SH SEE ALSO
.BR bamql (1),
.BR bamql-compile (1),
.BR samtools (1),
.BR pcrepattern (3),
.BR glob (7),
.BR sam (5).