%\VignetteIndexEntry{1.7 Strategies for working with large data}

<<setup,echo=FALSE>>=
library(RNAseqData.HNRNPC.bam.chr14)
library(TxDb.Dmelanogaster.UCSC.dm3.ensGene)
library(TxDb.Hsapiens.UCSC.hg19.knownGene)
library(Rsamtools)
library(ShortRead)
library(parallel); options(mc.cores=detectCores())
library(lattice)
@ 

\chapter{Strategies for working with large data}
\label{chpt:large-data}

Bioinformatics data is now very large; it is not reasonable to expect
all of a fastq or bam file, for instance, to fit into memory. How is
this data to be processed? This challenge confronts us in whatever
language or tool we are using. In \R{} and \Bioconductor{}, main
approaches are to: (1) \emph{restrict} data input to the interesting
subset of the larger data set; (2) \emph{sample} from the large data,
knowing that an appropriately sized sample will accurately estimate
statistics we are interested in; (3) \emph{iterate} through large data
in chunks; and (4) use \emph{parallel} evaluation on one or several
computers. Let's look at each of these approaches.

\section{Restriction}

Just because a file contains a lot of data does not mean that we
are interested in all of it. In base \R, one might use the
\Rcode{colClasses} argument to \Rcode{read.delim} or similar function
(e.g., setting some elements to \code{NULL}) to read only some columns
of a large comma-separated value file. In addition to the obvious
benefit of using less memory than if all of the file had been read in,
input will be substantially faster because less computation needs to
be done to coerce values from their representation in the file to
their representation in \R's memory.

A variation on the idea of restricting data input is to organize the
data on disk into a representation that facilitates restriction. In
base \R, large data might be stored in a relational data base like the
\software{sqlite} data bases that are built in to \R{} and used in the
\Biocpkg{AnnotationDbi} \Bioconductor{} packages.
%% FIXME: simple relational database
In addition to facilitating restriction, these approaches are
typically faster than parsing a plain text file, because the data base
software has stored data in a way that efficiently transforms data
from its on-disk to in-memory representation.

Let's use BAM files and the \Biocpkg{Rsamtools} package to illustrate
restriction. Rather than using simple text files (``sam'' format), we
use ``bam'' (binary alignment) files that have been indexed. Use of a
binary format enhances data input speed, while the index facilitates
restrictions that take into account genomic coordinates. The basic
approach will use the \Rfunction{ScanBamParam} function to specify a
restriction.

A BAM record can contain a lot of information, including the
relatively large sequence, quality, and query name strings. Not all
information in the BAM file is needed for some calculations. For
instance, one could calculate coverage (number of nucleotides
overlapping each reference position) using only the \code{rname}
(reference sequence name), \code{pos}, and \code{cigar} fields. We
could arrange to input just this information with
<<restriction-what>>=
param <- ScanBamParam(what=c("rname", "pos", "cigar"))
@ 

Another common restriction is to particular genomic regions of
interest. Regions of interest might be known genes in an RNAseq
differential expression study, or promoter regions in a ChIP-seq
study. Restrictions in genome space are specified using
\Robject{GRanges} objects (typically computed from some reference
source, rather than entered by hand) provided as the \Rfunarg{which}
argument to \Rfunction{ScanBamParam}. Here we create a
\Robject{GRanges} instance representing all exons on \Dmel{}
chromosome 3L, and use that as a restriction to
\Rfunction{ScanBamParam}'s \Rfunarg{which} argument:
<<restriction-which>>=
library(TxDb.Dmelanogaster.UCSC.dm3.ensGene)
exByGn <- exonsBy(TxDb.Dmelanogaster.UCSC.dm3.ensGene, "gene")
seqlevels(exByGn, force=TRUE) <- "chr3L"
gns <- unlist(range(exByGn))
param <- ScanBamParam(which=gns)
@ 
%% 
The \Rfunarg{what} and \Rfunarg{which} restrictions can be combined
with other restrictions into a single \Robject{ScanBamParam} object
<<restrictions-several>>=
param <- ScanBamParam(what=c("rname", "pos", "cigar"), which=gns)
@ 
%% 
We can also restrict input to, e.g., paired-end reads that represent
the primary (best) alignment; see the help page \Rcode{?ScanBamParam}
for a more complete description.
 
Here are some BAM files from an experiment in \Dmel; the BAM files
contain a subset of aligned reads
<<bigdata>>=
bigdata <- "~/BigData"
bamfls <- dir(file.path(bigdata, "bam"), ".bam$", full=TRUE) #$
names(bamfls) <- sub("_subset.bam", "", basename(bamfls))
@ 
%% 
We'll read in the first BAM file, but restrict input to ``rname'' to
find how many reads map to each chromosome. We use the ``low-level''
function \Rfunction{scanBam} to input the data
<<scanBam-restriction>>=
param <- ScanBamParam(what="rname")
bam <- scanBam(bamfls[1], param=param)[[1]]
table(bam$rname)
@ 

Restriction is such a useful concept that many \Bioconductor{} high
throughput sequence analysis functions enable doing the right
thing. Functions such as \code{coverage,BamFile-method} use
restrictions to read in the specific data required for them to compute
the statistic of interest. Most ``higher level'' functions have a
\Rfunarg{param=ScanBamParam()} argument. For instance, the primary
user-friendly function for reading BAM files is
\Rfunction{readGappedAlignments} (this is renammed to
\Rfunction{readGAlignments} in \Bioconductor{} version 2.13) reads the
most useful information, allowing the user to specify additional
fields if desired.
<<readGappedAlignments-restriciton>>=
gal <- readGappedAlignments(bamfls[1])
head(gal, 3)
param <- ScanBamParam(what="seq")  ## also input sequence
gal <- readGappedAlignments(bamfls[1], param=param)
head(mcols(gal)$seq)
@ 

\section{Sampling}

\R{} is after all a statistical language, and it sometimes makes sense
to draw inferences from a sample of large data. For instance, many
quality assessment statistics summarize overall properties (e.g., GC
content or per-nucleotide base quality of FASTQ reads) that do not
require processing of the entire data. For these statistics to be
valid, the sample from the file needs to be a random sample, rather
than a sample of convenience. 

There are two advantages to sampling from a FASTQ (or BAM) file. The
sample uses less memory than the full data. And because less data
needs to be parsed from the on-disk to in-memory representation the
input is faster.

The \Biocpkg{ShortRead} package \Rfunction{FastqSampler} function (see also
\Rfunction{Rsamtools::BamSampler} visits a FASTQ file but retains only
a random sample from the file. Here is our function to calculate GC
content from a \Robject{DNAStringSet}:
<<GC-sampling>>=
library(Morgan2013)
gcFunction
@ 
%% 
and a subset of a FASTQ file with 1 million reads:
<<fastq-files>>=
bigdata <- "~/BigData"
fqfl <- dir(file.path(bigdata, "fastq"), ".fastq$", full=TRUE) #$
@
%% 
\Rfunction{FastqSampler} works by specifying the file name and desired
sample size, e.g., 100,000 reads. This creates an object from which an
independent sample can be drawn using the \code{yield} function. 
<<fastq-sampling>>=
sampler <- FastqSampler(fqfl, 100000)
fq <- yield(sampler)                    # 100,000 reads
lattice::densityplot(gcFunction(sread(fq)), plot.points=FALSE)
fq <- yield(sampler)                    # a different 100,000 reads
@ 
%% 
Generally, one would choose a sample size large enough to adequately
characterize the data but not so large as to consume all (or a
fraction, see section~\ref{sec:parallel} below) of the memory. The
default (1 million) is a reasonable starting point.

\Robject{FastqSampler} relies on the \R{} random number generator, so
the same sequence of reads can be sampled by using the same random
number seed. This is a convenient way to sample the same read pairs
from the FASTQ files typically used to represent paired-end reads
<<fastq-paired-end, eval=FALSE>>=
## NOT RUN
set.seed(123)
end1 <- yield(FastqSampler("end_1.fastq"))
set.seed(123)
end2 <- yield(FastqSampler("end_2.fastq"))
@ 

\section{Iteration}

Restriction may not be enough to wrestle large data down to size, and
sampling may be inappropriate for the task at hand. A solution is then
to iterate through the file. An example in base \R{} is to open a file
connection, and then read and process successive chunks of the file,
e.g., reading chunks of 10000 lines
<<readLines-chunks, eval=FALSE>>=
## NOT RUN
con <- file("<hypothetical-file>.txt")
open(f)
while (length(x <- readLines(f, n=10000))) {
    ## work on character vector 'x'
}
close(f)
@ 
%% 
The pattern \Rcode{length(x <- readLines(f, n=10000))} is a convenient
short-hand pattern that reads the \emph{next} 10000 lines into a
variable \Robject{x} and then asks \Robject{x} it's
length. \Robject{x} is created in the calling environment (so it is
available for processing in the \Rcode{while} loop) When there are no
lines left to read, \Rcode{length(x)} will evaluate to zero and the
\Rcode{while} loop will end.

Iteration really involves three steps: input of a `chunk' of the data;
calculation of a desired summary; and aggregation of summaries across
chunks. We illustrate this with BAM files in the \Biocpkg{Rsamtools}
package; see also \Rfunction{FastqStreamer} in the \Biocpkg{ShortRead}
package and \Rfunction{TabixFile} for iterating through large VCF
files (via \Rfunction{readVCF} in the \Biocpkg{VariantAnnotation}
package).

The first stage in iteration is to arrange for input of a chunk of
data. In many programming languages one would iterate
`record-at-a-time', reading in one record, processing it, and moving
to the next. \R{} is not efficient when used in this fashion. Instead,
we want to read in a larger chunk of data, typically as much as can
comfortably fit in available memory. In \Biocpkg{Rsamtools} for
reading BAM files we arrange for this by creating a \Robject{BamFile}
(or \Robject{TabixFile} for VCF) object where we specify an
appropriate \Rfunarg{yieldSize}. Here we go for a bigger BAM file
<<iteration-bam>>=
library(RNAseqData.HNRNPC.bam.chr14)
bamfl <- RNAseqData.HNRNPC.bam.chr14_BAMFILES[1]
countBam(bamfl)
bf <- BamFile(bamfl, yieldSize=200000) # could be larger, e.g., 2-10 million
@ 
%% 
A \Robject{BamFile} object can be used to read data from a BAM file,
e.g., using \Rfunction{readGappedAlignments}. Instead of reading all
the data, the reading function will read just \Rfunarg{yieldSize}
records. The idea then is to open the BAM file and iterate through
until no records are input. The pattern is
<<BamFile-iter, eval=FALSE>>=
## initialize, e.g., for step 3 ...
open(bf)
while (length(gal <- readGappedAlignments(bf))) {
    ## step 2: do work...
    ## step 3: aggregate results...
}
close(bf)
@ 

The second step is to perform a useful calculation on the chunk of
data. This is particularly easy to do if chunks are independent of one
another. For instance, a common operation is to count the number of
times reads overlap regions of interest. There are functions that
implement a variety of counting modes (see, e.g.,
\Rfunction{summarizeOverlaps} in the \Biocpkg{GenomicRanges} package);
here we'll go for a simple counter that arranges to tally one `hit'
each time a read overlaps (in any way) at most one range. Here is our
counter function, taking as arguments a \Robject{GRanges} or
\Robject{GRangesList} object representing the regions for which counts
are desired, and a \Robject{GappedAlignments} object representing
reads to be counted:
<<counter-iter>>=
counter <- 
    function(query, subject, ..., ignore.strand=TRUE)
    ## query: GRanges or GRangesList
    ## subject: GappedAlignments
{
    if (ignore.strand)
        strand(subject) <- "*"
    hits <- countOverlaps(subject, query)
    countOverlaps(query, subject[hits==1])
}
@ 
%% 
To set this step up a little more completely, we need to know the
regions over which counting is to occur. Here we retrieve exons
grouped by gene for the genome to which the BAM files were
aligned:
<<counter-roi>>=
library(TxDb.Hsapiens.UCSC.hg19.knownGene)
query <- exonsBy(TxDb.Hsapiens.UCSC.hg19.knownGene, "gene")
@ 
Thus we'll add the following lines to our loop:
<<counter-work, eval=FALSE>>=
## initialize, e.g., for step 3 ...
open(bf)
while (length(gal <- readGappedAlignments(bf))) {
    ## step 2: do work...
    count0 <- counter(query, gal, ignore.strand=TRUE)
    ## step 3: aggregate results...
}
close(bf)
@ 
%% 
It is worth asking whether the `do work' step will always be as
straight-forward. The current example is easy because counting
overlaps of one read does not depend on other reads, so the chunks can
be processed independently of one another. \FIXME{These are paired-end
  reads, so counting is \emph{not} this easy! See the lab on RNA-seq}.

The final step in iteration is to aggregate results across chunks. In
our present case, \Rfunction{counter} always returns an integer vector
with \Rcode{length(query)} elements. We want to add these over chunks,
and we can arrange to do this by starting with an initial vector of
counts \Rcode{counts <- integer(length(query))} and simply adding
\Rcode{count0} at each iteration. The complete iteration, packaged as
a function to facilitate re-use is
<<counter-work>>=
counter1 <- 
    function(bf, query, ...)
{
    ## initialize, e.g., for step 3 ...
    counts <- integer(length(query))
    open(bf)
    while (length(gal <- readGappedAlignments(bf, ...))) {
        ## step 2: do work...
        count0 <- counter(query, gal, ignore.strand=TRUE)
        ## step 3: aggregate results...
        counts <- counts + count0
    }
    close(bf)
    counts
}
@ 
%% 
In action, this is invoked as
<<counter1>>=
bf <- BamFile(bamfl, yieldSize=500000)
counts <- counter1(bf, query)
@ 


Counting is a particularly simple operation; one will often need to
think carefully about how to aggregate statistics derived from
individual chunks. A useful general approach is to identify
\emph{sufficient statistics} that represent a description of the data
and can be easily aggregated across chunks. This is the approach taken
by, for instance, the \Rpackage{biglm} package for fitting linear
models to large data sets -- the linear model is fit to successive
chunks and the fit reduced to sufficient statistics, the sufficient
statistics are then aggregated across chunks to arrive at an overall
fit.

\section{Parallel evaluation}
\label{sec:parallel}

Each of the preceding sections addressed memory management; what about
overall performance?

The starting point is really writing efficient, vectorized code, with
common strategies outlined elsewhere. Performance differences between
poorly written versus well written \R{} code can easily span two
orders of magnitude, whereas parallel processing can only increase
throughput by an amount inversely proportional to the number of
processing units (e.g., CPUs) available.

The memory management techniques outlined earlier in this chapter are
important in a parallel evaluation context. This is because we will
typically be trying to exploit multiple processing cores on a single
computer, and the cores will be competing for the same pool of shared
memory. We thus want to arrange for the collection of processors to
cooperate in dividing available memory between them, i.e., each
processor needs to use only a fraction of total memory.

There are a number of ways in which \R{} code can be made to run in
parallel. The least painful and most effective will use `multicore'
functionality provided by the \Rpackage{parallel} package; the
\Rpackage{parallel} package is installed by default with base \R, and
has a useful vignette~\cite{Rparallel}. Unfortunately, multicore
facilities are not available on Windows computers.

Parallel evaluation on several cores of a single Linux or MacOS
computer is particularly easy to achieve when the code is already
vectorized. The solution on these operating systems is to use the
functions \Rfunction{mclapply} or \Rfunction{pvec}. These functions
allow the `master' process to `fork' processes for parallel evaluation
on each of the cores of a single machine. The forked processes
initially share memory with the master process, and only make copies
of data when the forked process modifies a memory location (`copy on
write' semantics). On Linux and MacOS, the \Rfunction{mclapply}
function is meant to be a `drop-in' replacement for
\Rfunction{lapply}, but with iterations being evaluated on different
cores.  The following illustrates the use of \Rfunction{mclapply} for
a toy vectorized function \Rfunction{f}:
\begin{Schunk}
\begin{Sinput}
> library(parallel)
> f <- function(i) {
+     cat("'f' called, length(i) = ", length(i), "\n")
+     sqrt(i)
+ }
> res0 <- mclapply(1:5, f, mc.cores=2)
\end{Sinput}
\begin{Soutput}
'f' called, length(i) =  1 
'f' called, length(i) =  1 
'f' called, length(i) =  1 
'f' called, length(i) =  1 
'f' called, length(i) =  1 
\end{Soutput}
\end{Schunk}
\Rfunction{pvec} is a little more subtle. It takes a vectorized
function and distributes computation of different chunks of the vector
across cores.
\begin{Schunk}
\begin{Sinput}
> res1 <- pvec(1:5, f, mc.cores=2)
\end{Sinput}
\begin{Soutput}
'f' called, length(i) =  3 
'f' called, length(i) =  2 
\end{Soutput}
\begin{Sinput}
> identical(unlist(res0), res1)
\end{Sinput}
\begin{Soutput}
[1] TRUE
\end{Soutput}
\end{Schunk}
%% 
Both \Rfunction{mclapply} and \Rfunction{pvec} functions allow the
user to specify the number of cores used, and how the data are divided
into chunks.

The \Rpackage{parallel} package does not support fork-like
behavior on Windows, where users need to more explicitly create a
cluster of \R{} workers and arrange for each to have the same data
loaded into memory; similarly, parallel evaluation across computers
(e.g., in a cluster) require more elaborate efforts to coordinate
workers; this is typically done using \Rfunction{lapply}-like
functions provided by the \Rpackage{parallel} package but specialized
for simple (`snow') or more robust (`MPI') communication protocols
between workers.

Data movement and random numbers are two important additional
considerations in parallel evaluation.  Moving data to and from cores
to the manager can be expensive, so strategies that minimize explicit
movement (e.g., passing file names or data base queries rather than
\R{} objects read from files; reducing data on the worker before
transmitting results to the manager) are important.  Random numbers
need to be synchronized across cores to avoid generating the same
sequences on each `independent' computation.

How might parallel evaluation be used in \Bioconductor{} work flows?
One approach when working with BAM files exploits the fact that data
are often organized with one sample per BAM file. Suppose we are
interested in running our iterating counter \Rfunction{counter1} over
several BAM files. We could do this by creating a \Rcode{BamFileList}
with appropriate \Rfunarg{yieldSize}
<<count-bfl-make>>=
fls <- RNAseqData.HNRNPC.bam.chr14_BAMFILES  # 8 BAM files
bamfls <- BamFileList(fls, yieldSize=500000) # use a larger yieldSize
@ 
%% 
and using an \Rfunction{lapply} to take each file in turn and
performing the count.
<<count-bams, eval=FALSE>>=
counts <- simplify2array(lapply(bamfls, counter1, query))
@ 
%% 
The parallel equivalent of this is simply (note the change from
\Rfunction{lapply} to \Rfunction{mclapply})
<<count-bams-mclapply>>=
options(mc.cores=detectCores())         # use all cores
counts <- simplify2array(mclapply(bamfls, counter1, query))
head(counts[rowSums(counts) != 0,], 3)
@ 
%% 
Note that \Rcode{names(bamfls)} and \Rcode{names(query)} are returned
as row and column names of the data, reducing chance of mistaken
labelling.

\section{And\ldots}

The \Rpackage{foreach} package can be useful for parallel evaluation
written using coding styles more like \Rcode{for} loops rather than
\Rcode{lapply}. The \Rpackage{iterator} package is an abstraction that
simplifies the notion of iterating over objects. The \Rpackage{Rmpi}
package provides access to MPI, a structured environment for
calculation on clusters. The \Rpackage{pbdR}
formalism\footnote{\url{https://rdav.nics.tennessee.edu/2012/09/pbdr/}}
is especially useful for well-structured distributed matrix
computations.

The \Rpackage{MatrixEQTL} package is an amazing example implementing
high performance algorithms on large data; the corresponding
publication~\cite{shabalin2012matrix} is well worth studying.

\Bioconductor{} provides an Amazon machine instance with MPI and
\Rpackage{Rmpi}
installed\footnote{\url{http://bioconductor.org/help/bioconductor-cloud-ami/}}. This
can be an effective way to gain access to large computing resources.

