%\VignetteIndexEntry{B.1 Gene-centric Annotation}

\SweaveOpts{keep.source=TRUE, pdf=FALSE, png=TRUE}
<<setup, echo=FALSE>>=
options(digits=2)
library(org.Dm.eg.db)
CONNECTION_OK <- tryCatch({ 
    con <- url("http://google.com", "r"); close(con); TRUE
}, error=function(...) FALSE)
@ 

\chapter{Gene-centric Annotation}
\label{chpt:GeneCentricAnnotation}

Be sure to update your package to the current version by visiting the
course web server 192.168.0.9 and installing (if neccessary) the
current version of the \Rpackage{Morgan2013} package.
<<PKG_OK>>=
PKG_OK <- packageDescription("Morgan2013")$Version >= "0.9.0"
stopifnot(PKG_OK)
@ 

\Bioconductor{} provides extensive annotation resources, some of which
are summarized in Figure~\ref{fig:dbtypes}. These can be \emph{gene}-,
or \emph{genome}-centric. Annotations can be provided in packages
curated by \Bioconductor, or obtained from web-based
resources. Gene-centric \Biocpkg{AnnotationDbi} packages include:
\begin{packeditemize}
  \item Organism level: e.g. \Biocannopkg{org.Mm.eg.db},
    \Biocannopkg{Homo.sapiens}.
  \item Platform level: e.g. \Biocannopkg{hgu133plus2.db},
        \Biocannopkg{hgu133plus2.probes}, \Biocannopkg{hgu133plus2.cdf}.
  \item Homology level: e.g. \Biocannopkg{hom.Dm.inp.db}.
  \item System biology level: \Biocpkg{GO.db}, \Biocpkg{KEGG.db},
    \Biocpkg{Reactome.db}.
\end{packeditemize}
Examples of genome-centric packages include:
\begin{packeditemize}
  \item \Biocpkg{GenomicFeatures}, to represent genomic features,
    including constructing reproducible feature or transcript data
    bases from file or web resources.
  \item Pre-built transcriptome packages,
    e.g. \Biocannopkg{TxDb.Hsapiens.UCSC.hg19.knownGene} based on the
    \Hsap{} UCSC hg19 knownGenes track.
  \item \Biocpkg{BSgenome} for whole genome sequence representation
    and manipulation.
  \item Pre-built genomes, e.g.,
    \Biocannopkg{BSgenome.Hsapiens.UCSC.hg19} based on the \Hsap{}
    UCSC hg19 build.
\end{packeditemize}
Web-based resources include
\begin{packeditemize}
  \item \Biocpkg{biomaRt} to query
    \href{http://www.biomart.org/}{biomart} resource for genes,
    sequence, SNPs, and etc.
  \item \Biocpkg{rtracklayer} for interfacing with browser tracks,
    especially the \href{http:://genome.ucsc.edu}{UCSC} genome browser.
  \item \Biocpkg{AnnotationHub} for retrieving common large genomic
    resources in formats that readily fit into \R{} work flows.
\end{packeditemize}

\begin{figure}
  \centering
  \includegraphics[width=.5\textwidth]{databaseTypes.pdf}
  \caption{Annotation Packages: the big picture}
  \label{fig:dbtypes}
\end{figure}

\section{Gene-centric annotations with \Biocpkg{AnnotationDbi}}

Organism-level (`org') packages contain mappings between a central
identifier (e.g., Entrez gene ids) and other identifiers (e.g. GenBank
or Uniprot accession number, RefSeq id, etc.).  The name of an
Rpackage{org} package is always of the form \emph{org.<Sp>.<id>.db}
(e.g. \Biocannopkg{org.Sc.sgd.db}) where \emph{<Sp>} is a 2-letter
abbreviation of the organism (e.g. \texttt{Sc} for \emph{Saccharomyces
  cerevisiae}) and \emph{<id>} is an abbreviation (in lower-case)
describing the type of central identifier (e.g. \texttt{sgd} for gene
identifiers assigned by the \emph{Saccharomyces} Genome Database, or
\texttt{eg} for Entrez gene ids).  The `How to use the ``.db''
annotation packages' vignette in the \Biocpkg{AnnotationDbi} package
(org packages are only one type of `.db' annotation packages) is a
key reference.  The `.db' and most other \Bioconductor{} annotation
packages are updated every 6 months.

Annotation packages contain an object named after the package itself.
These objects are collectively called \Rclass{AnnotationDb} objects,
with more specific classes named \Rclass{OrgDb}, \Rclass{ChipDb} or
\Rclass{TranscriptDb} objects.  Methods that can be applied to these
objects include \Rfunction{cols}, \Rfunction{keys},
\Rfunction{keytypes} and \Rfunction{select}.  Common operations for
retrieving annotations are summarized in Table~\ref{tab:select-ops}.

\begin{table}
  \centering
  \caption{Common operations for retrieving and manipulating annotations.}
  \label{tab:select-ops}
  \begin{tabular}{lll}
    Category & Function & Description \\
    \hline

    Discover & \Rfunction{cols} & List the kinds of columns that can be returned \\
    & \Rfunction{keytypes} & List columns that can be used as keys \\
    & \Rfunction{keys} & List values that can be expected for a given keytype \\
    & \Rfunction{select} & Retrieve annotations matching
    \Rfunarg{keys}, \Rfunarg{keytype} and \Rfunarg{cols} \\
    
    Manipulate & \Rfunction{setdiff}, \Rfunction{union}, \Rfunction{intersect} & Operations on sets \\
    & \Rfunction{duplicated}, \Rfunction{unique} & Mark or remove duplicates \\
    & \Rfunction{\%in\%},  \Rfunction{match} &  Find matches  \\
    & \Rfunction{any}, \Rfunction{all} &  Are any \code{TRUE}?  Are all? \\
    & \Rfunction{merge} & Combine two different \Robject{data.frames} based on shared keys \\

    \Rclass{GRanges*} & \Rfunction{transcripts}, \Rfunction{exons}, \Rfunction{cds} & 
        Features (transcripts, exons, coding sequence) as \Rclass{GRanges}. \\

    & \Rfunction{transcriptsBy} , \Rfunction{exonsBy} & 
    Features group by  gene, transcript, etc., as \Rclass{GRangesList}.\\
    & \Rfunction{cdsBy}\\
    
    \hline
  \end{tabular}
\end{table}

%% select exercise
\begin{Exercise}
  What is the name of the org package for \emph{Drosophila}? Load it.
  Display the \Rclass{OrgDb} object for the \Biocpkg{org.Dm.eg.db}
  package.  Use the \Rfunction{cols} method to discover which sorts of
  annotations can be extracted from it.

  Use the \Rfunction{keys} method to extract UNIPROT identifiers and
  then pass those keys in to the \Rfunction{select} method in such a
  way that you extract the SYMBOL (gene symbol) and KEGG pathway
  information for each.

  Use \Rfunction{select} to retrieve the ENTREZ and SYMBOL
  identifiers of all genes in the KEGG pathway \texttt{00310}.
\end{Exercise}
\begin{Solution}
  The \Rclass{OrgDb} object is named \Rcode{org.Dm.eg.db}.
<<select>>=
library(org.Dm.eg.db)
cols(org.Dm.eg.db)
keytypes(org.Dm.eg.db)
uniprotKeys <- head(keys(org.Dm.eg.db, keytype="UNIPROT"))
cols <- c("SYMBOL", "PATH")
select(org.Dm.eg.db, keys=uniprotKeys, cols=cols, keytype="UNIPROT")
@
Selecting UNIPROT and SYMBOL ids of KEGG pathway \texttt{00310} is
very similar:
<<select-kegg>>=
kegg <- select(org.Dm.eg.db, "00310", c("UNIPROT", "SYMBOL"), "PATH")
nrow(kegg)
head(kegg, 3)
@
\end{Solution}

\begin{Exercise}
  For convenience, \Robject{lrTest}, a \Rclass{DGEGLM} object from an
  RNA-seq experiment, is included in the \Rpackage{Morgan2013}
  package.  The following code loads this data and creates a `top
  table' of the ten most differentially represented genes. This top
  table is then coerced to a \Rcode{data.frame}.
<<tt-to-eg>>=
library(Morgan2013)
library(edgeR)
library(org.Dm.eg.db)
data(lrTest)
tt <- as.data.frame(topTags(lrTest))
@
%% 

  Extract the Flybase gene identifiers (\Rcode{FLYBASE}) from the row
  names of this table and map them to their corresponding Entrez gene
  (\Rcode{ENTREZID}) and symbol ids (\Rcode{SYMBOL}) using
  \Rfunction{select}. Use \Rfunction{merge} to add the results of
  \Rfunction{select} to the top table.
\end{Exercise}

\begin{Solution}
<<tt-to-eg2>>=
fbids <- rownames(tt)
cols <- c("ENTREZID", "SYMBOL")
anno <- select(org.Dm.eg.db, fbids, cols, "FLYBASE")
ttanno <- merge(tt, anno, by.x=0, by.y="FLYBASE")
dim(ttanno)
head(ttanno, 3)
@
\end{Solution}

\section{\Biocpkg{biomaRt} and other web-based resources}

A short summary of select \Bioconductor{} packages enabling web-based
queries is in Table~\ref{tab:webannoservices}.

\begin{table}
  \centering
  \caption{Selected packages querying web-based annotation services.}
  \label{tab:webannoservices}
  \begin{tabular}{ll}
    Package & Description\\
    \hline
    \Biocpkg{biomaRt} & \url{http://biomart.org}, Ensembl and other annotations\\
    \Biocpkg{rtracklayer} & \url{http://genome.ucsc.edu}, genome tracks.\\
    \Biocpkg{AnnotationHub} & Ensembl, Encode, dbSNP, UCSC data objects \\
    \Biocpkg{uniprot.ws} & \url{http://uniprot.org}, protein annotations\\
    \Biocpkg{KEGGREST} & \url{http://www.genome.jp/kegg}, KEGG pathways\\
    \Biocpkg{SRAdb} & \url{http://www.ncbi.nlm.nih.gov/sra}, sequencing experiments.\\
    \Biocpkg{GEOquery} & \url{http://www.ncbi.nlm.nih.gov/geo/}, array and other data\\
    \Biocpkg{ArrayExpress} & \url{http://www.ebi.ac.uk/arrayexpress/}, array and other data\\
    \hline
  \end{tabular}
\end{table}

\subsection{Using biomaRt}

The \Biocpkg{biomaRt} package offers access to the online
\href{http://www.biomart.org}{biomart} resource. this consists of
several data base resources, referred to as `marts'.  Each mart allows
access to multiple data sets; the \Biocpkg{biomaRt} package provides
methods for mart and data set discovery, and a standard method
\Rfunction{getBM} to retrieve data.

\begin{Exercise}
  \WARNING{Internet connection required}

  Load the \Biocpkg{biomaRt} package and list the available marts.
  Choose the \emph{ensembl} mart and list the datasets for that mart.
  Set up a mart to use the \emph{ensembl} mart and the
  \emph{hsapiens\_gene\_ensembl} dataset.
 
  A \Biocpkg{biomaRt} dataset can be accessed via
  \Rfunction{getBM}. In addition to the mart to be accessed, this
  function takes filters and attributes as arguments.  Use
  \Rfunction{filterOptions} and \Rfunction{listAttributes} to discover
  values for these arguments.  Call \Rfunction{getBM} using filters
  and attributes of your choosing.
\end{Exercise}

\begin{Solution}
<<biomaRt1, eval=FALSE, results=hide>>=
library(biomaRt)
head(listMarts(), 3)                      ## list the marts
head(listDatasets(useMart("ensembl")), 3) ## mart datasets
ensembl <-                                ## fully specified mart
    useMart("ensembl", dataset = "hsapiens_gene_ensembl")

head(listFilters(ensembl), 3)             ## filters
myFilter <- "chromosome_name"
head(filterOptions(myFilter, ensembl), 3) ## return values
myValues <- c("21", "22")
head(listAttributes(ensembl), 3)          ## attributes
myAttributes <- c("ensembl_gene_id","chromosome_name")

## assemble and query the mart
res <- getBM(attributes =  myAttributes, filters =  myFilter,
             values =  myValues, mart = ensembl)
@
Use \code{head(res)} to see the results.
\end{Solution}

%% \FiXME{update this}
%% \section{Advanced aspects of \Rpackage{*.org} packages}

%% \subsection{\Biocpkg{AnnotationForge} to create custom packages}

%% \subsection{\software{mysql} data base and \code{SQL} queries}
