So day 2 of the #JHUSMARTHack was last week, but I figured this would be a good time to discuss what was accomplished. I created some packages that are somewhat specialized and aren't fully finished yet, so I'll hold off. What I really want to discuss though is why I like using RStudio for making packages.

What was accomplished?

I had spoken before about the repositories I had worked on, and also about Developing Packages in RStudio. I'll discuss the workflow I settled into for for making a package.

Workflow for an R package

I'm assuming your R code is already available, presumably functions you had created during a project or analysis. If code is not available, GREAT! You can start your workflow for your new package or product all the same. I'll try to put command-line equivalents in double brackets [[ ]].

Workflow:

1. Start RStudio.
2. Go to File -> New Project. (Save any unsaved work).
3. Select New Directory. Now this may be counterintuitive if you have work saved, but if you're creating a package choose this. This will setup a new folder and copy over any code you have already created.
4. Select R Package. This will allow you to name your package (it will be used in the library statement unless changed later); let's call it mypckg. You can also choose code you have to operationalize, e.g. put into a package. Also – select you want to create a git repository.
5. Voila! The folder is created where you have the components of an R package, such as the documentation (man), extra install dir (inst), the R code (R), etc. [[library(devtools); create("mypckg");]]

Now, here is one of the main reasons I like using RStudio for projects: the .Rproj file. The .Rproj file is an RStudio project file. It allows you to work on stuff in multiple tabs/scripts, then close the project, and pop up the other tabs/scripts you were working on before opening up that project. If you are in RStudio, the top right should show a Project: None if you don't have a project loaded. These project files allows me to segregate my workflows and scripts, and they help me organize a bit more. I highly recommend checking out Hilary Parker's post before continuing, especially if you're not an RStudio fan.

Using RStudio Build Tools

Now, when I say RStudio Build Tools, I essentially mean wrappers for the devtools package. The package is amazing (hardly shocking since Hadley Wickham is the main author), and along with the roxygen2 package, allow package creation to be as easy as possible.

Now, let's set up our options for Build Tools. In RStudio, go to Build -> Configure Build Tools (again you must be in an RStudio project). For Check Package, I recommend putting the --as-cran option (especially if you plan to submit to CRAN. You should also see a checkbox saying Generate documentation using Roxygen. If this is not available, run install.packages("roxygen2"), close and reopen the project. Check this box, and click the Configure button and I usually click all options.

Setting up a remote git repository

Before, we checked for a git repository to be created. Now, you can create a new repository in your favorite GitHub remote repository. Mine is GitHub. You can use the GUIs such as the GitHub GUI or SourceTree, but I generally set this up using the Terminal by just adding the remote. (Here is a link to create ssh keys so you don't have to type in passwords for git). Now, if you restarted the RStudio project, go to Build -> Configure Build Tools and you should see the remote repository if you click the Git/SVN tab.

Now that the repository is set up (even if you don't use a remote repository), you can go to Tools --> Commit to commit to the repository. This allows you to add and stage the changed files while adding a commit comment. You can also see a visual history of the differences and changes as well as do much of what you would need to from the command line. Again, I like the Terminal, but I like having this all in one program and not having to switch back and forth.

DOCUMENTATION! EXAMPLES! VIGNETTES!

Now that you have everything set up, you have to do the big things that differentiate a bunch of functions from a package: documentation and examples (including vignettes). Again, for documentation, we'll be using the roxygen2 package. Roxygen is essentially a format that starts with a line with #' followed by @ followed by a “tag”. The tags can be found at ?rd_roclet. Now, I highly recommend vignettes, but I'm not an expert on these and think we'll just stick to function docs right now.

Jumping to Sublime Text

Before we start documentation, let me again tell about MY workflow rather than Roxygen. My workflow now jumps to Sublime Text. I have Line-by-line installed (which you will need), which allows me to run a script to parse an R function and create the necessary Roxygen tags. See Alyssa's post for the description and a more command-line workflow for R packages.

Now that we're in Sublime Text, I open the .R files from my packages R directory. Select the function definition such as x = function(z, y, l=4, ...){ and use CMD+D to create Roxygen tags! This is like meta-programming for documentation: running scripts to make documentation (granted it's in Python). As an aside, one powerful feature of this documentation is that if you have code as:

LFPCAg <- function(
Y,# an n x D matrix
                   # Y is assumed to be centered by its mean function
                   gridpoints = 1:ncol(Y),       # a vector of grid points along curves, corresponding to columns of Y 
                   Zlist=NA,
                   G=NA,
                   Ivec=NA,
                   ...){

this will parse the Roxygen tags as the comments for each argument/parameter (even if multi-line!):

#' @title <brief desc>
#'
#' @description <full description>
#' @param Y an n x D matrix Y is assumed to be centered by its mean function
#' @param gridpoints a vector of grid points along curves corresponding to columns of Y
#' @param Zlist
#' @param G
#' @param Ivec
#' @param ... 
#' @export
#' @keywords
#' @seealso
#' @return
#' @aliases
#' @examples \dontrun{
#'
#'}

This also puts in your mind, even if you're only creating functions and not a package, that you'll almost have documentation ready made when using this function format from day 1.

Jumping Back to RStudio

Now, opening these Roxygen-tagged functions, I can fill in the rest in RStudio. One thing to note is that RStudio will assume you're trying to stay in Roxygen notation with a return of line (which is great for multi-line descriptions/titles/etc). Also, if you have #' @ starting a line, then RStudio will do tab completion of Roxygen tags. Not leaps and bounds saved on time, but hey, I like tab completion.

Now you have to write your examples, the description of arguments (denoted as parameters), the overall function description and title, and use the @export to allow this function accessible to the user. One note is that if you depend on another function or package, use the @import pkgname or @importsFrom pkgname::funcName tags. R CMD check will warn you if you don't have anything in @keywords, @aliases, or @examples, so remove these if not necessary.

Just let me check my functions!

If you're still working on the package and want to play with functions and no so much the documentation, you can use Build -> Load All [[devtools::load_all]] to load the functions (even those not exported) into memory.

Compile and Load

Now let's fast-forward to when you have created the the documentation for your functions. While still in your project, go to Build -> Build and Reload to get your package loaded into memory [[devtools::build then devtools::install]]. Roxygen will create the docs. FYI – if you change around function names and recompile, the man folder may have obsolete .Rd files, so you can delete old ones.

You should edit the DESCRIPTION file to change some specifications, such as Depends: fields for package dependencies. That's documented many places on the web to find about what goes in there.

Now edit your functions and docs, push to the remote repository and then allow people install your package by using:

library(devtools)
install_github("mypckg", "myGitHubUserName")

and there you have it – you've released software. Build -> Check Package is good for testing your package (will tests your examples) and make sure everything looks OK.

Conclusion

R package creation seems like a daunting task. You can use tools in RStudio such as Code -> Extract Function to take loads of code to try to functional-ize it. When you have a collection of functions, creating an RStudio project allows you to separate your package creation process from regular RStudio analysis and use, let's you have a one-stop shop for git version control, building, and checking of packages. It let's you get over any hurdles of learning new functions in devtools (which may not be a good thing) and get you running in a short amount of time. The Sublime Text plugin is not a crucial step, but can allow you to parse semi-documented functions and create a Roxygen header that's partially filled in. This workflow allowed me to develop multiple projects and get them documented quickly at the hackathon.

Hopefully this helps and good luck packaging!

Recently I was tasked with evaluating and most importantly removing analytical variance form a longitudinal metabolomic analysis carried out over a few years and including >2,5000 measurements for >5,000 patients. Even using state-of-the-art analytical instruments and techniques long term biological studies are plagued with unwanted trends which are unrelated to the original experimental design and stem from analytical sources of variance (added noise by the process of measurement). Below is an example of a metabolomic measurement with and without analytical variance.

The noise pattern can be estimated based on replicated measurements of quality control samples embedded at a ratio of 1:10 within the larger experimental design. The process of data normalization is used to remove analytical noise from biological signal on a variable specific basis. At the bottom of this post, you can find an in-depth presentation of how data quality can be estimated and a comparison of many common data normalization approaches. From my analysis I concluded that a relatively simple LOESS normalization is a very powerful method for removal of analytical variance. While LOESS (or LOWESS), locally weighted scatterplot smoothing, is a relatively simple approach to implement; great care has to be taken when optimizing each variable-specific model.

In particular, the span parameter or alpha controls the degree of smoothing and is a major determinant if the model  (calculated from repeated measures) is underfit, just right or overfit with regards to correcting analytical noise in samples. Below is a visualization of the effect of the span parameter on the model fit.

One method to estimate the appropriate span parameter is to use cross-validation with quality control samples. Having identified an appropriate span, a LOESS model can be generated from repeated measures data (black points) and is used to remove the analytical noise from all samples (red points).



Having done this we can now evaluate the effect of removing analytical noise from quality control samples (QCs, training data, black points above) and samples (test data, red points) by calculating the relative standard deviation of the measured variable (standard deviation/mean *100). In the case of the single analyte, ornithine, we can see (above) that the LOESS normalization will reduce the overall analytical noise to a large degree. However we can not expect that the performance for the training data (noise only) will converge with that of the test set, which contains both noise and true biological signal.

In addition to evaluating the normalization specific removal of analytical noise on a univariate level we can also use principal components analysis (PCA) to evaluate this for all variables simultaneously. Below is an example of the PCA scores for non-normalized and LOESS normalized data.



We can clearly see that the two largest modes of variance in the raw data explain differences in when the samples were analyzed, which is termed batch effects. Batch effects can mask true biological variability, and one goal of normalizations is to remove them, which we can see is accomplished in the LOESS normalized data (above right).

However be forewarned, proper model validation is critical to avoiding over-fitting and producing complete nonsense.



In case you are interested the full analysis and presentation can be found below as well as the majority of the R code used for the analysis and visualizations.

Data

Density plots

r12 <- merge(r1,r2)
# density
cols <- colorRampPalette(c('violet','gold','seagreen'))(4) 
library(lattice)
long <- reshape(r12,direction='long',
    v.names='Response',
    varying=list(colnames(r2)[-1]),
    idvar=c('ID','group'),
    timevar='Variable',
    times=colnames(r2)[-1])
densityplot(~ Response | Variable ,groups= group,
    data=long ,scales=list(relation='free'),
    col=cols,
    auto.key=list(
        text=c('Lower Paleolithic, Homo ergaster?, oldest',
            'Levallois technique',
            'Middle Paleolithic, probably Neanderthals',
            'Homo Sapiens, youngest'),
        col=cols,
        lines=FALSE))

Biplot

A bipot is easily made. However, I am a bit of a fan of the biplots detailed in Gower and Hand's book. Since the heavy lifting for that is now in package calibrate they are easily made.
r12c <- r12[complete.cases(r12),]
pr1 <- princomp(~ LBI + RTI + WDI + FLA + PSF + FSF + ZDF1 + PROZD,
    r12c,
    cor=TRUE,
    scores=TRUE)
biplot(pr1,xlabs=r12c$ID)

Most of the following code is from calibrate's vignette. The colors in point labels are an annotation which I made. Unfortunately the textxy() function did not get color as I intended, so a for loop is made to get it correct. The length of the blue axis are made via trial and error. It should be noted that, similar to any biplot, there is some deformation, the axis are approximate.
library(calibrate)
X0<- subset(r12c,,c(LBI,RTI,WDI,FLA,PSF,FSF,ZDF1 ,PROZD))
X <- scale(X0)
rownames(X) <- r12c$ID
pca.results <- princomp(X,cor=FALSE)
Fp <- pca.results$scores
Gs <- pca.results$loadings
# no margins
par(mar=rep(0.05,4))
plot(Fp[,1],Fp[,2],
    pch=16,asp=1,
    xlim=c(-5,5),ylim=c(-5,5),
    frame.plot=TRUE,axes=FALSE,
    cex=0.5,type='n',
    col=cols[r12c$group])

for( ii in unique(r12c$group))
  textxy(Fp[r12c$group==ii,1],
      Fp[r12c$group==ii,2],
      rownames(X)[r12c$group==ii],
      cex=0.75,
      col=cols[ii],offset=0)

for (ii in 1:ncol(X)) {
  myseq <- seq(-2,2)
  if (colnames(X)[ii]=='LBI') myseq <-seq(-2,3)
  if (colnames(X)[ii] %in% c('RTI','FSF','PROZD')) myseq <-seq(-1.4,1.4)
  if (colnames(X)[ii]=='ZDF1') myseq <-seq(-1.5,2)
  ticklab <- pretty(myseq*attr(X,'scaled:scale')[ii]+attr(X,'scaled:center')[ii])
  
  ticklabc <- (ticklab-attr(X,'scaled:center')[ii])/attr(X,'scaled:scale')[ii]
  yc <- X[,ii]
  g <- Gs[ii,1:2]
  Calibrate.X3 <- calibrate(g,yc,ticklabc,Fp[,1:2],ticklab,tl=0.1,
      axislab=colnames(X)[ii],cex.axislab=0.75,where=1,labpos=4)
}

legend(x='topleft',
    legend=c('Lower Paleolithic, Homo ergaster?, oldest',
        'Levallois technique',
        'Middle Paleolithic, probably Neanderthals',
        'Homo Sapiens, youngest'),
    text.col=cols,
    ncol=1,cex=.75)

Hierarchical clustering

Interpretation

Introduction

The point is that my line of business requires travel, and sometimes that is a lot of the time, like say almost all of last year. Inevitable comparisons to George Clooney's character in Up in the Air were made (ironically I started to read that book, then left it on a plane in a seatback pocket), requests about favours involving duty free, and of course many observations and gently probing questions about frequent flier miles (FYI I've got more than most people, but a lot less than the entrepreneur I sat next to one time, who claimed to have close to 3 million).

But I digress.

Background

The point is that, as I said, I spent quite a bit of time travelling for work last year. Apparently the story with frequent fliers miles is that it's best just to pick one airline and stick with it - and this also worked out well as most companies, including my employer, have preferred airlines and so you often don't have much of a choice in the matter.

In my case this means flying Delta.

So I happened to notice in one of my many visits to Delta's website that they have data on all of their aircraft in a certain site section. I thought this would be an interesting data set on which to do some analysis, as it has both quantitative and qualitative information and is relatively complex. What can we say about the different aircraft in Delta's fleet, coming at it with 'fresh eyes'? Which planes are similar? Which are dissimilar?

Aircraft data card from Delta.com

The data set comprises 33 variables on 44 aircraft taken from Delta.com, including both quantitative measures on attributes like cruising speed, accommodation and range in miles, as well as categorical data on, say, whether a particular aircraft has Wi-Fi or video. These binary categorical variables were transformed into quantitative variables by assigning them values of either 1 or 0, for yes or no respectively.

Analysis

data <- read.csv(file="delta.csv", header=T, sep=",", row.names=1)

# scatterplot matrix of intermediary (size/non-categorical) variables
plot(data[,16:22])

We can see that there are pretty strong positive correlations between all these variables, as all of them are related to the aircraft's overall size. Remarkably there is an almost perfectly linear relationship between wingspan and tail height, which perhaps is related to some principle of aeronautical engineering of which I am unaware.

The exception here is the variable right in the middle which is the number of engines. There is one lone outlier [Boeing 747-400 (74S)] which has four, while all the other aircraft have two. In this way the engines variable is really more like a categorical variable, but we shall as the analysis progresses that this is not really important, as there are other variables which more strongly discern the aircraft from one another than this.

How do we easier visualize a high-dimensional data set like this one? By using a dimensionality reduction technique like principal components analysis.

Principal Components Analysis

Next let's say I know nothing about dimensionality reduction techniques and just naively apply principle components to the data in R:

# Naively apply principal components analysis to raw data and plot
pc <- princomp(data)
plot(pc)

Taking that approach we can see that the first principal component has a standard deviation of around 2200 and accounts for over 99.8% of the variance in the data. Looking at the first column of loadings, we see that the first principle component is just the range in miles.

# First component dominates greatly. What are the loadings?
summary(pc) # 1 component has > 99% variance
loadings(pc) # Can see all variance is in the range in miles 

Importance of components:
                             Comp.1       Comp.2       Comp.3       Comp.4
Standard deviation     2259.2372556 6.907940e+01 2.871764e+01 2.259929e+01
Proportion of Variance    0.9987016 9.337038e-04 1.613651e-04 9.993131e-05
Cumulative Proportion     0.9987016 9.996353e-01 9.997966e-01 9.998966e-01
            

                         Comp.1 Comp.2 Comp.3 Comp.4 Comp.5 Comp.6 Comp.7 Comp.8
Seat.Width..Club.                                    -0.144 -0.110              
Seat.Pitch..Club.                                    -0.327 -0.248         0.189
Seat..Club.                                                                     
Seat.Width..First.Class.                0.250        -0.160        -0.156  0.136
Seat.Pitch..First.Class.                0.515 -0.110 -0.386  0.112 -0.130  0.183
Seats..First.Class.                     0.258 -0.124 -0.307 -0.109  0.160  0.149
Seat.Width..Business.                  -0.154  0.142 -0.108                     
Seat.Pitch..Business.                  -0.514  0.446 -0.298  0.154 -0.172  0.379
Seats..Business.                       -0.225  0.187                            
Seat.Width..Eco.Comfort.                                     0.285 -0.224       
Seat.Pitch..Eco.Comfort.                0.159                0.544 -0.442       
Seats..Eco.Comfort.                                          0.200 -0.160       
Seat.Width..Economy.                                  0.125  0.110              
Seat.Pitch..Economy.                                  0.227  0.190        -0.130
Seats..Economy.                  0.597        -0.136  0.345 -0.165         0.168
Accommodation                    0.697               -0.104                0.233
Cruising.Speed..mph.                    0.463  0.809  0.289 -0.144  0.115       
Range..miles.             0.999                                                 
Engines                                                                         
Wingspan..ft.                    0.215         0.103 -0.316 -0.357 -0.466 -0.665
Tail.Height..ft.                                     -0.100        -0.187       
Length..ft.                      0.275         0.118 -0.318  0.467  0.582 -0.418
Wifi                                                                            
Video                                                                           
Power                                                                           
Satellite                                                                       
Flat.bed                                                                        
Sleeper                                                                         
Club                                                                            
First.Class                                                                     
Business                                                                        
Eco.Comfort                                                                     

Economy                                         

This is because the scale of the different variables in the data set is quite variable; we can see this by plotting the variance of the different columns in the data frame (regular scaling on the left, logarithmic on the right):

# verify by plotting variance of columns
mar <- par()$mar
par(mar=mar+c(0,5,0,0))
barplot(sapply(data, var), horiz=T, las=1, cex.names=0.8)
barplot(sapply(data, var), horiz=T, las=1, cex.names=0.8, log='x')
par(mar=mar)

We correct for this by scaling the data using the scale() function. We can then verify that the variances across the different variables are equal so that when we apply principal components one variable does not dominate.

# Scale
data2 <- data.frame(scale(data))
# Verify variance is uniform
plot(sapply(data2, var))

After applying the scale() function the variance is now constant across variables

Now we can apply principal components to the scaled data. Note that this can also be done automatically in call to the prcomp() function by setting the parameter scale=TRUE. Now we see a result which is more along the lines of something we would expect:

# Proceed with principal components
pc <- princomp(data2)
plot(pc)
plot(pc, type='l')
summary(pc) # 4 components is both 'elbow' and explains >85% variance

Great, so now we're in business. There are various rules of thumb for selecting the number of principal components to retain in an analysis of this type, two of which I've read about are:

1. Pick the number of components which explain 85% or greater of the variation
2. Use the 'elbow' method of the scree plot (on right)

Here we are fortunate in that these two are the same, so we will retain the first four principal components. We put these into new data frame and plot.

# Get principal component vectors using prcomp instead of princomp
pc <- prcomp(data2)

# First for principal components
comp <- data.frame(pc$x[,1:4])
# Plot
plot(comp, pch=16, col=rgb(0,0,0,0.5))

library(rgl)
# Multi 3D plot
plot3d(comp$PC1, comp$PC2, comp$PC3)
plot3d(comp$PC1, comp$PC3, comp$PC4)

# Determine number of clusters
wss <- (nrow(mydata)-1)*sum(apply(mydata,2,var))
for (i in 2:15) wss[i] <- sum(kmeans(mydata,
                                     centers=i)$withinss)
plot(1:15, wss, type="b", xlab="Number of Clusters",
     ylab="Within groups sum of squares")

Clustering without the nstart parameter can lead to variable results for each run

Clustering with the nstart and iter.max parameters leads to consistent results, allowing proper interpretation of the scree plot

# From scree plot elbow occurs at k = 4
# Apply k-means with k=4
k <- kmeans(comp, 4, nstart=25, iter.max=1000)
library(RColorBrewer)
library(scales)
palette(alpha(brewer.pal(9,'Set1'), 0.5))
plot(comp, col=k$clust, pch=16)

# 3D plot
plot3d(comp$PC1, comp$PC2, comp$PC3, col=k$clust)
plot3d(comp$PC1, comp$PC3, comp$PC4, col=k$clust)

# Cluster sizes
sort(table(k$clust))
clust <- names(sort(table(k$clust)))

# First cluster
row.names(data[k$clust==clust[1],])
# Second Cluster
row.names(data[k$clust==clust[2],])
# Third Cluster
row.names(data[k$clust==clust[3],])
# Fourth Cluster
row.names(data[k$clust==clust[4],])

The first cluster contains a single aircraft, the Airbus A319 VIP. This plane is on its own and rightly so - it is not part of Delta's regular fleet but one of Airbus' corporate jets. This is a plane for people with money, for private charter. It includes "club seats" around tables for working (or not). Below is a picture of the inside of the A319 VIP:

Ahhh, that's the way fly (some day, some day...). This is apparently the plane professional sports teams and the American military often charter to fly - this article in the Sydney Morning Herald has more details.

The second cluster contains four aircraft - the two CRJ 100/200's and the Embraer E120 and ERJ-145. These are the smallest passenger aircraft, with the smallest accommodations - 28 for the E120 and 50 for the remaining craft. As such, there is only economy seating in these planes which is what distinguishes them from the remainder of the fleet. The E120 also has the distinction of being the only plane in the fleet with turboprops. Photos below.

Top: CRJ100/200. Bottom left: Embraer E120. Bottom right: Embraer ERJ-145.

I've flown many times in the venerable CRJ 100/200 series planes, in which I can assure you there is only economy seating, and which I like to affectionately refer to as "little metal tubes of suffering."

The other two clusters comprise the remainder of the fleet, the planes with which most commercial air travellers are familiar - your Boeing 7-whatever-7's and other Airbus and McDonnell-Douglas planes.

These are split into two clusters, which seem to again divide the planes approximately by size (both physical and accommodation), though there is crossover in the Boeing craft.

# Compare accommodation by cluster in boxplot
boxplot(data$Accommodation ~ k$cluster,
        xlab='Cluster', ylab='Accommodation',
        main='Plane Accommodation by Cluster')

# Compare presence of seat classes in largest clusters
data[k$clust==clust[3],30:33]
data[k$clust==clust[4],30:33]

	First.Class	Business	Eco.Comfort	Economy
Airbus A330-200	0	1	1	1
Airbus A330-200 (3L2)	0	1	1	1
Airbus A330-200 (3L3)	0	1	1	1
Airbus A330-300	0	1	1	1
Boeing 747-400 (74S)	0	1	1	1
Boeing 757-200 (75E)	0	1	1	1
Boeing 757-200 (75X)	0	1	1	1
Boeing 767-300 (76G)	0	1	1	1
Boeing 767-300 (76L)	0	1	1	1
Boeing 767-300 (76T)	0	1	1	1
Boeing 767-300 (76Z V.1)	0	1	1	1
Boeing 767-300 (76Z V.2)	0	1	1	1
Boeing 767-400 (76D)	0	1	1	1
Boeing 777-200ER	0	1	1	1
Boeing 777-200LR	0	1	1	1
	First.Class	Business	Eco.Comfort	Economy
Airbus A319	1	0	1	1
Airbus A320	1	0	1	1
Airbus A320 32-R	1	0	1	1
Boeing 717	1	0	1	1
Boeing 737-700 (73W)	1	0	1	1
Boeing 737-800 (738)	1	0	1	1
Boeing 737-800 (73H)	1	0	1	1
Boeing 737-900ER (739)	1	0	1	1
Boeing 757-200 (75A)	1	0	1	1
Boeing 757-200 (75M)	1	0	1	1
Boeing 757-200 (75N)	1	0	1	1
Boeing 757-200 (757)	1	0	1	1
Boeing 757-200 (75V)	1	0	1	1
Boeing 757-300	1	0	1	1
Boeing 767-300 (76P)	1	0	1	1
Boeing 767-300 (76Q)	1	0	1	1
Boeing 767-300 (76U)	0	1	1	1
CRJ 700	1	0	1	1
CRJ 900	1	0	1	1
E170	1	0	1	1
E175	1	0	1	1
MD-88	1	0	1	1
MD-90	1	0	1	1
MD-DC9-50	1	0	1	1

Looking at the raw data, the difference I can ascertain between the largest two clusters is that all the aircraft in the one have first class seating, whereas all the planes in the other have business class instead [the one exception being the Boeing 767-300 (76U)].

Conclusions

This was a little analysis which for me not only allowed me to explore my interest in commercial aircraft, but was also educational about finer points of what to look out for when using more advanced data science techniques like principal components, clustering and advanced visualization.

All in all, the techniques did a pretty admirable job in separating out the different type of aircraft into distinct categories. However I believe the way I structured the data may have biased it towards categorizing the aircraft by seating class, as that quality was replicated in the data set compared to other variables, being represented both in quantitative variables (seat pitch & width, number of seat in class) and categorical (class presence). So really the different seating classes where represented in triplicate within the data set compared to other variables, which is why the methods separated the aircraft in this way.

If I did this again, I would structure the data differently and see what relationships such analysis could draw out using only select parts of the data (e.g. aircraft measurements only). The interesting lesson here is that it when using techniques like dimensionality reduction and clustering it is not only important to be mindful of applying them correctly, but also what variables are in your data set and how they are represented.

This post shows how to build a custom univariate distribution in R from scratch, so that you end up with the essential functions: a probability density function, cumulative distribution function, quantile function and random number generator.

In the beginning all you need is an equation of the probability density function, from which everyting else can be derived sometimes analytically, and always numerically. The analytical solutions may require some math skills, or may be impossible to find, but the numerical solutions are always feasible, require no math literacy, and coding them in R is easy with the uniroot() and integrate() functions.

Throughout the post I will use a simple exponential distribution as an example.

Probability density function (PDF)

You will need to know what probability density (or mass) function is, and what is the difference between probability density and probability. I found the best intro to be this Khan's video (10 min). Here is a couple of additional points:

• Probability density is relative probability.
• Probability density function evelauated for a given data point is the point's likelihood.
• Probability density can be greater than one.

Example: Exponential PDF

The formula for the exponential probability density function (PDF) is:

In literature, small usually denotes probability density, while capital is used for probability.

We can code it in R:

  my.dexp <- function(x, lambda) lambda*exp(-lambda*x)
  my.dexp(x=0:5, lambda=2)

## [1] 2.0000000 0.2706706 0.0366313 0.0049575 0.0006709 0.0000908

And compare it with R's native PDF (it uses rate instead of ):

  dexp(x=0:5, rate=2)

## [1] 2.0000000 0.2706706 0.0366313 0.0049575 0.0006709 0.0000908

Here is a plot of my PDF using the R's built-in function curve():

  curve(my.dexp(x, lambda=2), from=0, to=2, main="Exponential PDF")

Cumulative distribution function (CDF) - analytical solution

This function gives, for a given point , the area under the PDF curve all the way down to the left of the point . This area is the probability that an outcome will have value lower or equal to :

I liked this stepbil's video (9 min) that shows how to flip between PDF and CDF, and why do we need the in the integral. I am really lame with calculus and what really works for me is Sage – it is free, it finds analytical solutions of integrals, and it may be worth a shot before diving into numerical integration.

For the exponential distribution the CDF is:

In R it is:

  my.pexp.anal <- function(x, lambda) 1- exp(-lambda*x)
  my.pexp.anal(x=0:5, lambda=2)

## [1] 0.0000 0.8647 0.9817 0.9975 0.9997 1.0000

Cumulative distribution function (CDF) - numerical solution

For most practical purposes, numerical solutions of one- or two- dimensional problems seem to be as good as the analytical solutions, the only disadvantage is the computational demands. Here I used the R's native function integrate() to calculate the exponential CDF numerically:

  my.pexp.numerical <- function(x, lambda)
  {
    # this my.int function allows my CDF to run over vectors
    my.int <- function(x, lambda) integrate(my.dexp, lambda=lambda, lower=0, upper=x)$value
    # apply the my.int to each element of x
    sapply(x, FUN=my.int, lambda)
  }
  my.pexp.numerical(x=0:5, lambda=2)

## [1] 0.0000 0.8647 0.9817 0.9975 0.9997 1.0000

And let's plot the numerical solution over the analytical one:

  curve(my.pexp.anal(x, lambda=2), from=0, to=2, 
        col="green", main="exponential CDF")
  curve(my.pexp.numerical(x, lambda=2), from=0, to=2, 
        lty=2, col="red", add=TRUE)
  legend("bottomright", lty=c(1,2), 
         legend=c("Analytical", "Numerical"),        
         col=c("green", "red"), lwd=2)

Practically identical.

Quantile function (QF) - analytical solution

Quantile functions are useful for two things: (1) assessing statistical significance, and (2) generating random numbers (see below). Quantile function () is the inverse of the cumulative distribution function. It means that you have to find a value of for a given , which is an inverse action to what we usually do with functions. Finding an analytical solution may be quite tricky, and again, if you don't have a mathematician at your service, Sage can help.

In case of our exponential distribution, the quantile function is:

In R:

  my.qexp.anal <- function(q, lambda) (-log(1-q))/lambda 
  my.qexp.anal(0.9, 2)

## [1] 1.151

Quantile function (QF) - numerical solution

The inverse may be hard to get for many CDFs, and hence one may need to go for a numerical solution. I found that the easiest way to solve for inverse of univariate CDFs in R is the uniroot() function. Alternatives are optimize() or optim(), but they can be unstable. In contrast, uniroot() is simple, quick, and stable.

Here is my numerical solution for the expoentnial QF:

  my.qexp.numerical <- function(q, lambda)
  { 
    # function to be solved for 0
    f <- function(P, fixed)
    {
      lambda <- fixed$lambda
      q <- fixed$q
      # this is the criterion to be minimized by uniroot():
      criterion <- q - my.pexp.numerical(P, lambda)
      return(criterion)
    }

    # for each element of vector P (the quantiles)
    P <- numeric(length(q))
    for(i in 1:length(q))
    {
      # parameters that stay fixed
      fixed <- list(lambda=lambda, q=q[i])
      # solving the f for 0 numerically by uniroot():
      root.p <- uniroot(f, 
                        lower=0, 
                        upper=100, # may require adjustment
                        fixed=fixed)
      P[i] <-root.p$root
    }
    return(P)
  }

my.qexp.numerical(0.9, 2)

## [1] 1.151

Let's plot the numerical solution over the analytical one:

  q <- seq(0.01, 0.9, by=0.01)
  plot(q, my.qexp.anal(q, lambda=2), type="l", col="green", lwd=2)
  lines(q, my.qexp.numerical(q, lambda=2), col="red", lty=2)
  legend("bottomright", lty=c(1,2), 
         legend=c("Analytical", "Numerical"),        
         col=c("green", "red"), lwd=2)

Random number generator - the inverse transform sampling

Have you ever wondered how R generates random draws from its distributions? How does it convert uniformly distributed pseudo-random numbers to, e.g., normally distributed numbers? For some time I naively thought that there are some giant built-in tables that R uses. Then I found that it's really simple.

The easiest way is the inverse transform sampling. It works like this: you generate a random number from and plug it into your quantile function (see above). That's it, and that's also why quantile functions can be really useful. Here is how you do it in R with the exponential distribution:

  my.rexp.inverse <- function(N, lambda)
  {
    U <- runif(N, min=0, max=1)
    rnd.draws <- my.qexp.anal(U, lambda)
    return(rnd.draws)
  }
  my.rexp.inverse(10, 2)

##  [1] 0.10087 0.56874 0.79258 0.01962 1.28166 0.39451 2.17646 0.48650
##  [9] 0.97453 0.33054

Here is a histogram of 10,000 random numbers generated by the inverse transform sampling. The solid smooth line is the exponential PDF from which the random numbers were drawn:

  hist(my.rexp.inverse(10000,2), freq=FALSE, breaks=20,
       xlab="x", main=NULL, col="grey")
  #hist(rexp(10000, rate=2), freq=FALSE, breaks=20)
  curve(dexp(x, rate=2), add=TRUE)

Random number generator - rejection sampling

The advantage of rejection sampling is that you don't need the quantile function.
Notable texts on rejection sampling are provided by Jay Emerson, on Quantitations blog and on tsperry's blog. I will try to provide my own description and notation, which will hopefully complement the other posts.

In short, wou will need: (1) the so-called proposal probability densidy function which I will call , (2) the PDF from which you want to sample, here called , and (3) a constant which will satisfy . In other words, the curve must lay entirely above the curve, and ideally as close to as possible (you will waste less samples). The can be any PDF from which you are able to sample.

For my exponential example I will use simple uniform proposal density . The following figure illustrates my exponential , my proposal , and :

  x <- seq(0, 5, by=0.01)
  # my exponential PDF:
  p <- dexp(x, rate=2)
  # my PROPOSAL distribution:
  f <- dunif(x, min=0, max=5) # my proposal is uniform and hence
                              # I need to choose an arbitrary upper bound 5
  # the CONSTANT m
  m <- 11
  fm <- f*m

  plot(c(0,5), c(0,2.5), type="n", xlab="x", ylab="density")
  lines(x, fm, type="l", col="red")
  lines(x, f, col="red", lty=2)
  lines(x, p)

  legend("right", lty=c(1,2,1), 
         legend=c("f(x)*m", "f(x)", "p(x)"), 
         col=c("red", "red", "black"), lwd=2)

My uniform propsal distribution is not optimal – it will lead to high rejection rate and will require a lot of unnecessary computations. The uniform proposal will also truncate my samples at the max value of the uniform distribution. However, the uniform will still be ok for the illustration purposes here.

In rejection sampling the algorithm of one sampling step is:

• 1. Sample a point from .
• 2. Draw a vertical line from all the way up to .
• 3. Sample a point from a uniform density along the vertical line.
• 4. If accept the sample, otherwise go to 1.

This step is repeated until the desired number of accepted samples is reached.

Here is my rejection sampler for the exponential PDF:

  my.rexp.reject <- function(N, lambda, max)
  {
    samples <- numeric(N)
    m <- 12
    fxm <- dunif(0, 0, max)*m
    for (i in 1:N) 
    {
      repeat
      {
      # 1. sample from the propsal distribution:
        rx <- runif(1, 0, max)
      # 2. sample a point along a vertical line 
      #    at the rx point from Uniform(0, f(x)*m):
        rv <- runif(1, 0, fxm)
      # 3. evaluate the density of p(x) at rx
        dx <- dexp(rx, lambda)
      # 4. compare and accept if appropriate
        if (rv <= dx) 
        {
          samples[i] <- rx
          break        
        }
      }
    }
    return(samples)
  }

This is how 10,000 samples look like, together with the original PDF superimposed:

  x <- my.rexp.reject(N=10000, lambda=2, max=4)
  hist(x, freq=FALSE, breaks=30, col="gray", main=NULL)
  curve(dexp(x, 2), add=T)

Recently I had the pleasure of speaking about one of my favorite topics, Network Mapping. This is a continuation of a general theme I’ve previously discussed and involves the merger of statistical and multivariate data analysis results with a network.

Over the past year I’ve been working on two major tools, DeviumWeb and MetaMapR, which aid the process of biological data (metabolomic) network mapping.



DeviumWeb- is a shiny based GUI written in R which is useful for:

• data manipulation, transformation and visualization
• statistical analysis (hypothesis testing, FDR, power analysis, correlations, etc)
• clustering (heiarchical, TODO: k-means, SOM, distribution)
• principal components analysis (PCA)
• orthogonal partial least squares multivariate modeling (O-/PLS/-DA)

 


MetaMapR- is also a shiny based GUI written in R which is useful for calculation and visualization of various networks including:

• biochemical
• structural similarity
• mass spectral similarity
• correlation

Both of theses projects are under development, and my ultimate goal is to design a one-stop-shop ecosystem for network mapping.


In addition to network mapping,the video above and presentation below also discuss normalization schemes for longitudinal data and genomic, proteomic and metabolomic functional analysis both on a pathway and global level.

As always happy network mapping!

Data

Packages

First analysis

Second analysis

T

Third analysis

 

Data

Deal

First Model

Second model

Third model

Final model

Conclusion

R has some extremely useful utilities for profiling, such as system.time(), Rprof(), the often overlooked tracemem(), and the rbenchmark package. But if you want more than just simple timings of code execution, you will mostly have to look elsewhere.

One of the best sources for profiling data is hardware performance counters, available in most modern hardware. This data can be invaluable to understanding what a program is really doing. The Performance Application Programming Interface (PAPI) library is a well-known profiling library, and allows users to easily access this profiling data. So we decided to bring PAPI to R. It's available now in the package pbdPAPI, and is supported in part as a 2014 Google Summer of Code project (thanks Googs!).

So what can you do with it? I'll show you.

How Many Numerical Operations does a PCA Need?

Flops, or "floating point operations per second" is an important measurement of performance of some kinds of programs. A very famous benchmark known as the LINPACK benchmark is a measurement of the flops of a system solving a system of linear equations using an LU decomposition with partial pivoting. You can see current and historical data for supercomputer performance on the LINPACK Benchmark, or even see how your computer stacks up with the biggest computers in the world at Top 500.

For an example, let's turn to the pbdPAPI package's Principal Components Analysis demo. This demo measures the number of floating point operations (things like addition, subtraction, multiplication, and division) executed by your compter to perform a PCA, and compares it against the number of operations theoretically required to compute a PCA. This theoretical value is determined by evaluating the different compute kernels that make up a PCA. For an mxn matrix with PCA computed via SVD of the data matrix (as in R's prcomp()), we need:

• 2mn + 1 operations to center the data.
• 6mn^2 + 20n^3 operations for the SVD.
• 2mn^2 operations for the projection onto the right singular vectors (the retx=TRUE part).

We only add the count for centering (and not scaling), because that's the R default (for some reason...). For more details, see Golub and Van Loan's "Matrix Computations".

An example output from running the demo on this machine is:

      m  n  measured theoretical difference pct.error   mflops
1 10000 50 212563800   203500001    9063799  4.264037 2243.717

So pbdPAPI measured 212.6 million floating point operations, while the theoretical number is 203.5 million. That difference is actually quite small, and seems fairly reasonable. Also note that we clock in at around 2.2 Gflops (double precision). And we achieve all of this with a simple system.flops() call from pbdPAPI:

library(pbdPAPI)

m <- 10000
n <- 50
x <- matrix(rnorm(m*n), m, n)

flops <- system.flops(prcomp(x, center=FALSE, scale.=FALSE))

Mathematically Equivalent, but Computationally Different Operations

Another interesting thing you can do with pbdPAPI is easily measure cache misses. Remember when some old grumpy jerk told you that "R matrices are column-major"? Or that, when operating on matrices, you should loop over columns first, then rows? Why is that? Short answer: computers are bad at abstraction. Long answer: cache.

If you're not entirely familiar with CPU caches, I would encourage you to take a gander at our spiffy vignette. But the quick summary is that lots of cache misses is bad. To understand why, you might want to take a look at this interactive visualization of memory access speeds.

To show off how this works, we're going to measure the cache misses of a simple operation: allocate a matrix and set all entries to 1. We're going to use Rcpp to do this, mostly because measuring the performance of for loops in R is too depressing.

First, let's do this by looping over rows and then columns. Said another way, we fix a row and and fill all of its entries with 1 before moving to the next row:

SEXP rows_first(SEXP n)
{
  int i, j;
  const int n = INTEGER(n_)[0];
  Rcpp::NumericMatrix x(n, n);

  for (i=0; i<n; i++)
    for (j=0; j<n; j++)
      x(i, j) = 1.;

  return x;
}

Next, we'll loop over columns first, then rows. Here we fix a column and fill each row's entry in that column with 1 before proceeding:

SEXP cols_first(SEXP n)
{
  int i, j;
  const int n = INTEGER(n_)[0];
  Rcpp::NumericMatrix x(n, n);

  for (j=0; j<n; j++)
    for (i=0; i<n; i++)
      x(i, j) = 1.;

  return x;
}

Assuming these have been compiled for use with R, say with the first as bad() and the second as good(), we can easily measure the cache misses like so:

library(pbdPAPI)

n <- 10000L

system.cache(bad(n))
system.cache(good(n))

Again using this machine as a reference we get:

$`Level 1 cache misses`
[1] 202536304

$`Level 2 cache misses`
[1] 168382934

$`Level 3 cache misses`
[1] 21552970

for bad(), and:

$`Level 1 cache misses`
[1] 15771212

$`Level 2 cache misses`
[1] 1889270

$`Level 3 cache misses`
[1] 1286338

for good().  Just staring at these huge values may not be easy on the eyes, so here's a plot showing this same information:

Here, lower is better, and so the clear winner is, as the name implies, good(). Another valuable measurement is the ratio of total cache misses (data and instruction) to total cache accesses.  Again, with pbdPAPI, measuring this is trivial:

system.cache(bad(n), events="l2.ratio")
system.cache(good(n), events="l2.ratio")

On this machine, we see:

L2 cache miss ratio 
          0.8346856 
L2 cache miss ratio 
           0.112331 

Here too, lower is better, and so we again see a clear winner. The full source for this example is available here.

Wrapup

pbdPAPI can measure much, much more than just flops and cache misses. See the package vignette for more information about what you can measure with pbdPAPI. The package is available now on GitHub and is permissively licensed under the BSD 2-clause license, and it will come to CRAN eventually.

Ok now the downside; at the moment, it doesn't work on Windows or Mac.

We have spent the last month working on extending support to Windows and/or Mac, but it's not entirely trivial for a variety of reasons, as PAPI itself only supports Linux and FreeBSD at this time. We are committed to platform independence, and I believe we'll get there soon, in some capacity. But for now, it works fantastically on your friendly neighborhood Linux cluster.

Finally, a quick thanks again to the Googs, and also thanks to the folks who run the R organization for Google Summer of Code, especially Brian. And thanks to our student, who I think is doing a great job so far.

Annual R User Conference 2014

What's H2O?

Deep Learning in R

The H2O Experiment

Findings

Setting Up and Connecting to a H2O Cluster

Loading Data

In order to train models with the H2O engine, I need to link the datasets to the H2O cluster first. There are many ways to do it. In this case, I linked a data frame (Breast Cancer) and imported CSVs (MNIST) using the following code.

Training a Deep Neural Network Model

Using the Model for Prediction

Out-of-Bag Performance (Breast Cancer Dataset)

Memory Usage (MNIST Dataset)

Conclusions

Personal Highlight of useR! 2014

Mateusz Żółtak asked me to spread the word about his new R package for parameterized SQL queries. Below you can find the copy of package vignette. If you work with SQL in R you may find it useful.

library(RODBC)

connHandle <- odbcConnect("cakesDatabase")
newData <- read.csv("newData.csv", stringsAsFactors = F)

for(row in 1:nrow(newData)){
  query <- paste0(
    "UPDATE cakes 
     SET price = ", newData$price[row], " 
     WHERE cake = '", newData$cake[row], "'"
  )
  sqlQuery(connHandle, query)
}

odbcClose(connHandle)

library(RODBC)

connHandle <- odbcConnect('studentsDatabase')
newStudents <- read.csv('newStudents.csv', stringsAsFactors = F)

for(row in 1:nrow(newStudents)){
  query <- paste0(
    "INSERT INTO students (first_name, last_name)
     VALUES (
       '", newStudents$first_name[row],"', 
       '", newStudents$last_name[row],"', 
     )"
  )
  sqlQuery(P, query)
}

odbcClose(connHandle)

INSERT INTO students (last_name, first_name)
  VALUES ('Smith', 'Robert'); DROP TABLE students; --')

library(RODBCext)

connHandle <- odbcConnect("cakesDatabase")
newData <- read.csv("newData.csv", stringsAsFactors = F)

query <- "UPDATE cakes SET price = ? WHERE cake = ?"
for(row in 1:nrow(newData)){
  sqlExecute(connHandle, query, newData[i, ])
}

odbcClose(connHandle)

library(RODBCext)

connHandle <- odbcConnect("cakesDatabase")
newData <- read.csv("newData.csv", stringsAsFactors = F)

query <- "UPDATE cakes SET price = ? WHERE cake = ?"
sqlExecute(connHandle, query, newData)

odbcClose(connHandle)

library(RODBCext)
connHandle <- odbcConnect('EWD') # my sample ODBC database
data <- data.frame(1:10000, letters[rep(1:10, 1000)])

# Ordinary query - paste0() called in every loop
system.time({
  for(row in 1:nrow(data)){
    query <- paste0("INSERT INTO my_table VALUES (", data[row, 1], "'", data[row, 2],"')")
    sqlQuery(connHandle, query)
  }
})
#   user  system elapsed 
#  5.384   2.288  16.397

# Ordinary query - paste0() called only once
system.time({
  queries <- paste0(
    "INSERT INTO my_table VALUES (", data[, 1], "'", data[, 2],"')"
  )
  for(query in queries){
    sqlQuery(connHandle, query)
  }
})
#   user  system elapsed 
#  2.088   2.028   7.255 

# Parameterized query
system.time({
  sqlExecute(connHandle, "INSERT INTO my_table VALUES (?, ?)", data)
})
#   user  system elapsed 
#  0.300   0.232   3.935 
odbcClose(connHandle)

library(RODBCext)
connHandle <- odbcConnect('EWD') # my sample ODBC database

pupils = sqlQuery(
  connHandle, "SELECT id_obserwacji FROM obserwacje LIMIT 10000", 
  stringsAsFactors = F
)[, 1]

# Ordinary query - paste0() called in every loop
system.time({
  for(i in pupils){
    query <- paste0(
      "SELECT count(*) 
       FROM testy_obserwacje JOIN testy USING (id_testu) JOIN arkusze USING (arkusz) 
       WHERE id_obserwacji = ", pupils[i]
    )
    tmp <- sqlQuery(connHandle, query)
    # some other computations here
  }
})
#   user  system elapsed 
# 10.896   1.508  61.424 

# Ordinary query - paste0() called only once
system.time({
  queries <- paste0(
    "SELECT count(*) 
     FROM testy_obserwacje JOIN testy USING (id_testu) JOIN arkusze USING (arkusz) 
     WHERE id_obserwacji = ", pupils
  )
  for(query in queries){
    tmp <- sqlQuery(connHandle, query)
    # some other computations here
  }
})
#   user  system elapsed 
# 11.016   1.108  51.766 

# Parameterized query
system.time({
  query = "
    SELECT count(*) 
    FROM testy_obserwacje JOIN testy USING (id_testu) JOIN arkusze USING (arkusz) 
    WHERE id_obserwacji = ?"
  sqlPrepare(connHandle, query)
  for(i in pupils){
    tmp = sqlExecute(connHandle, NULL, pupils[i], fetch=T)
    # some other computations here
  }
})
#   user  system elapsed 
# 12.140   0.312  26.468

library(RODBCext)
connHandle <- odbcConnect("myDatabase")

# good old RODBC call
data <- sqlQuery(connHandle, "SELECT * FROM myTable WHERE column = 'myValue'") 
# RODBCext equivalent
data <- sqlExecute(connHandle, "SELECT * FROM myTable WHERE column = ?", 'myValue', fetch = TRUE) 

odbcClose(connHandle)

library(RODBCext)
connHandle <- odbcConnect("myDatabase")

filterData <- data.frame('column1' = 1:5, column2 = c('a', 'b', 'c', 'd', 'e'))
data <- sqlExecute(connHandle, "SELECT * FROM myTable WHERE column1 = ? AND column2 = ?", filterData, fetch = TRUE)

odbcClose(connHandle)

library(RODBCext)
connHandle <- odbcConnect("myDatabase")

sqlExecute(connHandle, "SELECT * FROM myTable WHERE column = ?", 'myValue', fetch = FALSE)
data <- sqlGetResults(connHandle, max = 10) # fetch no more than 10 first rows
# data processing comes here
data <- sqlGetResults(connHandle) # fetch all other rows

odbcClose(connHandle)

library(RODBCext)
connHandle <- odbcConnect("myDatabase")

sqlExecute(
  connHandle, "SELECT * FROM myTable WHERE column = ?", 'myValue', 
  fetch = TRUE, stringsAsFactors = FALSE, dec = ",", max = 50, as.is = TRUE
)

odbcClose(connHandle)

library(RODBCext)
connHandle <- odbcConnect('myDatabase') 

sqlPrepare(connHandle, "SELECT * FROM myTable WHERE column = ?") # prepare query

# for some reason (e.g. resources limits) data must be processed sequentialy
foreach(i in observations){
  data = sqlExecute(connHandle, NULL, i$column, fetch=T)
  # data processing for a given observations goes here
}
odbcClose(connHandle)

// add bootstrap table styles to pandoc tables $(document).ready(function () { $('tr.header').parent('thead').parent('table').addClass('table table-condensed'); });

The Interview

In the video above, Max provides some amazing insights into the why and how of caret, an R package he created. He also discusses his book on Applied Predictive Modeling which he co-authored with Kjell Johnson, including details on how he set out to write the book he wished he would have had. As a special bonus, Max also describes Quinlan’s C5.0, an alternate “forest of decision trees” algorithm, the secrets of which were hidden behind commercial licensing for years – and which has recently been ported and made available to the R ecosystem. Whether you are a beginner just getting your feet wet with R and predictive modeling, or a seasoned data scientist, this interview has something for everyone.

Expanding Your Superpowers with Caret

“What is your favorite superpower?” is a classic icebreaker, and the answers will tell you quite a lot about the people answering. Someone in the group will immediately claim that the ability to fly is paramount. Someone else invariably brings up invisibility. Super strength, super speed, laser vision, the ability to talk to sea creatures – these are all fine choices. For me, however, the best answer has always been the ability to predict the future. No other superpower seems to compare – if you can predict the future, then you know how a Flying Superhero will attack, and that you’ll need to bring a mirror to battle Laser-Vision Man. If you can predict the future, perhaps you’ll skip out on that whale watching adventure if you know Sea-Creatures Guy has it out for you. Considering the (current) impossibility of choosing one’s superpower, it’s a fun thought exercise but not much else.

In the context of data science, however, there may be a little something to be done about this “predicting the future” thing, and maybe, just maybe, Max Kuhn is the guy to show you how to do it.

A non-clinical statistician, is kind of exactly what it sounds like. – Max Kuhn

Max is the Director of Nonclinical Statistics at Pfizer, a position that involves supporting a great many scientists with software tools, analysis, and machine learning during the creation and validation of molecules in the pipeline to become potentially life-saving and life-giving medicines. He is also the creator of the caret package for the R language. caret (short for Classification And REgression Training) is a set of functions which attempt to streamline the process of creating predictive models. Put succintly, the caret package provides the ‘train’ function. This function is your gateway to nearly every awesome machine learning model that can be implemented in R. Want to train a neural network to predict Species from all other variables in the iris data set?

train(Species ~ ., data=iris, method="nnet”)

Turns out that a neural network didn’t provide the accuracy you wanted and instead you decide to try out a more powerful machine learning technique, like random forests?

train(Species ~ ., data=iris, method="rf")

Perhaps you’re willing to trade a little bit of predictive power in exchange for interpretability, in which case you’d switch over to traditional decision trees.

train(Species ~ ., data=iris, method="rpart")

That’s all it takes to get started. That’s it.

Inside of the train function, Max has taken his combined decades of experience and expertise creating predictive models and has hidden that complexity for the sake of usability. Each method has its own series of smart defaults and behavior, so that even if you only stick to the basics you can still hit the ground running and be productive. However, the caret package contains much, much more than just the train function.

Inside of the package, Max has encoded best-practice approaches for handling those pitfalls that both new and experienced data scientists might face. Perennial questions such as ‘How do you handle unbalanced classes?’ are answered in caret, providing functions to create balanced data partitions. How do you approach feature selection? Caret is helpful and provides recursive feature elimination. How do you make sure that scaling/centering/PCA pre-processing are properly handled during your cross-validation steps and that they don’t add bias to your results? Caret has your back. How do you test your newly-trained model on a held-back training set and view the accuracy metrics? Caret has a a buffet of options waiting patiently at your fingertips. Suddenly, you can start to predict the future… and you’re certainly a little closer to having a superpower with caret in your toolbox.

From my perspective the most important event that happened at useR! 2014 was that I got to meet the 0xdata team and now, long story short, here I am introducing the latest version of H₂O, labeled Lagrange (2.6.0.11), to the R and greater data science communities. Before joining 0xdata, I was working at a competitor on a rival project and was repeatedly asked why my generalized linear model analytic didn’t run as fast as H₂O’s GLM. The answer then as it is now is the same — because H₂O has a cutting edge distributed in-memory parallel computing architecture — but I no longer receive an electric shock every time I say so.

For those hearing about H₂O for the first time, it is an open-source distributed in-memory data analysis tool designed for extremely large data sets and the H₂O Lagrange (2.6.0.11) release provides scalable solutions for the following analysis techniques:

• Generalized Linear Model
• K-Means
• Random Forest
• Principal Components Analysis
• Summary
• Gradient Boosted Regression and Classification
• Naive Bayes
• Deep Learning

In my first blog post at 0xdata, I wanted to keep it simple and make sure R users know how to get the h2o package, which is cross-referenced on the High-Performance and Parallel Computing and Machine and Statistical Learning CRAN Task Views, up and running on their computers. To so do, open an R console of your choice and type

# Download, install, and initialize the H2O package
install.packages("h2o",
                 repos = c("http://h2o-release.s3.amazonaws.com/h2o/rel-lagrange/11/R", getOption("repos")))
library(h2o)
localH2O <- h2o.init()

# List and run some demos to see H2O at work
demo(package = "h2o")
demo(h2o.glm)
demo(h2o.deeplearning)

After you are done experimenting with the demos in R, you can open up a web browser to http://localhost:54321/ to give the H₂O web interface a once over and then hop over to 0xdata’s YouTube channel for some in-depth talks.

Over the coming weeks we at 0xdata will continue to blog about how to use H₂O through R and other interfaces. If there is a particular use case you would like to see addressed, join our h2ostream Google Groups conversation or e-mail us at support@0xdata.com. Until then, happy analyzing.

Related Blogs

R-bloggers

Data

Algorithm

Scaling

Unique solution

Code

Results

Product space

Descriptor space

Additional code

data

biplot

product plot