Lets imagine we have a 10 GB flat data file and that we want to select certain rows based on a given criteria. This requires a sequential read across the entire data set.

If we can store the file in memory:

• \(10~GB \times (250~\mu s / 1~MB) = 2.5\) seconds

If we have to access the file from an SSD:

• \(10~GB \times (1~ms / 1~MB) = 10\) seconds

If we have to access the file from a spinning disk:

• \(10~GB \times (30~ms / 1~MB) = 300\) seconds

This is just for reading data, if we make any modifications (writing) things are much worse. Also, this is actually incredibly optimistic since it assumes that all the data on disk can be read in one continuous read.

A simple but useful trick is to only work with the full data set when you absolutely have to. We are generally interested in the large scale patterns / features of the data and these are generally preserved by randomly sampling that data (this keeps statisticians employed).

library(dplyr)
library(pryr)

set.seed(20170330)
flights_sub = flights %>% sample_frac(0.1)

object.size(flights)

## 953617184 bytes

object_size(flights)

## 954 MB

object.size(flights_sub)

## 98427752 bytes

object_size(flights_sub)

## 98.4 MB

We can do better (better precision) using the microbenchmark package

library(microbenchmark)

d = abs(rnorm(1000))
r = microbenchmark(
      exp(log(d)/2),
      d^0.5,
      sqrt(d),
      times = 1000
    )
print(r)

## Unit: microseconds
##           expr    min     lq      mean  median     uq      max neval
##  exp(log(d)/2) 14.848 15.439 20.004378 17.9640 19.167  219.591  1000
##          d^0.5 24.195 24.582 32.358537 27.2815 28.820 2003.781  1000
##        sqrt(d)  2.337  2.775  5.910073  5.1995  5.718  894.631  1000

We can also use the rbenchmark package

library(rbenchmark)

d = abs(rnorm(1000))
benchmark(
  exp(log(d)/2),
  d^0.5,
  sqrt(d),
  replications = 1000,
  order = "relative"
)

##            test replications elapsed relative user.self sys.self user.child sys.child
## 3       sqrt(d)         1000   0.004     1.00     0.004    0.000          0         0
## 1 exp(log(d)/2)         1000   0.025     6.25     0.023    0.000          0         0
## 2         d^0.5         1000   0.029     7.25     0.029    0.001          0         0

for_grow = function(x) {
  res = c()
  for(x_i in x) 
    res = c(res,sqrt(x_i))
  res
}

for_init = function(x) {
  res = rep(NA, length(x))
  for(i in seq_along(x))
    res[i] = sqrt(x[i])
}

vect = function(x) {
  sqrt(x)
}

dplyr_mutate = function(x) {
  data.frame(x=x) %>% mutate(s = sqrt(x))
}

dplyr_transmute = function(x) {
  data.frame(x=x) %>% transmute(s = sqrt(x))
}

library(rbenchmark)

d = abs(rnorm(1000))
benchmark(
  for_grow(d),
  for_init(d),
  vect(d),
  dplyr_mutate(d),
  dplyr_transmute(d),
  replications = 100,
  relative = "elapsed"
) %>% 
  arrange(elapsed) %>% 
  select(test:relative)

##                 test replications elapsed relative
## 1            vect(d)          100   0.001        1
## 2    dplyr_mutate(d)          100   0.077       77
## 3        for_init(d)          100   0.097       97
## 4 dplyr_transmute(d)          100   0.099       99
## 5        for_grow(d)          100   0.290      290

d = abs(rnorm(10000))
benchmark(
  for_grow(d),
  for_init(d),
  vect(d),
  dplyr_mutate(d),
  dplyr_transmute(d),
  replications = 100,
  relative = "elapsed"
) %>% 
  arrange(elapsed) %>% 
  select(test:relative)

##                 test replications elapsed  relative
## 1            vect(d)          100   0.003     1.000
## 2    dplyr_mutate(d)          100   0.060    20.000
## 3 dplyr_transmute(d)          100   0.090    30.000
## 4        for_init(d)          100   0.862   287.333
## 5        for_grow(d)          100  36.920 12306.667

With large data sets overplotting is often a serious issue,

plot(AirTime ~ Distance, data = flights_small)

This can be addresseds with transparency (alpha) and plot character sizes (or binning)

The different parts of the raw data or any additional outside data you find may need to be merged together. This can be down with dplyr's join functions or base R's merge - but it important to think about the nature of the relationship between the two data sets.

• One to one - each entry in A corresponds to either 0 or 1 entries in B

• One to many or many to one - each entry in A corresponds to 0 or more entries in B, or vice versa.

• Many to many - each entry in A corresponds to 0 or more entries in B and each entry in B corresponds to 0 or more entries in A.

addr = data.frame(name = c("Alice","Alice", "Bob","Bob"),
                  email= c("alice@company.com","alice@gmail.com",
                           "bob@company.com","bob@hotmail.com"),
                  stringsAsFactors = FALSE)

phone = data.frame(name = c("Alice","Alice", "Bob","Bob"),
                   phone= c("919 555-1111", "310 555-2222", 
                            "919 555-3333", "310 555-3333"),
                   stringsAsFactors = FALSE)

library(dplyr)
full_join(addr, phone, by="name")

##    name             email        phone
## 1 Alice alice@company.com 919 555-1111
## 2 Alice alice@company.com 310 555-2222
## 3 Alice   alice@gmail.com 919 555-1111
## 4 Alice   alice@gmail.com 310 555-2222
## 5   Bob   bob@company.com 919 555-3333
## 6   Bob   bob@company.com 310 555-3333
## 7   Bob   bob@hotmail.com 919 555-3333
## 8   Bob   bob@hotmail.com 310 555-3333

A = data.frame(common="C", A_values=1:1000)
B = data.frame(common="C", B_values=1:1000)

inner_join(A,B) %>% tbl_df()

## Joining, by = "common"

## # A tibble: 1,000,000 × 3
##    common A_values B_values
##    <fctr>    <int>    <int>
## 1       C        1        1
## 2       C        1        2
## 3       C        1        3
## 4       C        1        4
## 5       C        1        5
## 6       C        1        6
## 7       C        1        7
## 8       C        1        8
## 9       C        1        9
## 10      C        1       10
## # ... with 999,990 more rows

Some advice given by Mark Suchard at UCLA (unsure of who said this originally)

• Linear complexity \((\mathcal{O}(n))\) - Great

  • Examples: Vectorized sqrt, lm

• Quadratic complexity \((\mathcal{O}(n^2))\) - Pray

  • Examples: Bubble sort, dist

• Cubic complexity \((\mathcal{O}(n^3))\) - Give up

  • Examples: %*%, solve, chol