March 28, 2016

Set up & Configuration

Datasets, presentation, source code

  • That data we will be using today is available via the nycflights13 package.

    • In R, run
install.packages("nycflights13")


Welcome to the Hadleyverse

For today's session we will be using a number of R packages that have been developed by Hadley Wickham.

All of these packages are available on CRAN, before starting we want to make sure that everyone is using the most current available versions.

install.packages(c("dplyr","readr","magrittr"))


Also - now is a good time to check that your version of R is up-to-date. If you are not running version 3.2.3 or later it is time to update. After updating R be sure to also update your packages.

readr

readr

This is a recent package that is designed to be a fast and friendly way of reading tabular data into R.


Core features:

  • Faster than base R (~4x)
  • No strings as factors
  • No column name mangling
  • Consistent argument/function naming scheme
  • Plays nice with dplyr (tbl_df)
  • Progress bars

Example (csv)

library(readr)
library(dplyr)
(mtcars = read_csv(system.file("extdata/mtcars.csv", package = "readr")))
## Source: local data frame [32 x 11]
## 
##      mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
##    (dbl) (int) (dbl) (int) (dbl) (dbl) (dbl) (int) (int) (int) (int)
## 1   21.0     6 160.0   110  3.90 2.620 16.46     0     1     4     4
## 2   21.0     6 160.0   110  3.90 2.875 17.02     0     1     4     4
## 3   22.8     4 108.0    93  3.85 2.320 18.61     1     1     4     1
## 4   21.4     6 258.0   110  3.08 3.215 19.44     1     0     3     1
## 5   18.7     8 360.0   175  3.15 3.440 17.02     0     0     3     2
## 6   18.1     6 225.0   105  2.76 3.460 20.22     1     0     3     1
## 7   14.3     8 360.0   245  3.21 3.570 15.84     0     0     3     4
## 8   24.4     4 146.7    62  3.69 3.190 20.00     1     0     4     2
## 9   22.8     4 140.8    95  3.92 3.150 22.90     1     0     4     2
## 10  19.2     6 167.6   123  3.92 3.440 18.30     1     0     4     4
## ..   ...   ...   ...   ...   ...   ...   ...   ...   ...   ...   ...

str(mtcars)
## Classes 'tbl_df', 'tbl' and 'data.frame':    32 obs. of  11 variables:
##  $ mpg : num  21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
##  $ cyl : int  6 6 4 6 8 6 8 4 4 6 ...
##  $ disp: num  160 160 108 258 360 ...
##  $ hp  : int  110 110 93 110 175 105 245 62 95 123 ...
##  $ drat: num  3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
##  $ wt  : num  2.62 2.88 2.32 3.21 3.44 ...
##  $ qsec: num  16.5 17 18.6 19.4 17 ...
##  $ vs  : int  0 0 1 1 0 1 0 1 1 1 ...
##  $ am  : int  1 1 1 0 0 0 0 0 0 0 ...
##  $ gear: int  4 4 4 3 3 3 3 4 4 4 ...
##  $ carb: int  4 4 1 1 2 1 4 2 2 4 ...

(mtcars = read_csv(system.file("extdata/mtcars.csv.zip", package = "readr")))
## Source: local data frame [32 x 11]
## 
##      mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
##    (dbl) (int) (dbl) (int) (dbl) (dbl) (dbl) (int) (int) (int) (int)
## 1   21.0     6 160.0   110  3.90 2.620 16.46     0     1     4     4
## 2   21.0     6 160.0   110  3.90 2.875 17.02     0     1     4     4
## 3   22.8     4 108.0    93  3.85 2.320 18.61     1     1     4     1
## 4   21.4     6 258.0   110  3.08 3.215 19.44     1     0     3     1
## 5   18.7     8 360.0   175  3.15 3.440 17.02     0     0     3     2
## 6   18.1     6 225.0   105  2.76 3.460 20.22     1     0     3     1
## 7   14.3     8 360.0   245  3.21 3.570 15.84     0     0     3     4
## 8   24.4     4 146.7    62  3.69 3.190 20.00     1     0     4     2
## 9   22.8     4 140.8    95  3.92 3.150 22.90     1     0     4     2
## 10  19.2     6 167.6   123  3.92 3.440 18.30     1     0     4     4
## ..   ...   ...   ...   ...   ...   ...   ...   ...   ...   ...   ...

Pipes

magrittr

    

This is an idea that has been around for a long time - particularly in the unix ecosystem.

  • Complex tasks can be accomplished by stringing together many simple tools - syntax is important for efficiency / readability

  • The idea of pipes is that the result of the previous command is passed to the next command, which allows many tools to be strung together - from left to right.

  • In R, Magrittr introduces the %>% pipe operator which runs the expression of the left and passes the result as the first argument to expression on the right.

An example

You can think about the following sequence of actions we want to perform:

  • find key
  • start car with keys
  • drive car to campus
  • park car

Expressed as a set of nested functions in R pseudocode this would look like:

park(drive(start_car(find("keys")), to="campus"))

Writing it out using pipes give it a more natural (and easier to read) structure:

find("keys") %>%
    start_car() %>%
    drive(to="campus") %>%
    park()

Approaches

All of the following are fine, it amounts to preference.


Nested:

h( g( f(x), y=1), z=1 )

Piped:

f(x) %>% g(y=1) %>% h(z=1)

Intermediate:

res = f(x)
res = g(res, y=1)
res = h(res, z=1)

What about other arguments?

Sometimes we want to send our results to a function argument other than first one or we want to use the previous result for multiple arguments. In these cases we refer to the previous result using ..

data.frame(a=1:3,b=3:1) %>% lm(a~b,data=.)
## 
## Call:
## lm(formula = a ~ b, data = .)
## 
## Coefficients:
## (Intercept)            b  
##           4           -1
data.frame(a=1:3,b=3:1) %>% .[[1]]
## [1] 1 2 3
data.frame(a=1:3,b=3:1) %>% .[[length(.)]]
## [1] 3 2 1

dplyr

A Grammar of Data Manipulation

dplyr is based on the concepts of functions as verbs that manipulate data frames.

Single data frame functions / verbs:

  • tbl_df(): add the tbl_df class to a data frame
  • filter(): pick rows matching given criteria
  • slice(): pick rows by index(es)
  • select(): pick columns by name
  • rename(): rename column(s)
  • arrange(): sort / reorder row(s)
  • mutate(): add new variables / columns
  • transmute(): create new data frame
  • distinct(): filter for unique rows
  • sample_n() / sample_frac(): randomly sample rows
  • summarise(): reduce variables to values
  • group_by(): indicates values to group by when summarizing

dplyr rules

  • First argument is a data frame

  • Subsequent arguments say what to do with data frame

  • Always return a data frame

  • Don't modify in place

Example Data

We will demonstrate dplyr's functionality using the nycflights13 data.

str(flights)
## Classes 'tbl_df', 'tbl' and 'data.frame':    336776 obs. of  16 variables:
##  $ year     : int  2013 2013 2013 2013 2013 2013 2013 2013 2013 2013 ...
##  $ month    : int  1 1 1 1 1 1 1 1 1 1 ...
##  $ day      : int  1 1 1 1 1 1 1 1 1 1 ...
##  $ dep_time : int  517 533 542 544 554 554 555 557 557 558 ...
##  $ dep_delay: num  2 4 2 -1 -6 -4 -5 -3 -3 -2 ...
##  $ arr_time : int  830 850 923 1004 812 740 913 709 838 753 ...
##  $ arr_delay: num  11 20 33 -18 -25 12 19 -14 -8 8 ...
##  $ carrier  : chr  "UA" "UA" "AA" "B6" ...
##  $ tailnum  : chr  "N14228" "N24211" "N619AA" "N804JB" ...
##  $ flight   : int  1545 1714 1141 725 461 1696 507 5708 79 301 ...
##  $ origin   : chr  "EWR" "LGA" "JFK" "JFK" ...
##  $ dest     : chr  "IAH" "IAH" "MIA" "BQN" ...
##  $ air_time : num  227 227 160 183 116 150 158 53 140 138 ...
##  $ distance : num  1400 1416 1089 1576 762 ...
##  $ hour     : num  5 5 5 5 5 5 5 5 5 5 ...
##  $ minute   : num  17 33 42 44 54 54 55 57 57 58 ...

tbl_df()?

flights
## Source: local data frame [336,776 x 16]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1   2013     1     1      517         2      830        11      UA  N14228   1545    EWR
## 2   2013     1     1      533         4      850        20      UA  N24211   1714    LGA
## 3   2013     1     1      542         2      923        33      AA  N619AA   1141    JFK
## 4   2013     1     1      544        -1     1004       -18      B6  N804JB    725    JFK
## 5   2013     1     1      554        -6      812       -25      DL  N668DN    461    LGA
## 6   2013     1     1      554        -4      740        12      UA  N39463   1696    EWR
## 7   2013     1     1      555        -5      913        19      B6  N516JB    507    EWR
## 8   2013     1     1      557        -3      709       -14      EV  N829AS   5708    LGA
## 9   2013     1     1      557        -3      838        -8      B6  N593JB     79    JFK
## 10  2013     1     1      558        -2      753         8      AA  N3ALAA    301    LGA
## ..   ...   ...   ...      ...       ...      ...       ...     ...     ...    ...    ...
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute (dbl)

filter() - March flights

flights %>% filter(month == 3)
## Source: local data frame [28,834 x 16]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1   2013     3     1        4       125      318       142      B6  N706JB     11    JFK
## 2   2013     3     1       50        52      526        48      B6  N794JB    707    JFK
## 3   2013     3     1      117       152      223       149      B6  N328JB    608    JFK
## 4   2013     3     1      454        -6      633       -15      US  N177US   1117    EWR
## 5   2013     3     1      505       -10      746       -24      UA  N527UA    475    EWR
## 6   2013     3     1      521        -9      813       -14      UA  N76523   1714    LGA
## 7   2013     3     1      537        -3      856         6      AA  N5EPAA   1141    JFK
## 8   2013     3     1      541        -4     1014        -9      B6  N653JB    725    JFK
## 9   2013     3     1      549       -11      639       -24      US  N749US   2114    LGA
## 10  2013     3     1      550       -10      747       -14      EV  N760EV   4911    EWR
## ..   ...   ...   ...      ...       ...      ...       ...     ...     ...    ...    ...
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute (dbl)

filter() - Flights in the first 7 days of March

flights %>% filter(month == 3, day <= 7)
## Source: local data frame [6,530 x 16]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1   2013     3     1        4       125      318       142      B6  N706JB     11    JFK
## 2   2013     3     1       50        52      526        48      B6  N794JB    707    JFK
## 3   2013     3     1      117       152      223       149      B6  N328JB    608    JFK
## 4   2013     3     1      454        -6      633       -15      US  N177US   1117    EWR
## 5   2013     3     1      505       -10      746       -24      UA  N527UA    475    EWR
## 6   2013     3     1      521        -9      813       -14      UA  N76523   1714    LGA
## 7   2013     3     1      537        -3      856         6      AA  N5EPAA   1141    JFK
## 8   2013     3     1      541        -4     1014        -9      B6  N653JB    725    JFK
## 9   2013     3     1      549       -11      639       -24      US  N749US   2114    LGA
## 10  2013     3     1      550       -10      747       -14      EV  N760EV   4911    EWR
## ..   ...   ...   ...      ...       ...      ...       ...     ...     ...    ...    ...
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute (dbl)

filter() - Flights to LAX or RDU in March

flights %>% filter(dest == "LAX" | dest == "RDU", month==3)
## Source: local data frame [1,935 x 16]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1   2013     3     1      607        -3      832       -53      UA  N557UA    797    JFK
## 2   2013     3     1      608        -7      737       -13      MQ  N723MQ   4518    LGA
## 3   2013     3     1      623        -7      753       -17      EV  N14959   5667    EWR
## 4   2013     3     1      629        -3      844       -68      UA  N77518   1702    EWR
## 5   2013     3     1      657        -3      953       -41      DL  N712TW    763    JFK
## 6   2013     3     1      714        -1      939       -58      B6  N804JB    671    JFK
## 7   2013     3     1      716         6      958       -37      VX  N636VA    399    JFK
## 8   2013     3     1      727        -3     1007       -53      AA  N319AA     33    JFK
## 9   2013     3     1      803        -7      923       -32      MQ  N853MQ   4406    JFK
## 10  2013     3     1      823        -1      954       -20      EV  N22909   4548    EWR
## ..   ...   ...   ...      ...       ...      ...       ...     ...     ...    ...    ...
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute (dbl)

slice() - First 10 flights

flights %>% slice(1:10)
## Source: local data frame [10 x 16]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1   2013     1     1      517         2      830        11      UA  N14228   1545    EWR
## 2   2013     1     1      533         4      850        20      UA  N24211   1714    LGA
## 3   2013     1     1      542         2      923        33      AA  N619AA   1141    JFK
## 4   2013     1     1      544        -1     1004       -18      B6  N804JB    725    JFK
## 5   2013     1     1      554        -6      812       -25      DL  N668DN    461    LGA
## 6   2013     1     1      554        -4      740        12      UA  N39463   1696    EWR
## 7   2013     1     1      555        -5      913        19      B6  N516JB    507    EWR
## 8   2013     1     1      557        -3      709       -14      EV  N829AS   5708    LGA
## 9   2013     1     1      557        -3      838        -8      B6  N593JB     79    JFK
## 10  2013     1     1      558        -2      753         8      AA  N3ALAA    301    LGA
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute (dbl)

slice() - Last 5 flights

flights %>% slice((n()-5):n())
## Source: local data frame [6 x 16]
## 
##    year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##   (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1  2013     9    30       NA        NA       NA        NA      EV  N740EV   5274    LGA
## 2  2013     9    30       NA        NA       NA        NA      9E           3393    JFK
## 3  2013     9    30       NA        NA       NA        NA      9E           3525    LGA
## 4  2013     9    30       NA        NA       NA        NA      MQ  N535MQ   3461    LGA
## 5  2013     9    30       NA        NA       NA        NA      MQ  N511MQ   3572    LGA
## 6  2013     9    30       NA        NA       NA        NA      MQ  N839MQ   3531    LGA
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute (dbl)

select() - Individual Columns

flights %>% select(year, month, day)
## Source: local data frame [336,776 x 3]
## 
##     year month   day
##    (int) (int) (int)
## 1   2013     1     1
## 2   2013     1     1
## 3   2013     1     1
## 4   2013     1     1
## 5   2013     1     1
## 6   2013     1     1
## 7   2013     1     1
## 8   2013     1     1
## 9   2013     1     1
## 10  2013     1     1
## ..   ...   ...   ...

select() - Exclude Columns

flights %>% select(-year, -month, -day)
## Source: local data frame [336,776 x 13]
## 
##    dep_time dep_delay arr_time arr_delay carrier tailnum flight origin  dest air_time
##       (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr) (chr)    (dbl)
## 1       517         2      830        11      UA  N14228   1545    EWR   IAH      227
## 2       533         4      850        20      UA  N24211   1714    LGA   IAH      227
## 3       542         2      923        33      AA  N619AA   1141    JFK   MIA      160
## 4       544        -1     1004       -18      B6  N804JB    725    JFK   BQN      183
## 5       554        -6      812       -25      DL  N668DN    461    LGA   ATL      116
## 6       554        -4      740        12      UA  N39463   1696    EWR   ORD      150
## 7       555        -5      913        19      B6  N516JB    507    EWR   FLL      158
## 8       557        -3      709       -14      EV  N829AS   5708    LGA   IAD       53
## 9       557        -3      838        -8      B6  N593JB     79    JFK   MCO      140
## 10      558        -2      753         8      AA  N3ALAA    301    LGA   ORD      138
## ..      ...       ...      ...       ...     ...     ...    ...    ...   ...      ...
## Variables not shown: distance (dbl), hour (dbl), minute (dbl)

select() - Ranges

flights %>% select(year:day)
## Source: local data frame [336,776 x 3]
## 
##     year month   day
##    (int) (int) (int)
## 1   2013     1     1
## 2   2013     1     1
## 3   2013     1     1
## 4   2013     1     1
## 5   2013     1     1
## 6   2013     1     1
## 7   2013     1     1
## 8   2013     1     1
## 9   2013     1     1
## 10  2013     1     1
## ..   ...   ...   ...

select() - Exclusion Ranges

flights %>% select(-(year:day))
## Source: local data frame [336,776 x 13]
## 
##    dep_time dep_delay arr_time arr_delay carrier tailnum flight origin  dest air_time
##       (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr) (chr)    (dbl)
## 1       517         2      830        11      UA  N14228   1545    EWR   IAH      227
## 2       533         4      850        20      UA  N24211   1714    LGA   IAH      227
## 3       542         2      923        33      AA  N619AA   1141    JFK   MIA      160
## 4       544        -1     1004       -18      B6  N804JB    725    JFK   BQN      183
## 5       554        -6      812       -25      DL  N668DN    461    LGA   ATL      116
## 6       554        -4      740        12      UA  N39463   1696    EWR   ORD      150
## 7       555        -5      913        19      B6  N516JB    507    EWR   FLL      158
## 8       557        -3      709       -14      EV  N829AS   5708    LGA   IAD       53
## 9       557        -3      838        -8      B6  N593JB     79    JFK   MCO      140
## 10      558        -2      753         8      AA  N3ALAA    301    LGA   ORD      138
## ..      ...       ...      ...       ...     ...     ...    ...    ...   ...      ...
## Variables not shown: distance (dbl), hour (dbl), minute (dbl)

select() - Matching

flights %>% select(contains("dep"), 
                   contains("arr"))
## Source: local data frame [336,776 x 5]
## 
##    dep_time dep_delay arr_time arr_delay carrier
##       (int)     (dbl)    (int)     (dbl)   (chr)
## 1       517         2      830        11      UA
## 2       533         4      850        20      UA
## 3       542         2      923        33      AA
## 4       544        -1     1004       -18      B6
## 5       554        -6      812       -25      DL
## 6       554        -4      740        12      UA
## 7       555        -5      913        19      B6
## 8       557        -3      709       -14      EV
## 9       557        -3      838        -8      B6
## 10      558        -2      753         8      AA
## ..      ...       ...      ...       ...     ...
flights %>% select(contains("dep"), 
                   starts_with("arr"))
## Source: local data frame [336,776 x 4]
## 
##    dep_time dep_delay arr_time arr_delay
##       (int)     (dbl)    (int)     (dbl)
## 1       517         2      830        11
## 2       533         4      850        20
## 3       542         2      923        33
## 4       544        -1     1004       -18
## 5       554        -6      812       -25
## 6       554        -4      740        12
## 7       555        -5      913        19
## 8       557        -3      709       -14
## 9       557        -3      838        -8
## 10      558        -2      753         8
## ..      ...       ...      ...       ...

rename()

flights %>% rename(tail_number = tailnum)
## Source: local data frame [336,776 x 16]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tail_number flight
##    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)       (chr)  (int)
## 1   2013     1     1      517         2      830        11      UA      N14228   1545
## 2   2013     1     1      533         4      850        20      UA      N24211   1714
## 3   2013     1     1      542         2      923        33      AA      N619AA   1141
## 4   2013     1     1      544        -1     1004       -18      B6      N804JB    725
## 5   2013     1     1      554        -6      812       -25      DL      N668DN    461
## 6   2013     1     1      554        -4      740        12      UA      N39463   1696
## 7   2013     1     1      555        -5      913        19      B6      N516JB    507
## 8   2013     1     1      557        -3      709       -14      EV      N829AS   5708
## 9   2013     1     1      557        -3      838        -8      B6      N593JB     79
## 10  2013     1     1      558        -2      753         8      AA      N3ALAA    301
## ..   ...   ...   ...      ...       ...      ...       ...     ...         ...    ...
## Variables not shown: origin (chr), dest (chr), air_time (dbl), distance (dbl), hour
##   (dbl), minute (dbl)

arrange()

flights %>% filter(month==3,day==2) %>% arrange(origin, dest)
## Source: local data frame [765 x 16]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1   2013     3     2     1336         7     1426        -6      EV  N11535   4263    EWR
## 2   2013     3     2      628        -1      837       -12      DL  N357NB    575    EWR
## 3   2013     3     2      637        -3      903       -12      EV  N16919   4209    EWR
## 4   2013     3     2      743        -2      945       -25      DL  N338NB    807    EWR
## 5   2013     3     2      857        -3     1117        -9      DL  N306DQ    485    EWR
## 6   2013     3     2     1027        -3     1234       -13      DL  N301DQ   2343    EWR
## 7   2013     3     2     1134       -11     1332       -27      DL  N301NB    401    EWR
## 8   2013     3     2     1412        -3     1636         6      DL  N368NB    935    EWR
## 9   2013     3     2     1633        -3     1848       -20      EV  N16183   3273    EWR
## 10  2013     3     2     1655        -5     1857       -27      DL  N305DQ   2042    EWR
## ..   ...   ...   ...      ...       ...      ...       ...     ...     ...    ...    ...
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute (dbl)

arrange() & desc()

flights %>% filter(month==3,day==2) %>% arrange(desc(origin), dest)
## Source: local data frame [765 x 16]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1   2013     3     2      555        -5      822         7      FL  N928AT    345    LGA
## 2   2013     3     2      556        -4      822        -7      DL  N623DL    461    LGA
## 3   2013     3     2      758        -2     1009       -22      DL  N680DA   2047    LGA
## 4   2013     3     2      808        -2     1022       -21      FL  N996AT    361    LGA
## 5   2013     3     2      828        -7     1050       -15      MQ  N510MQ   4610    LGA
## 6   2013     3     2      857        -3     1126       -16      DL  N663DN   1747    LGA
## 7   2013     3     2      957        -3     1224       -10      DL  N942DL   1847    LGA
## 8   2013     3     2     1158        -2     1411       -19      MQ  N511MQ   4658    LGA
## 9   2013     3     2     1159        -1     1429        -9      DL  N910DE   1947    LGA
## 10  2013     3     2     1259        -1     1510       -25      DL  N902DE    781    LGA
## ..   ...   ...   ...      ...       ...      ...       ...     ...     ...    ...    ...
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute (dbl)

mutate()

library(lubridate)
## Loading required package: methods
flights %>% select(1:3) %>% mutate(date = paste(month,day,year,sep="/") %>% mdy())
## Source: local data frame [336,776 x 4]
## 
##     year month   day       date
##    (int) (int) (int)     (time)
## 1   2013     1     1 2013-01-01
## 2   2013     1     1 2013-01-01
## 3   2013     1     1 2013-01-01
## 4   2013     1     1 2013-01-01
## 5   2013     1     1 2013-01-01
## 6   2013     1     1 2013-01-01
## 7   2013     1     1 2013-01-01
## 8   2013     1     1 2013-01-01
## 9   2013     1     1 2013-01-01
## 10  2013     1     1 2013-01-01
## ..   ...   ...   ...        ...

transmute()

flights %>% select(1:3) %>% transmute(date = paste(month,day,year,sep="/") %>% mdy())
## Source: local data frame [336,776 x 1]
## 
##          date
##        (time)
## 1  2013-01-01
## 2  2013-01-01
## 3  2013-01-01
## 4  2013-01-01
## 5  2013-01-01
## 6  2013-01-01
## 7  2013-01-01
## 8  2013-01-01
## 9  2013-01-01
## 10 2013-01-01
## ..        ...

distinct()

flights %>% select(origin, dest) %>% distinct() %>% arrange(origin,dest)
## Source: local data frame [224 x 2]
## 
##    origin  dest
##     (chr) (chr)
## 1     EWR   ALB
## 2     EWR   ANC
## 3     EWR   ATL
## 4     EWR   AUS
## 5     EWR   AVL
## 6     EWR   BDL
## 7     EWR   BNA
## 8     EWR   BOS
## 9     EWR   BQN
## 10    EWR   BTV
## ..    ...   ...

sample_n()

flights %>% sample_n(10)
## Source: local data frame [10 x 16]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1   2013     5     1      826        -7     1123       -16      B6  N821JB    181    JFK
## 2   2013     2    27     1722        -3     2018         1      UA  N17128   1109    EWR
## 3   2013     4     6     1125        -5     1311       -19      US  N197UW   1625    LGA
## 4   2013     7     2     1541        11     1910        24      DL  N193DN     95    JFK
## 5   2013     3    28     2018        -3     2202         3      UA  N21723   1464    EWR
## 6   2013    12     4     2224        -1     2331       -10      EV  N11121   4695    EWR
## 7   2013     7    25     1610        15     1852       -12      DL  N371DA   2459    JFK
## 8   2013     6     5     1856         1     2013       -28      EV  N718EV   5562    LGA
## 9   2013     7    12     2143       148        4        94      AA  N320AA     21    JFK
## 10  2013     9    23     1837        38     2111        19      B6  N504JB    299    LGA
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute (dbl)

sample_frac()

flights %>% sample_frac(0.001)
## Source: local data frame [337 x 16]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1   2013     6    16     1535       115     1647        97      WN  N760SW    427    LGA
## 2   2013     9     4      741         0      954       -31      UA  N37437   1591    EWR
## 3   2013     8    15     2145         0     2351       -27      B6  N536JB    611    JFK
## 4   2013     8    14     1343        -2     1639        -6      AA  N335AA    117    JFK
## 5   2013    10    18     1835        54     2050        66      EV  N16170   4667    EWR
## 6   2013     8    16     1624        79     1906        76      UA  N563UA    468    EWR
## 7   2013     1    10      632        -7      943       -14      UA  N848UA    910    EWR
## 8   2013     2    28      558        -2      853         4      B6  N632JB    145    JFK
## 9   2013     2     8     1237        37     1419        70      US  N742PS   2173    LGA
## 10  2013     6     2     1556        57     1806        91      9E  N8968E   4105    JFK
## ..   ...   ...   ...      ...       ...      ...       ...     ...     ...    ...    ...
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute (dbl)

summarise()

flights %>% mutate(date = paste(month,day,year,sep="/") %>% mdy()) %>% 
            summarize(n(), min(date), max(date))
## Source: local data frame [1 x 3]
## 
##      n()  min(date)  max(date)
##    (int)     (time)     (time)
## 1 336776 2013-01-01 2013-12-31

group_by()

flights %>% group_by(origin)
## Source: local data frame [336,776 x 16]
## Groups: origin [3]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##    (int) (int) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1   2013     1     1      517         2      830        11      UA  N14228   1545    EWR
## 2   2013     1     1      533         4      850        20      UA  N24211   1714    LGA
## 3   2013     1     1      542         2      923        33      AA  N619AA   1141    JFK
## 4   2013     1     1      544        -1     1004       -18      B6  N804JB    725    JFK
## 5   2013     1     1      554        -6      812       -25      DL  N668DN    461    LGA
## 6   2013     1     1      554        -4      740        12      UA  N39463   1696    EWR
## 7   2013     1     1      555        -5      913        19      B6  N516JB    507    EWR
## 8   2013     1     1      557        -3      709       -14      EV  N829AS   5708    LGA
## 9   2013     1     1      557        -3      838        -8      B6  N593JB     79    JFK
## 10  2013     1     1      558        -2      753         8      AA  N3ALAA    301    LGA
## ..   ...   ...   ...      ...       ...      ...       ...     ...     ...    ...    ...
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute (dbl)

summarise() with group_by()

flights %>% group_by(origin) %>%
            mutate(date = paste(month,day,year,sep="/") %>% mdy()) %>% 
            summarize(n(), min(date), max(date))
## Source: local data frame [3 x 4]
## 
##   origin    n()  min(date)  max(date)
##    (chr)  (int)     (time)     (time)
## 1    EWR 120835 2013-01-01 2013-12-31
## 2    JFK 111279 2013-01-01 2013-12-31
## 3    LGA 104662 2013-01-01 2013-12-31

Examples

  1. Which destinations have the highest average delays?

  2. Which plane (check the tail number) flew out of each New York airport the most?

Exercises - Part 1

  1. What was the shortest flight out of each airport in terms of distance? In terms of duration?

  2. Which date should you fly on if you want to have the lowest possible average depature delay? What about arrival delay?

  3. How many flights to Los Angeles (LAX) did each of the legacy carriers (AA, UA, DL or US) have in May from JFK, and what was their average duration?

Merging Data

A Grammar of Data Manipulation (cont.)

Two table functions / verbs, all functions have the form f(a,b):

  • left_join - Join matching rows from b to a, preserving all rows of a
  • right_join - Join matching rows from a to b, preserving all rows of b.
  • inner_join - Join data, preserving only rows with keys in both a and b.
  • full_join - Join data, preserving all rows in both a and b.
  • semi_join - Subset rows in a that have a match in b.
  • anti_join - Subset rows in a that do not have a match in b.

Joining Data

addr = data.frame(name = c("Alice","Bob",
                           "Carol","dave",
                           "Eve"),
                  email= c("alice@company.com",
                           "bob@company.com",
                           "carol@company.com",
                           "dave@company.com",
                           "eve@company.com"),
                  stringsAsFactors = FALSE)
phone = data.frame(name = c("Bob","Carol",
                            "Eve","Eve",
                            "Frank"),
                   phone= c("919 555-1111",
                            "919 555-2222",
                            "919 555-3333",
                            "310 555-3333",
                            "919 555-4444"),
                   stringsAsFactors = FALSE)
addr
##    name             email
## 1 Alice alice@company.com
## 2   Bob   bob@company.com
## 3 Carol carol@company.com
## 4  dave  dave@company.com
## 5   Eve   eve@company.com
phone
##    name        phone
## 1   Bob 919 555-1111
## 2 Carol 919 555-2222
## 3   Eve 919 555-3333
## 4   Eve 310 555-3333
## 5 Frank 919 555-4444

Outer Join

dplyr:

full_join(addr, phone)
## Joining by: "name"
##    name             email        phone
## 1 Alice alice@company.com         <NA>
## 2   Bob   bob@company.com 919 555-1111
## 3 Carol carol@company.com 919 555-2222
## 4  dave  dave@company.com         <NA>
## 5   Eve   eve@company.com 919 555-3333
## 6   Eve   eve@company.com 310 555-3333
## 7 Frank              <NA> 919 555-4444

Base R:

merge(addr, phone, all=TRUE)
##    name             email        phone
## 1 Alice alice@company.com         <NA>
## 2   Bob   bob@company.com 919 555-1111
## 3 Carol carol@company.com 919 555-2222
## 4  dave  dave@company.com         <NA>
## 5   Eve   eve@company.com 919 555-3333
## 6   Eve   eve@company.com 310 555-3333
## 7 Frank              <NA> 919 555-4444


Inner Join

dplyr:

inner_join(addr,phone)
## Joining by: "name"
##    name             email        phone
## 1   Bob   bob@company.com 919 555-1111
## 2 Carol carol@company.com 919 555-2222
## 3   Eve   eve@company.com 919 555-3333
## 4   Eve   eve@company.com 310 555-3333

Base R:

merge(addr, phone, all=FALSE)
##    name             email        phone
## 1   Bob   bob@company.com 919 555-1111
## 2 Carol carol@company.com 919 555-2222
## 3   Eve   eve@company.com 919 555-3333
## 4   Eve   eve@company.com 310 555-3333


Left Join

dplyr:

left_join(addr,phone)
## Joining by: "name"
##    name             email        phone
## 1 Alice alice@company.com         <NA>
## 2   Bob   bob@company.com 919 555-1111
## 3 Carol carol@company.com 919 555-2222
## 4  dave  dave@company.com         <NA>
## 5   Eve   eve@company.com 919 555-3333
## 6   Eve   eve@company.com 310 555-3333

Base R:

merge(addr, phone, all.x=TRUE)
##    name             email        phone
## 1 Alice alice@company.com         <NA>
## 2   Bob   bob@company.com 919 555-1111
## 3 Carol carol@company.com 919 555-2222
## 4  dave  dave@company.com         <NA>
## 5   Eve   eve@company.com 919 555-3333
## 6   Eve   eve@company.com 310 555-3333


Right Join

dplyr:

right_join(addr, phone)
## Joining by: "name"
##    name             email        phone
## 1   Bob   bob@company.com 919 555-1111
## 2 Carol carol@company.com 919 555-2222
## 3   Eve   eve@company.com 919 555-3333
## 4   Eve   eve@company.com 310 555-3333
## 5 Frank              <NA> 919 555-4444

Base R:

merge(addr, phone, all.y=TRUE)
##    name             email        phone
## 1   Bob   bob@company.com 919 555-1111
## 2 Carol carol@company.com 919 555-2222
## 3   Eve   eve@company.com 919 555-3333
## 4   Eve   eve@company.com 310 555-3333
## 5 Frank              <NA> 919 555-4444


Semi and Anti Joins

semi_join(addr, phone)
## Joining by: "name"
##    name             email
## 1   Bob   bob@company.com
## 2 Carol carol@company.com
## 3   Eve   eve@company.com
anti_join(addr, phone)
## Joining by: "name"
##    name             email
## 1  dave  dave@company.com
## 2 Alice alice@company.com

Special case - many-to-many relationships

addr = data.frame(name = c("Alice","Alice", "Bob","Bob"),
                  email= c("alice@company.com","alice@gmail.com", "bob@company.com","bob@hotmail.com"),
                  stringsAsFactors = FALSE)
phone = data.frame(name = c("Alice","Alice", "Bob","Bob"),
                   phone= c("919 555-1111", "310 555-2222", "919 555-3333", "310 555-3333"),
                   stringsAsFactors = FALSE)

dplyr:

full_join(addr, phone, by="name")
##    name             email        phone
## 1 Alice alice@company.com 919 555-1111
## 2 Alice alice@company.com 310 555-2222
## 3 Alice   alice@gmail.com 919 555-1111
## 4 Alice   alice@gmail.com 310 555-2222
## 5   Bob   bob@company.com 919 555-3333
## 6   Bob   bob@company.com 310 555-3333
## 7   Bob   bob@hotmail.com 919 555-3333
## 8   Bob   bob@hotmail.com 310 555-3333


Base R:

merge(addr, phone)
##    name             email        phone
## 1 Alice alice@company.com 919 555-1111
## 2 Alice alice@company.com 310 555-2222
## 3 Alice   alice@gmail.com 919 555-1111
## 4 Alice   alice@gmail.com 310 555-2222
## 5   Bob   bob@company.com 919 555-3333
## 6   Bob   bob@company.com 310 555-3333
## 7   Bob   bob@hotmail.com 919 555-3333
## 8   Bob   bob@hotmail.com 310 555-3333


Example - Enhancing NYC Flights

The nycflights13 package also contains additional information about

  • weather - hourly meterological data for each airport
  • planes - construction information about each plane
  • airports - airport names and locations
  • airlines - translation between two letter carrier codes and names

Weather and flight delays

Lets take a quick look at the weather data, with an eye towards examining how it might affect things like flight departure delays,

weather
## Source: local data frame [8,719 x 14]
## Groups: month, day, hour [8719]
## 
##    origin  year month   day  hour  temp  dewp humid wind_dir wind_speed wind_gust precip
##     (chr) (dbl) (dbl) (int) (int) (dbl) (dbl) (dbl)    (dbl)      (dbl)     (dbl)  (dbl)
## 1     EWR  2013     1     1     0 37.04 21.92 53.97      230   10.35702 11.918651      0
## 2     EWR  2013     1     1     1 37.04 21.92 53.97      230   13.80936 15.891535      0
## 3     EWR  2013     1     1     2 37.94 21.92 52.09      230   12.65858 14.567241      0
## 4     EWR  2013     1     1     3 37.94 23.00 54.51      230   13.80936 15.891535      0
## 5     EWR  2013     1     1     4 37.94 24.08 57.04      240   14.96014 17.215830      0
## 6     EWR  2013     1     1     6 39.02 26.06 59.37      270   10.35702 11.918651      0
## 7     EWR  2013     1     1     7 39.02 26.96 61.63      250    8.05546  9.270062      0
## 8     EWR  2013     1     1     8 39.02 28.04 64.43      240   11.50780 13.242946      0
## 9     EWR  2013     1     1     9 39.92 28.04 62.21      250   12.65858 14.567241      0
## 10    EWR  2013     1     1    10 39.02 28.04 64.43      260   12.65858 14.567241      0
## ..    ...   ...   ...   ...   ...   ...   ...   ...      ...        ...       ...    ...
## Variables not shown: pressure (dbl), visib (dbl)

Join by?

Weather column definitions:

  • origin - Weather station. Named origin to faciliate merging with flights data
  • year, month, day, hour - Time of recording
  • temp, dewp - Temperature and dewpoint in F
  • humid - Relative humidity
  • wind_dir, wind_speed, wind_gust - Wind direction (in degrees), speed and gust speed (in mph)
  • precip - Preciptation, in inches
  • pressure - Sea level pressure in millibars
  • visib - Visibility in miles
intersect(names(weather), names(flights))
## [1] "origin" "year"   "month"  "day"    "hour"

Joining flights and weather

When joining these two data frames I will use an inner join (because I only want the flights that have the weather data, and I only want weather data when there was a flight)

(flightsw = inner_join(flights, weather))
## Joining by: c("year", "month", "day", "origin", "hour")
## Source: local data frame [117,251 x 25]
## 
##     year month   day dep_time dep_delay arr_time arr_delay carrier tailnum flight origin
##    (dbl) (dbl) (int)    (int)     (dbl)    (int)     (dbl)   (chr)   (chr)  (int)  (chr)
## 1   2013     1     1      601         1      844        -6      B6  N644JB    343    EWR
## 2   2013     1     1      606        -4      858       -12      AA  N633AA   1895    EWR
## 3   2013     1     1      607         0      858       -17      UA  N53442   1077    EWR
## 4   2013     1     1      608         8      807        32      MQ  N9EAMQ   3768    EWR
## 5   2013     1     1      615         0      833        -9      DL  N326NB    575    EWR
## 6   2013     1     1      622        -8     1017         3      US  N807AW    245    EWR
## 7   2013     1     1      624        -6      909        29      EV  N11107   4626    EWR
## 8   2013     1     1      628        -2     1016        29      UA  N33289   1665    EWR
## 9   2013     1     1      629        -1      824        -9      US  N426US   1019    EWR
## 10  2013     1     1      632        24      740        12      EV  N13553   4144    EWR
## ..   ...   ...   ...      ...       ...      ...       ...     ...     ...    ...    ...
## Variables not shown: dest (chr), air_time (dbl), distance (dbl), hour (dbl), minute
##   (dbl), temp (dbl), dewp (dbl), humid (dbl), wind_dir (dbl), wind_speed (dbl), wind_gust
##   (dbl), precip (dbl), pressure (dbl), visib (dbl)

plot(flightsw$visib, flightsw$dep_delay)

plot(flightsw$visib, flightsw$dep_delay, log="y", 
     pch=16, cex=0.5, col=adjustcolor("black",0.1))
## Warning in xy.coords(x, y, xlabel, ylabel, log): 64694 y values <= 0 omitted from
## logarithmic plot

boxplot(dep_delay~visib,data=flightsw)

filter(flightsw, dep_delay > 0)  %>% 
  boxplot(dep_delay~visib,data=.,log="y")

table(flightsw$visib)
## 
##  0.12  0.25   0.5  0.75     1  1.25   1.5  1.75     2   2.5     3     4     5     6     7 
##    50   539   603    96   415    18   645    37  1414   754  1488  1241  1682  2218  2280 
##     8     9    10 
##  2227  4669 96875

Exercises - Part 2

  1. Check some of the other weather variables (e.g. temp, wind_speed, etc) and see if you can find any relationship between them and departure delay (dep_delay).

  2. Merge the flights data with the planes data set (pay attention to what columns are being used for the join). Are older planes more likely to be delayed? What about planes from Airbus vs Boeing vs Embraer?

Acknowledgments

Acknowledgments