Version info: Code for this page was tested in R Under development (unstable) (2012-07-05 r59734)
On: 2012-08-08
With: knitr 0.6.3
You may at times wish to read a set of data files into R. The code below demonstrates how to do so looping through the names of the files to be read in. The datasets we read in here are from Stata, so we will be using the foreign package. All of the data sets are stored in one list, each element of which is a data frame class object of one of the data sets. We use the file.path() function to combine the path to each file with the actual file names.
library(foreign) ## create and view an object with file names and full paths (f <- file.path("https://stats.idre.ucla.edu/stat/data", c("auto.dta", "cancer.dta", "efa_cfa.dta", "hsbmar.dta")))
## [1] "https://stats.idre.ucla.edu/stat/data/auto.dta" ## [2] "https://stats.idre.ucla.edu/stat/data/cancer.dta" ## [3] "https://stats.idre.ucla.edu/stat/data/efa_cfa.dta" ## [4] "https://stats.idre.ucla.edu/stat/data/hsbmar.dta"
Now we would like to use a loop to read all of these files from the internet and store them in one list, d. Note that this would work just as well if the files were on a local disk instead of the internet. R can easily read local or remote files. lapply loops through each file in f, passes it to the function specified (in this case read.dta) and returns all of the results as a list which is then assigned to d.
d <- lapply(f, read.dta) ## view the structure of d str(d, give.attr = FALSE)
## List of 4 ## $ :'data.frame': 74 obs. of 12 variables: ## ..$ make : chr [1:74] "AMC Concord" "AMC Pacer" "AMC Spirit" "Buick Century" ... ## ..$ price : int [1:74] 4099 4749 3799 4816 7827 5788 4453 5189 10372 4082 ... ## ..$ mpg : int [1:74] 22 17 22 20 15 18 26 20 16 19 ... ## ..$ rep78 : int [1:74] 3 3 NA 3 4 3 NA 3 3 3 ... ## ..$ headroom : num [1:74] 2.5 3 3 4.5 4 4 3 2 3.5 3.5 ... ## ..$ trunk : int [1:74] 11 11 12 16 20 21 10 16 17 13 ... ## ..$ weight : int [1:74] 2930 3350 2640 3250 4080 3670 2230 3280 3880 3400 ... ## ..$ length : int [1:74] 186 173 168 196 222 218 170 200 207 200 ... ## ..$ turn : int [1:74] 40 40 35 40 43 43 34 42 43 42 ... ## ..$ displacement: int [1:74] 121 258 121 196 350 231 304 196 231 231 ... ## ..$ gear_ratio : num [1:74] 3.58 2.53 3.08 2.93 2.41 ... ## ..$ foreign : Factor w/ 2 levels "Domestic","Foreign": 1 1 1 1 1 1 1 1 1 1 ... ## $ :'data.frame': 48 obs. of 7 variables: ## ..$ id : num [1:48] 1 2 3 4 5 6 7 8 9 10 ... ## ..$ studytime: int [1:48] 1 1 2 3 4 4 5 5 8 8 ... ## ..$ died : int [1:48] 1 1 1 1 1 1 1 1 1 0 ... ## ..$ drug : num [1:48] 0 0 0 0 0 0 0 0 0 0 ... ## ..$ age : int [1:48] 61 65 59 52 56 67 63 58 56 58 ... ## ..$ distime : num [1:48] 1 1 1 1 1 1 1 1 2 2 ... ## ..$ censor : num [1:48] 0 0 0 0 0 0 0 0 0 1 ... ## $ :'data.frame': 500 obs. of 6 variables: ## ..$ y1: num [1:500] -0.535 0.522 1.581 -0.458 0.276 ... ## ..$ y2: num [1:500] 0.0996 0.959 2.4845 0.8851 -0.8402 ... ## ..$ y3: num [1:500] -0.887 2.619 3.085 0.25 -2.08 ... ## ..$ y4: num [1:500] -1.245 2.668 0.908 -1.076 -0.441 ... ## ..$ y5: num [1:500] -3.493 0.438 2.218 -1.365 -2.67 ... ## ..$ y6: num [1:500] 1.4408 0.7939 -0.0673 -0.9105 -1.2395 ... ## $ :'data.frame': 200 obs. of 13 variables: ## ..$ id : int [1:200] 52 155 33 78 17 189 70 101 36 199 ... ## ..$ female : Factor w/ 2 levels "male","female": NA NA NA NA NA NA NA NA NA NA ... ## ..$ ses : Factor w/ 3 levels "low","middle",..: 1 2 1 2 2 2 1 3 1 3 ... ## ..$ schtyp : Factor w/ 2 levels "public","private": 1 1 1 1 1 2 1 1 1 2 ... ## ..$ prog : Factor w/ 3 levels "general","academic",..: 2 1 2 2 2 2 1 2 1 2 ... ## ..$ read : int [1:200] 50 44 57 39 47 47 57 60 44 NA ... ## ..$ write : int [1:200] 46 44 65 54 57 59 52 62 49 59 ... ## ..$ math : int [1:200] 53 46 72 54 48 63 41 67 44 50 ... ## ..$ science: int [1:200] 53 39 54 53 44 53 47 50 35 61 ... ## ..$ socst : int [1:200] 66 51 56 41 41 46 57 56 51 61 ... ## ..$ honors : Factor w/ 2 levels "not enrolled",..: 1 1 2 1 1 1 1 2 1 1 ... ## ..$ awards : int [1:200] 0 0 5 1 2 2 1 3 0 2 ... ## ..$ cid : int [1:200] 9 3 18 9 8 13 8 16 3 13 ...
You can then operate on the list as a whole or specific elements. For example, you could return a list of al the variable names in each data set by applying the function names to each element of d using the function lapply.
lapply(d, names)
## [[1]] ## [1] "make" "price" "mpg" "rep78" ## [5] "headroom" "trunk" "weight" "length" ## [9] "turn" "displacement" "gear_ratio" "foreign" ## ## [[2]] ## [1] "id" "studytime" "died" "drug" "age" "distime" ## [7] "censor" ## ## [[3]] ## [1] "y1" "y2" "y3" "y4" "y5" "y6" ## ## [[4]] ## [1] "id" "female" "ses" "schtyp" "prog" "read" "write" ## [8] "math" "science" "socst" "honors" "awards" "cid" ##
You can also work with just one element. For example, to view a summary of the first three variable of the first data set, first use [[ to extract the first element of the list, data set 1 (auto.dta) and then [ to extract the first 3 columns and call summary on those.
summary(d[[1]][, 1:3])
## make price mpg ## Length:74 Min. : 3291 Min. :12.0 ## Class :character 1st Qu.: 4220 1st Qu.:18.0 ## Mode :character Median : 5006 Median :20.0 ## Mean : 6165 Mean :21.3 ## 3rd Qu.: 6332 3rd Qu.:24.8 ## Max. :15906 Max. :41.0
To make working with the list easier, we might want to name the elements. That way, we know which dataset is in which element and can refer to them by name instead of just by position if we want. To do this, let’s use regular expressions to parse the full path names and extract the portion after the last “/” and before the “.”. Then we assign those as the names of d.
names(d) <- gsub(".*/(.*)\\..*", "\\1", f) ## view the structure of d again str(d, give.attr = FALSE)
## List of 4 ## $ auto :'data.frame': 74 obs. of 12 variables: ## ..$ make : chr [1:74] "AMC Concord" "AMC Pacer" "AMC Spirit" "Buick Century" ... ## ..$ price : int [1:74] 4099 4749 3799 4816 7827 5788 4453 5189 10372 4082 ... ## ..$ mpg : int [1:74] 22 17 22 20 15 18 26 20 16 19 ... ## ..$ rep78 : int [1:74] 3 3 NA 3 4 3 NA 3 3 3 ... ## ..$ headroom : num [1:74] 2.5 3 3 4.5 4 4 3 2 3.5 3.5 ... ## ..$ trunk : int [1:74] 11 11 12 16 20 21 10 16 17 13 ... ## ..$ weight : int [1:74] 2930 3350 2640 3250 4080 3670 2230 3280 3880 3400 ... ## ..$ length : int [1:74] 186 173 168 196 222 218 170 200 207 200 ... ## ..$ turn : int [1:74] 40 40 35 40 43 43 34 42 43 42 ... ## ..$ displacement: int [1:74] 121 258 121 196 350 231 304 196 231 231 ... ## ..$ gear_ratio : num [1:74] 3.58 2.53 3.08 2.93 2.41 ... ## ..$ foreign : Factor w/ 2 levels "Domestic","Foreign": 1 1 1 1 1 1 1 1 1 1 ... ## $ cancer :'data.frame': 48 obs. of 7 variables: ## ..$ id : num [1:48] 1 2 3 4 5 6 7 8 9 10 ... ## ..$ studytime: int [1:48] 1 1 2 3 4 4 5 5 8 8 ... ## ..$ died : int [1:48] 1 1 1 1 1 1 1 1 1 0 ... ## ..$ drug : num [1:48] 0 0 0 0 0 0 0 0 0 0 ... ## ..$ age : int [1:48] 61 65 59 52 56 67 63 58 56 58 ... ## ..$ distime : num [1:48] 1 1 1 1 1 1 1 1 2 2 ... ## ..$ censor : num [1:48] 0 0 0 0 0 0 0 0 0 1 ... ## $ efa_cfa:'data.frame': 500 obs. of 6 variables: ## ..$ y1: num [1:500] -0.535 0.522 1.581 -0.458 0.276 ... ## ..$ y2: num [1:500] 0.0996 0.959 2.4845 0.8851 -0.8402 ... ## ..$ y3: num [1:500] -0.887 2.619 3.085 0.25 -2.08 ... ## ..$ y4: num [1:500] -1.245 2.668 0.908 -1.076 -0.441 ... ## ..$ y5: num [1:500] -3.493 0.438 2.218 -1.365 -2.67 ... ## ..$ y6: num [1:500] 1.4408 0.7939 -0.0673 -0.9105 -1.2395 ... ## $ hsbmar :'data.frame': 200 obs. of 13 variables: ## ..$ id : int [1:200] 52 155 33 78 17 189 70 101 36 199 ... ## ..$ female : Factor w/ 2 levels "male","female": NA NA NA NA NA NA NA NA NA NA ... ## ..$ ses : Factor w/ 3 levels "low","middle",..: 1 2 1 2 2 2 1 3 1 3 ... ## ..$ schtyp : Factor w/ 2 levels "public","private": 1 1 1 1 1 2 1 1 1 2 ... ## ..$ prog : Factor w/ 3 levels "general","academic",..: 2 1 2 2 2 2 1 2 1 2 ... ## ..$ read : int [1:200] 50 44 57 39 47 47 57 60 44 NA ... ## ..$ write : int [1:200] 46 44 65 54 57 59 52 62 49 59 ... ## ..$ math : int [1:200] 53 46 72 54 48 63 41 67 44 50 ... ## ..$ science: int [1:200] 53 39 54 53 44 53 47 50 35 61 ... ## ..$ socst : int [1:200] 66 51 56 41 41 46 57 56 51 61 ... ## ..$ honors : Factor w/ 2 levels "not enrolled",..: 1 1 2 1 1 1 1 2 1 1 ... ## ..$ awards : int [1:200] 0 0 5 1 2 2 1 3 0 2 ... ## ..$ cid : int [1:200] 9 3 18 9 8 13 8 16 3 13 ...
## reference list elements by name create a summary of the cancer dataset ## in d summary(d$cancer)
## id studytime died drug ## Min. : 1.0 Min. : 1.0 Min. :0.000 Min. :0.000 ## 1st Qu.:12.8 1st Qu.: 7.8 1st Qu.:0.000 1st Qu.:0.000 ## Median :24.5 Median :12.5 Median :1.000 Median :1.000 ## Mean :24.5 Mean :15.5 Mean :0.646 Mean :0.583 ## 3rd Qu.:36.2 3rd Qu.:23.0 3rd Qu.:1.000 3rd Qu.:1.000 ## Max. :48.0 Max. :39.0 Max. :1.000 Max. :1.000 ## age distime censor ## Min. :47.0 Min. :1.00 Min. :0.000 ## 1st Qu.:50.8 1st Qu.:2.00 1st Qu.:0.000 ## Median :56.0 Median :2.50 Median :0.000 ## Mean :55.9 Mean :2.98 Mean :0.354 ## 3rd Qu.:60.0 3rd Qu.:4.00 3rd Qu.:1.000 ## Max. :67.0 Max. :6.00 Max. :1.000
This code easily expands to other types of files. CSV text files could be read using read.csv, general text files with read.table. If you wanted to read all of the files in a particular directory, it can be done by first getting a list of all the file names using list.dirs(), then simply reading them in as before.