README.Rmd

---
output:
  github_document:
    html_preview: false
---

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
options(tibble.print_min = 3)
```

# 🏎💨vroom  <a href="http://r-lib.github.io/vroom"><img src="https://github.com/r-lib/vroom/raw/gh-pages/taylor.gif" align="right" width = "30%"/></a>

<!-- badges: start -->
[![Travis build status](https://travis-ci.org/r-lib/vroom.svg?branch=master)](https://travis-ci.org/r-lib/vroom)
[![AppVeyor build status](https://ci.appveyor.com/api/projects/status/github/r-lib/vroom?branch=master&svg=true)](https://ci.appveyor.com/project/r-lib/vroom)
[![Codecov test coverage](https://codecov.io/gh/r-lib/vroom/branch/master/graph/badge.svg)](https://codecov.io/gh/r-lib/vroom?branch=master)
[![CRAN status](https://www.r-pkg.org/badges/version/vroom)](https://cran.r-project.org/package=vroom)
[![Lifecycle: maturing](https://img.shields.io/badge/lifecycle-maturing-blue.svg)](https://www.tidyverse.org/lifecycle/#maturing)
<!-- badges: end -->

```{r echo = FALSE, message = FALSE}
tm <- vroom::vroom(system.file("bench", "taxi-times.tsv", package = "vroom"))
versions <- vroom::vroom(system.file("bench", "sessioninfo.tsv", package = "vroom"))

# Use the base version number for read.delim
versions$package[versions$package == "base"] <- "read.delim"

library(dplyr)
tbl <- tm %>% dplyr::filter(type == "real", op == "read", package %in% c("vroom (full altrep)_base", "data.table", "readr", "read.delim")) %>%
  mutate(package = sub(" .*", "", package)) %>%
  left_join(versions) %>%
  transmute(
    package = package,
    version = ondiskversion,
    "time (sec)" = time,
    speedup = max(time) / time,
    "throughput" = paste0(prettyunits::pretty_bytes(size / time))
  )
```

The fastest delimited reader for R, **`r dplyr::filter(tbl, package == "vroom") %>% pull("throughput") %>% paste0("/sec") %>% trimws()`**.

But that's impossible! How can it be [so fast](http://vroom.r-lib.org/articles/benchmarks.html)?

vroom doesn't stop to actually _read_ all of your data, it simply indexes where
each record is located so it can be read later. The vectors returned use the
[Altrep framework](https://svn.r-project.org/R/branches/ALTREP/ALTREP.html) to
lazily load the data on-demand when it is accessed, so you only pay for what
you use. This lazy access is done automatically, so no changes to your R
data-manipulation code are needed.

vroom also uses multiple threads for indexing, materializing non-character
columns, and when writing to further improve performance.

```{r, echo = FALSE}
knitr::kable(tbl, digits = 2, align = "lrrrr")
```

## Features

vroom has nearly all of the parsing features of
[readr](https://readr.tidyverse.org) for delimited and fixed width files, including

- delimiter guessing\*
- custom delimiters (including multi-byte\* and Unicode\* delimiters)
- specification of column types (including type guessing)
  - numeric types (double, integer, number)
  - logical types
  - datetime types (datetime, date, time)
  - categorical types (characters, factors)
- column selection, like `dplyr::select()`\*
- skipping headers, comments and blank lines
- quoted fields
- double and backslashed escapes
- whitespace trimming
- windows newlines
- [reading from multiple files or connections\*](#reading-multiple-files)
- embedded newlines in headers and fields\*\*
- writing delimited files with as-needed quoting.
- robust to invalid inputs (vroom has been extensively tested with the
  [afl](http://lcamtuf.coredump.cx/afl/) fuzz tester)\*.

\* *these are additional features only in vroom.*

\*\* *requires `num_threads = 1`.*

## Installation

Install vroom from CRAN with:

```r
install.packages("vroom")
```

Alternatively, if you need the development version from
[GitHub](https://github.com/) install it with:

``` r
# install.packages("devtools")
devtools::install_github("r-lib/vroom")
```
## Usage

See [getting started](https://r-lib.github.io/vroom/articles/vroom.html)
to jump start your use of vroom!

vroom uses the same interface as readr to specify column types.

```{r, include = FALSE}
tibble::rownames_to_column(mtcars, "model") %>%
  vroom::vroom_write("mtcars.tsv", delim = "\t")
```

```{r example}
vroom::vroom("mtcars.tsv",
  col_types = list(cyl = "i", gear = "f",hp = "i", disp = "_",
                   drat = "_", vs = "l", am = "l", carb = "i")
)
```

```{r, include = FALSE}
unlink("mtcars.tsv")
```

## Reading multiple files

vroom natively supports reading from multiple files (or even multiple
connections!).

First we generate some files to read by splitting the nycflights dataset by
airline.
```{r}
library(nycflights13)
purrr::iwalk(
  split(flights, flights$carrier),
  ~ { str(.x$carrier[[1]]); vroom::vroom_write(.x, glue::glue("flights_{.y}.tsv"), delim = "\t") }
)
```

Then we can efficiently read them into one tibble by passing the filenames
directly to vroom.

```{r}
files <- fs::dir_ls(glob = "flights*tsv")
files
vroom::vroom(files)
```

```{r, include = FALSE}
fs::file_delete(files)
```

## Benchmarks

The speed quoted above is from a real dataset with 14,776,615 rows and 11 columns,
see the [benchmark article](http://vroom.r-lib.org/articles/benchmarks.html)
for full details of the dataset and
[bench/](https://github.com/r-lib/vroom/blob/master/inst/bench) for the code
used to retrieve the data and perform the benchmarks.

# Environment variables

In addition to the arguments to the `vroom()` function, you can control the
behavior of vroom with a few environment variables. Generally these will not
need to be set by most users.

- `VROOM_TEMP_PATH` - Path to the directory used to store temporary files when
  reading from a R connection. If unset defaults to the R session's temporary
  directory (`tempdir()`).
- `VROOM_THREADS` - The number of processor threads to use when indexing and
  parsing. If unset defaults to `parallel::detectCores()`.
- `VROOM_SHOW_PROGRESS` - Whether to show the progress bar when indexing.
  Regardless of this setting the progress bar is disabled in non-interactive
  settings, R notebooks, when running tests with testthat and when knitting
  documents.
- `VROOM_CONNECTION_SIZE` - The size (in bytes) of the connection buffer when
  reading from connections (default is 128 KiB).
- `VROOM_WRITE_BUFFER_LINES` - The number of lines to use for each buffer when
  writing files (default: 1000).

There are also a family of variables to control use of the Altrep framework.
For versions of R where the Altrep framework is unavailable (R < 3.5.0) they
are automatically turned off and the variables have no effect. The variables
can take one of `true`, `false`, `TRUE`, `FALSE`, `1`, or `0`.

- `VROOM_USE_ALTREP_NUMERICS` - If set use Altrep for _all_ numeric types
  (default `false`).

There are also individual variables for each type. Currently only
`VROOM_USE_ALTREP_CHR` defaults to `true`.

- `VROOM_USE_ALTREP_CHR`
- `VROOM_USE_ALTREP_FCT`
- `VROOM_USE_ALTREP_INT`
- `VROOM_USE_ALTREP_DBL`
- `VROOM_USE_ALTREP_NUM`
- `VROOM_USE_ALTREP_LGL`
- `VROOM_USE_ALTREP_DTTM`
- `VROOM_USE_ALTREP_DATE`
- `VROOM_USE_ALTREP_TIME`

## RStudio caveats

RStudio's environment pane calls `object.size()` when it refreshes the pane, which
for Altrep objects can be extremely slow. RStudio 1.2.1335+ includes the fixes
([RStudio#4210](https://github.com/rstudio/rstudio/pull/4210),
[RStudio#4292](https://github.com/rstudio/rstudio/pull/4292)) for this issue,
so so it is recommended you use at least that version.

## Thanks

- [Gabe Becker](https://twitter.com/groundwalkergmb), [Luke
  Tierney](https://stat.uiowa.edu/~luke/) and [Tomas Kalibera](https://github.com/kalibera) for
  conceiving, Implementing and maintaining the [Altrep
  framework](https://svn.r-project.org/R/branches/ALTREP/ALTREP.html)
- [Romain François](https://twitter.com/romain_francois), whose
  [Altrepisode](https://purrple.cat/blog/2018/10/14/altrep-and-cpp/) package
  and [related blog-posts](https://purrple.cat/blog/2018/10/14/altrep-and-cpp/) were a great guide for creating new Altrep objects in C++.
- [Matt Dowle](https://twitter.com/mattdowle) and the rest of the [Rdatatable](https://github.com/Rdatatable) team, `data.table::fread()` is blazing fast and great motivation!