data.table::fwrite() was contributed by Otto Seiskari and is available in versions 1.9.8+. Matt has made additional enhancements on top (including parallelisation) and wrote an article about it. Please report any issues on the tracker.
First, here’s a comparison on the same dimensions used by @chase above (i.e., a very large number of columns: 65,000 columns (!) x 256 rows), together with fwrite and write_feather, so that we have some consistency across machines. Note the huge difference compress=FALSE makes in base R.
# -----------------------------------------------------------------------------
# function | object type | output type | compress= | Runtime | File size |
# -----------------------------------------------------------------------------
# save | matrix | binary | FALSE | 0.3s | 134MB |
# save | data.frame | binary | FALSE | 0.4s | 135MB |
# feather | data.frame | binary | FALSE | 0.4s | 139MB |
# fwrite | data.table | csv | FALSE | 1.0s | 302MB |
# save | matrix | binary | TRUE | 17.9s | 89MB |
# save | data.frame | binary | TRUE | 18.1s | 89MB |
# write.csv | matrix | csv | FALSE | 21.7s | 302MB |
# write.csv | data.frame | csv | FALSE | 121.3s | 302MB |
Note that fwrite() runs in parallel. The timing shown here is on a 13′ Macbook Pro with 2 cores and 1 thread/core (+2 virtual threads via hyperthreading), 512GB SSD, 256KB/core L2 cache and 4MB L4 cache. Depending on your system spec, YMMV.
I also reran the benchmarks on relatively more likely (and bigger) data:
library(data.table)
NN <- 5e6 # at this number of rows, the .csv output is ~800Mb on my machine
set.seed(51423)
DT <- data.table(
str1 = sample(sprintf("%010d",1:NN)), #ID field 1
str2 = sample(sprintf("%09d",1:NN)), #ID field 2
# varying length string field--think names/addresses, etc.
str3 = replicate(NN,paste0(sample(LETTERS,sample(10:30,1),T), collapse="")),
# factor-like string field with 50 "levels"
str4 = sprintf("%05d",sample(sample(1e5,50),NN,T)),
# factor-like string field with 17 levels, varying length
str5 = sample(replicate(17,paste0(sample(LETTERS, sample(15:25,1),T),
collapse="")),NN,T),
# lognormally distributed numeric
num1 = round(exp(rnorm(NN,mean=6.5,sd=1.5)),2),
# 3 binary strings
str6 = sample(c("Y","N"),NN,T),
str7 = sample(c("M","F"),NN,T),
str8 = sample(c("B","W"),NN,T),
# right-skewed (integer type)
int1 = as.integer(ceiling(rexp(NN))),
num2 = round(exp(rnorm(NN,mean=6,sd=1.5)),2),
# lognormal numeric that can be positive or negative
num3 = (-1)^sample(2,NN,T)*round(exp(rnorm(NN,mean=6,sd=1.5)),2))
# -------------------------------------------------------------------------------
# function | object | out | other args | Runtime | File size |
# -------------------------------------------------------------------------------
# fwrite | data.table | csv | quote = FALSE | 1.7s | 523.2MB |
# fwrite | data.frame | csv | quote = FALSE | 1.7s | 523.2MB |
# feather | data.frame | bin | no compression | 3.3s | 635.3MB |
# save | data.frame | bin | compress = FALSE | 12.0s | 795.3MB |
# write.csv | data.frame | csv | row.names = FALSE | 28.7s | 493.7MB |
# save | data.frame | bin | compress = TRUE | 48.1s | 190.3MB |
# -------------------------------------------------------------------------------
So fwrite is ~2x faster than feather in this test. This was run on the same machine as noted above with fwrite running in parallel on 2 cores.
feather seems quite fast binary format as well, but no compression yet.
Here’s an attempt at showing how fwrite compares with respect to scale:
NB: the benchmark has been updated by running base R’s save() with compress = FALSE (since feather also is not compressed).
So fwrite is fastest of all of them on this data (running on 2 cores) plus it creates a .csv which can easily be viewed, inspected and passed to grep, sed etc.
Code for reproduction:
require(data.table)
require(microbenchmark)
require(feather)
ns <- as.integer(10^seq(2, 6, length.out = 25))
DTn <- function(nn)
data.table(
str1 = sample(sprintf("%010d",1:nn)),
str2 = sample(sprintf("%09d",1:nn)),
str3 = replicate(nn,paste0(sample(LETTERS,sample(10:30,1),T), collapse="")),
str4 = sprintf("%05d",sample(sample(1e5,50),nn,T)),
str5 = sample(replicate(17,paste0(sample(LETTERS, sample(15:25,1),T), collapse="")),nn,T),
num1 = round(exp(rnorm(nn,mean=6.5,sd=1.5)),2),
str6 = sample(c("Y","N"),nn,T),
str7 = sample(c("M","F"),nn,T),
str8 = sample(c("B","W"),nn,T),
int1 = as.integer(ceiling(rexp(nn))),
num2 = round(exp(rnorm(nn,mean=6,sd=1.5)),2),
num3 = (-1)^sample(2,nn,T)*round(exp(rnorm(nn,mean=6,sd=1.5)),2))
count <- data.table(n = ns,
c = c(rep(1000, 12),
rep(100, 6),
rep(10, 7)))
mbs <- lapply(ns, function(nn){
print(nn)
set.seed(51423)
DT <- DTn(nn)
microbenchmark(times = count[n==nn,c],
write.csv=write.csv(DT, "writecsv.csv", quote=FALSE, row.names=FALSE),
save=save(DT, file = "save.RData", compress=FALSE),
fwrite=fwrite(DT, "fwrite_turbo.csv", quote=FALSE, sep=","),
feather=write_feather(DT, "feather.feather"))})
png("microbenchmark.png", height=600, width=600)
par(las=2, oma = c(1, 0, 0, 0))
matplot(ns, t(sapply(mbs, function(x) {
y <- summary(x)[,"median"]
y/y[3]})),
main = "Relative Speed of fwrite (turbo) vs. rest",
xlab = "", ylab = "Time Relative to fwrite (turbo)",
type = "l", lty = 1, lwd = 2,
col = c("red", "blue", "black", "magenta"), xaxt = "n",
ylim=c(0,25), xlim=c(0, max(ns)))
axis(1, at = ns, labels = prettyNum(ns, ","))
mtext("# Rows", side = 1, las = 1, line = 5)
legend("right", lty = 1, lwd = 3,
legend = c("write.csv", "save", "feather"),
col = c("red", "blue", "magenta"))
dev.off()