Real data come in many formats, and synthetic data often have to be
delivered in the format of the original. SynthesizerPlus offers one pair
of functions, read_data() and write_data(),
that dispatch on the file extension.
supported_formats()
#> format extensions read
#> 1 csv csv base (data.table if installed)
#> 2 tsv tsv, tab, txt base (data.table if installed)
#> 3 rds rds base
#> 4 rdata rdata, rda base
#> 5 parquet parquet, pq arrow or nanoparquet
#> 6 feather feather, arrow, ipc arrow
#> 7 hdf5 h5, hdf5 hdf5r
#> 8 json json jsonlite
#> 9 ndjson ndjson, jsonl jsonlite
#> 10 excel xlsx, xls readxl
#> 11 spss sav, zsav haven
#> 12 stata dta haven
#> 13 sas sas7bdat haven
#> 14 xpt xpt haven
#> 15 fst fst fst
#> write available
#> 1 base (data.table if installed) TRUE
#> 2 base (data.table if installed) TRUE
#> 3 base TRUE
#> 4 base TRUE
#> 5 arrow or nanoparquet TRUE
#> 6 arrow TRUE
#> 7 hdf5r TRUE
#> 8 jsonlite TRUE
#> 9 jsonlite TRUE
#> 10 writexl or openxlsx TRUE
#> 11 haven TRUE
#> 12 haven TRUE
#> 13 - TRUE
#> 14 haven TRUE
#> 15 fst TRUEOnly CSV, TSV, RDS and RData need nothing beyond base R. The other
backends are optional: install the package listed in the
read/write columns for the formats you need
(for example install.packages("arrow") for Parquet and
Feather). The available column shows what is installed on
this machine. Text formats can be compressed by appending
.gz, .bz2 or .xz.
dir <- tempfile("sp-formats-")
dir.create(dir)
syn <- synthesize(iris, n = 300, seed = 1)
write_data(syn, file.path(dir, "iris.csv"))
write_data(syn, file.path(dir, "iris.tsv.gz"))
write_data(syn, file.path(dir, "iris.rds"))
list.files(dir)
#> [1] "iris.csv" "iris.rds" "iris.tsv.gz"Text formats do not store R classes. Pass a template to
restore them — factor levels, Date, POSIXct
(with time zone), integer and logical columns:
x <- read_data(file.path(dir, "iris.csv"))
class(x$Species)
#> [1] "character"
x <- read_data(file.path(dir, "iris.csv"), template = iris)
levels(x$Species)
#> [1] "setosa" "versicolor" "virginica"match_types() does the same for any data frame you read
by other means.
Columnar formats are the usual choice for large data in data science pipelines (Spark, Python/pandas, DuckDB, Polars). Parquet is read and written with arrow or, if that is not installed, the lightweight nanoparquet; Feather / Arrow IPC needs arrow.
f <- file.path(dir, "iris.parquet")
write_data(syn, f)
str(read_data(f))
#> 'data.frame': 300 obs. of 5 variables:
#> $ Sepal.Length: num 5.8 5.91 5.7 5.68 7.44 ...
#> $ Sepal.Width : num 2.5 2.6 2.8 3.1 2.5 ...
#> $ Petal.Length: num 4 4.9 4.6 5.1 6.11 ...
#> $ Petal.Width : num 1.7 1.8 1.73 1.8 2.3 ...
#> $ Species : Factor w/ 3 levels "setosa","versicolor",..: 3 3 3 3 3 2 3 1 3 3 ...HDF5 is common in scientific computing and is readable from Python
(h5py, pandas), MATLAB and Julia.
write_data() stores one dataset per column in a group
(dataset = "data" by default), with attributes that record
column names, factor levels, dates and time zones;
read_data() restores the data frame exactly. A plain 1- or
2-dimensional dataset written by other software can be read by giving
its name in dataset.
write_data(head(syn, 2), file.path(dir, "iris.json"))
cat(readLines(file.path(dir, "iris.json")), sep = "\n")
#> [{"Sepal.Length":5.8,"Sepal.Width":2.5,"Petal.Length":4,"Petal.Width":1.7,"Species":"virginica"},{"Sepal.Length":5.91383310270021,"Sepal.Width":2.6,"Petal.Length":4.9,"Petal.Width":1.8,"Species":"virginica"}]write_data(syn, file.path(dir, "iris.sav")) # SPSS
str(read_data(file.path(dir, "iris.sav")))
#> 'data.frame': 300 obs. of 5 variables:
#> $ Sepal.Length: num 5.8 5.91 5.7 5.68 7.44 ...
#> $ Sepal.Width : num 2.5 2.6 2.8 3.1 2.5 ...
#> $ Petal.Length: num 4 4.9 4.6 5.1 6.11 ...
#> $ Petal.Width : num 1.7 1.8 1.73 1.8 2.3 ...
#> $ Species : Factor w/ 3 levels "setosa","versicolor",..: 3 3 3 3 3 2 3 1 3 3 ...
write_data(syn, file.path(dir, "iris.dta")) # Stata: names are made valid
#> Warning: Column names changed to valid Stata names: Sepal.Length ->
#> Sepal_Length, Sepal.Width -> Sepal_Width, Petal.Length -> Petal_Length,
#> Petal.Width -> Petal_Width
names(read_data(file.path(dir, "iris.dta")))
#> [1] "Sepal_Length" "Sepal_Width" "Petal_Length" "Petal_Width" "Species"Value labels from SPSS/Stata files are converted to factors when
reading. Stata only allows letters, digits and underscores in variable
names, so write_data() renames columns such as
Sepal.Length (with a warning).
generate_to_file() generates data in chunks, so the full
data set never has to fit in memory. Appendable formats (CSV, TSV,
NDJSON) are written to a single file; for other formats a directory of
part files is created, which read_data() — or
arrow::open_dataset(), Spark, DuckDB — reads as one
table.
fit <- fit_synthesizer(iris)
res <- generate_to_file(fit, n = 50000, path = file.path(dir, "big.csv"),
chunk_size = 10000, seed = 1)
nrow(read_data(file.path(dir, "big.csv")))
#> [1] 50000
res <- generate_to_file(fit, n = 50000, path = file.path(dir, "big_rds"),
format = "rds", chunk_size = 20000, seed = 1)
basename(res$files)
#> [1] "part-00001.rds" "part-00002.rds" "part-00003.rds"
nrow(read_data(file.path(dir, "big_rds")))
#> [1] 50000# e.g. 100 million rows as a partitioned Parquet data set
generate_to_file(fit, n = 1e8, path = "synthetic_parquet", format = "parquet",
chunk_size = 1e6, seed = 1)
ds <- arrow::open_dataset("synthetic_parquet")Output is reproducible for a given seed and
chunk_size.