Sometimes synthetic data should be as faithful as possible; sometimes
they should deliberately be further from the real data, for
example
One knob: closeness
grid <- c(1, 0.8, 0.6, 0.4, 0.2, 0)
dist <- t(sapply(grid, function(cl) {
syn <- generate(fit, closeness = cl, seed = 1)
c(closeness = cl,
`discriminator AUC` = discriminator_auc(Boston, syn, seed = 1),
`DCR ratio` = dcr(Boston, syn, seed = 1)$ratio,
`marginal distance` = mean(marginal_metrics(Boston, syn)$distance),
`association error` = mean(abs(association_matrix(syn) - association_matrix(Boston))))
}))
round(dist, 3)
#> closeness discriminator AUC DCR ratio marginal distance association error
#> [1,] 1.0 0.769 3.322 0.032 0.069
#> [2,] 0.8 0.888 4.503 0.151 0.196
#> [3,] 0.6 0.955 5.368 0.175 0.285
#> [4,] 0.4 0.970 6.107 0.190 0.352
#> [5,] 0.2 0.985 6.240 0.195 0.380
#> [6,] 0.0 0.981 6.520 0.200 0.394
As closeness decreases, a classifier distinguishes
synthetic from real records more easily (higher AUC, which levels off
close to 1 once the data are clearly different), synthetic records are
further from their nearest real record (higher DCR ratio, see
?dcr), and the marginal distributions and associations
drift away from the real ones.
Note that even at closeness = 1 the discriminator AUC is
above 0.5: the synthesizer is a model of the data and the classifier,
which uses squares and interactions of all variables, picks up the
remaining differences. closeness = 1 is the closest the
model can get; it is not a copy of the data.
Internally, closeness \(c\) is translated into a perturbation with
\(d = 1 - c\):
noise |
\(1.5\,d\) |
smoothing of continuous variables |
swap |
\(0.5\,d\) |
re-drawing of categorical/discrete values |
temperature |
\(1 + d\) |
flatter category probabilities |
dependence |
\(c\) |
weaker dependence between variables |
Record level: further from real records, same distributions
noise replaces each numeric value by a smoothed version
that approximately keeps the mean and variance of the variable;
swap re-draws categorical and discrete values from their
marginal distribution. The distributions stay (nearly) the same, but
individual synthetic records move away from the real ones.
near <- generate(fit, seed = 2)
far <- generate(fit, perturb = perturbation(noise = 0.6, swap = 0.3), seed = 2)
c(near = dcr(Boston, near, seed = 1)$ratio, far = dcr(Boston, far, seed = 1)$ratio)
#> near far
#> 3.392399 5.045538
round(rbind(real = sapply(Boston[c("medv", "lstat", "rm")], sd),
near = sapply(near[c("medv", "lstat", "rm")], sd),
far = sapply(far[c("medv", "lstat", "rm")], sd)), 2)
#> medv lstat rm
#> real 9.20 7.14 0.70
#> near 9.40 7.33 0.68
#> far 9.47 7.00 0.71
plot_dcr(dcr(Boston, far, seed = 1))
Distribution level: scenarios
shift, scale, temperature and
dependence change the distributions on purpose. They accept
a single number for all eligible variables or a named vector for
specific variables. Here is a scenario with higher crime rates, more
variable room counts and a stronger dependence between variables:
scenario <- perturbation(shift = c(crim = 1), scale = c(rm = 1.5), dependence = 1.3)
scenario
#> <sp_perturbation>
#> record level : noise 0 | swap 0
#> distribution level: shift (crim = 1) | scale (rm = 1.5) | temperature 1 | dependence 1.3
#> bounds : sign
stress <- generate(fit, perturb = scenario, seed = 3)
plot_marginals(Boston, stress, vars = c("crim", "rm", "medv"))
By default (bounds = "sign"), variables that are never
negative in the real data, such as crim, stay non-negative;
use bounds = "observed" to stay within the observed range,
or "none" for no restriction.
For categorical variables, temperature > 1 makes the
categories more equally frequent and temperature < 1
concentrates the data on the most frequent ones:
fit_i <- fit_synthesizer(iris[c(1:50, 51:70, 101:105), ]) # unbalanced species
round(rbind(
real = prop.table(table(iris[c(1:50, 51:70, 101:105), "Species"])),
`T = 1` = prop.table(table(generate(fit_i, n = 5000, seed = 1)$Species)),
`T = 3` = prop.table(table(generate(fit_i, n = 5000, seed = 1,
perturb = perturbation(temperature = 3))$Species))
), 2)
#> setosa versicolor virginica
#> real 0.67 0.27 0.07
#> T = 1 0.67 0.26 0.07
#> T = 3 0.45 0.33 0.21
closeness and perturb can be used together:
their record-level effects add up, so
generate(fit, closeness = 0.8, perturb = scenario) produces
a smoothed version of the scenario.
Hitting a target distance
Rather than guessing a value of closeness, state the
distance you want. calibrate_closeness() evaluates a metric
over a grid of closeness values (with the same random numbers each
time), refines the solution by bisection and finishes with a linear
interpolation step. Here we ask for synthetic data that a classifier can
distinguish from the real data with an AUC of 0.85:
cal <- calibrate_closeness(fit, Boston, target = 0.85, metric = "auc",
grid = seq(0, 1, by = 0.1), refine = 3)
cal
#> <sp_calibration>
#> metric : auc (target 0.85)
#> closeness : 0.887 (achieved auc = 0.845)
#> use: generate(fit, closeness = <value>)
plot(cal)
The metric is itself an estimate, so a check with a fresh seed gives
a value close to, but not exactly at, the target:
syn <- generate(fit, closeness = cal$closeness, seed = 99)
discriminator_auc(Boston, syn, seed = 5)
#> [1] 0.8343163
Other targets are the DCR ratio (metric = "dcr", useful
as a disclosure risk threshold), the pMSE ratio and the mean marginal
distance. Directional perturbations can be kept fixed while calibrating:
calibrate_closeness(fit, Boston, 0.9, perturb = scenario).
Use with augmentation
Lower closeness yields more diverse synthetic training data for
machine learning. augment_data() passes
closeness and perturb on:
big <- augment_data(Boston[1:60, ], n = 1000, closeness = 0.8, seed = 1)
dim(big)
#> [1] 1060 14
Whether more diversity helps a given model is an empirical question;
tune closeness by cross-validation on the training data as
shown in vignette("regularisation").