Evaluating and visualising synthetic data

Synthetic data are only useful if they are faithful to the real data, useful for the analyses people will run, and safe to release. No single number captures all three, so SynthesizerPlus provides a set of complementary measures and plots.

Everything at once: `compare_synthetic()`

cmp <- compare_synthetic(real, syn, seed = 1)
cmp
#> <sp_comparison> real: 153 rows | synthetic: 153 rows | 6 variables
#> 
#> Marginal distributions (KS for numeric, TVD for categorical):
#>  variable statistic distance wasserstein missing_real missing_synthetic
#>     Ozone        KS    0.065       0.094        0.242             0.248
#>   Solar.R        KS    0.079       0.107        0.046             0.105
#>      Wind        KS    0.052       0.061        0.000             0.000
#>      Temp        KS    0.065       0.108        0.000             0.000
#>     Month        KS    0.059       0.144        0.000             0.000
#>       Day        KS    0.105       0.205        0.000             0.000
#> 
#> Association: mean |difference| = 0.091
#> Utility: pMSE = 0.00604 (ratio to null = 1.85), discriminator AUC = 0.457
#> Privacy: exact copies = 0.00%, median DCR ratio (synthetic / real) = 1.02
summary(cmp)
#>                          metric      value ideal
#> 1        mean marginal distance 0.07086339     0
#> 2         max marginal distance 0.10457516     0
#> 3 mean |association difference| 0.09131944     0
#> 4                    pMSE ratio 1.84753033    ~1
#> 5             discriminator AUC 0.45713187   0.5
#> 6               exact copy rate 0.00000000     0
#> 7                     DCR ratio 1.01892849  >= 1

The ideal column of summary() gives the value expected for a perfect synthesizer that still produces new records. The components are described below.

Marginal fidelity

marginal_metrics() reports, for each variable, the Kolmogorov–Smirnov distance (numeric) or the total variation distance (categorical), the Wasserstein-1 distance in units of the real standard deviation, and moments and missing-value rates.

marginal_metrics(real, syn)[, c("variable", "distance", "wasserstein",
                                "mean_real", "mean_synthetic",
                                "missing_real", "missing_synthetic")]
#>   variable   distance wasserstein  mean_real mean_synthetic missing_real
#> 1    Ozone 0.06499250  0.09365719  42.129310      41.826087   0.24183007
#> 2  Solar.R 0.07914209  0.10715300 185.931507     193.970803   0.04575163
#> 3     Wind 0.05228758  0.06098008   9.957516       9.803409   0.00000000
#> 4     Temp 0.06535948  0.10783303  77.882353      77.379085   0.00000000
#> 5    Month 0.05882353  0.14411924   6.993464       6.790850   0.00000000
#> 6      Day 0.10457516  0.20461319  15.803922      14.039216   0.00000000
#>   missing_synthetic
#> 1         0.2483660
#> 2         0.1045752
#> 3         0.0000000
#> 4         0.0000000
#> 5         0.0000000
#> 6         0.0000000

plot_marginals(real, syn)

plot_qq(real, syn)

For categorical variables, plot_marginals() (or plot_categorical()) shows the category shares side by side:

syn_iris <- synthesize(iris, by = "Species", seed = 1)
plot_categorical(iris, syn_iris)

Dependence

association_matrix() works for mixed data: Spearman correlation for pairs of numeric variables, bias-corrected Cramér’s V for pairs of categorical variables and the correlation ratio for mixed pairs.

plot_association(real, syn, vars = c("Ozone", "Solar.R", "Wind", "Temp"))

General utility: can a classifier tell them apart?

If real and synthetic records are exchangeable, a classifier should not be able to predict which set a record comes from.

pMSE (Snoke et al., 2018) is the mean squared deviation of the estimated propensity scores from the share of synthetic records. Its ratio to the expected value under the null of identical distributions should be close to 1.
The discriminator AUC is computed from cross-validated predictions; 0.5 means indistinguishable.

pmse(real, syn, seed = 1)
#> $pmse
#> [1] 0.006037681
#> 
#> $pmse_null
#> [1] 0.003267974
#> 
#> $pmse_ratio
#> [1] 1.84753
#> 
#> $n_params
#> [1] 9
discriminator_auc(real, syn, seed = 1)
#> [1] 0.4571319

Specific utility: do analyses give the same answers?

ci_overlap() fits the same regression to both data sets and compares the confidence intervals (Karr et al., 2006):

ci_overlap(real, syn, Ozone ~ Temp + Wind + Solar.R)[, c("term", "estimate_real",
                                                        "estimate_synthetic",
                                                        "std_diff", "overlap")]
#>          term estimate_real estimate_synthetic   std_diff   overlap
#> 1 (Intercept)  -64.34207893       -93.34779040 -1.2581244 0.6612310
#> 2        Temp    1.65209291         1.83531124  0.7226698 0.8053111
#> 3        Wind   -3.33359131        -1.94885902  2.1160105 0.4551058
#> 4     Solar.R    0.05982059         0.06740518  0.3271126 0.9152968

Overlaps close to 1 mean that inference on synthetic data leads to the same conclusions.

Disclosure risk: distance to closest record

A synthesizer that copies real records has perfect fidelity but no privacy. dcr() computes, for each synthetic record, the Gower distance to the closest real record, and compares it to how close real records are to each other.

risk <- dcr(real, syn, seed = 1)
risk$quantiles
#>                   q1         q5        q25        q50
#> synthetic 0.01680262 0.02858173 0.04705751 0.06706663
#> real      0.01776686 0.03058257 0.04446862 0.06582074
risk$exact_match_rate
#> [1] 0
plot_dcr(risk)

Exact copies, or a synthetic distance distribution well to the left of the real one, are warning signs. Remedies include knots (coarser quantile functions), lowering dependence, or a larger min_stratum_size.

Plots from a comparison object

All plots are also available from the comparison object:

plot(cmp, type = "marginals")
plot(cmp, type = "association")
plot(cmp, type = "pairs")
plot(cmp, type = "qq")
plot(cmp, type = "dcr")
ggplot2::autoplot(cmp, type = "association") + ggplot2::ggtitle("airquality")

Every plotting function returns a ggplot object, so themes, scales and labels can be changed with the usual ggplot2 syntax.

References

Karr, A. F., Kohnen, C. N., Oganian, A., Reiter, J. P. and Sanil, A. P. (2006). A framework for evaluating the utility of data altered to protect confidentiality. The American Statistician, 60(3), 224–232.

Snoke, J., Raab, G. M., Nowok, B., Dibben, C. and Slavkovic, A. (2018). General and specific utility measures for synthetic data. Journal of the Royal Statistical Society A, 181(3), 663–688.