LUCID’s Three Model Architectures: Early, Parallel, and Serial – Binary Outcome (HELIX Example)

1) Tutorial Goal and Scope

This tutorial is designed as a hands-on, end-to-end guide for fitting LUCID models on HELIX-style multi-omics data, with a binary outcome. Its companion vignette, lucid_3models_normal_outcome.Rmd, walks through the identical pipeline step by step for a continuous outcome, including the lucid() tuning wrapper, missing-data diagnostics, and prediction/g-computation – material this vignette does not repeat. Visualization (Sankey diagram, cluster omics profiles) is shown here too, since neither depends on the outcome family.

What you will learn here:

Important runtime note:

2) Data Objects and Statistical Roles

The HELIX example data (simulated_HELIX_data.rda) provides:

Model inputs used throughout:

3) Hyperparameter Guide (Practical)

Hyperparameter Meaning Tutorial choice and rationale
K number of latent clusters fixed to small values for speed and interpretability
Rho_G penalty on G -> X coefficients positive in screening fit, zero in inference refit
Rho_Z_Mu penalty on cluster-specific omics means positive in screening fit, zero in inference refit
Rho_Z_Cov penalty on omics covariance matrices positive in screening fit, zero in inference refit
max_itr, max_tot.itr, tol EM controls modest values to balance speed/stability
family outcome model family normal for this tutorial
seed reproducibility fixed before each fit/bootstrap

4) Setup and Source Package Code

# Keep knitting on error, so that one failing step reports itself and the rest of
# the tutorial still runs. The status table in section 13 records what happened.
knitr::opts_chunk$set(error = TRUE)

# Lightweight registry so the document can verify itself rather than relying on
# the reader to notice a missing output.
.reg <- new.env(parent = emptyenv()); .reg$rows <- list()
check_obj <- function(name, expected_class = NULL, section = "") {
  ok <- exists(name, envir = globalenv())
  cls <- if (ok) class(get(name, envir = globalenv()))[1] else NA_character_
  status <- if (!ok) "MISSING"
            else if (!is.null(expected_class) && !identical(cls, expected_class)) "unexpected class"
            else "ok"
  .reg$rows[[length(.reg$rows) + 1L]] <-
    data.frame(section = section, object = name, class = cls,
               status = status, stringsAsFactors = FALSE)
  invisible(NULL)
}

library(LUCIDus)

# The HELIX simulation object bundled with the package.
data(simulated_HELIX_data)

5) Build Modeling Inputs (With Missingness Injection)

This chunk creates a compact tutorial dataset and deliberately injects both:

so we can observe missing-data handling in summaries.

# Use a smaller subset for vignette speed while preserving model behavior.
idx <- 1:90
ph <- simulated_HELIX_data$phenotype[idx, ]
n <- nrow(ph)

set.seed(2026)

# ---------------------------------------------------------------------------
# A tutorial dataset with a KNOWN answer.
#
# The HELIX omics matrices are real simulated data with their own structure, and
# the exposures shipped with them have no relationship to it. That is fine for
# demonstrating that code runs, but it makes feature selection impossible to
# judge: there is no right answer to compare against. So we plant one.
#
# The generating story, which is the DAG LUCID assumes:
#
#     causal exposures  ->  latent subgroup  ->  omics profile
#                                            ->  outcome
#
# Three exposures carry the subgroup signal with graded strength; six are pure
# noise. Half the features of each omics layer are shifted by subgroup
# membership; the rest are left as they came. Selection therefore has an
# unambiguous target, and the tutorial can check its answer instead of asserting
# it.
# ---------------------------------------------------------------------------

# The true latent subgroup. Retained so every selection claim below can be
# checked against it.
x_true <- rbinom(n, 1, 0.5)

# Exposures. g_causal_* predict subgroup membership; g_noise_* do not.
# Effect sizes are deliberately moderate. Stronger exposures make selection
# look better but drive the G -> X model to saturation, where every subject sits
# at posterior probability 1 and no counterfactual shift can move anything --
# which would make the g-computation demonstration in the continuous-outcome
# companion vignette vacuous.
G <- cbind(
  g_causal_1 =  1.0 * (x_true - 0.5) + rnorm(n, sd = 0.8),   # strongest
  g_causal_2 = -0.8 * (x_true - 0.5) + rnorm(n, sd = 0.8),   # moderate, negative
  g_causal_3 =  0.6 * (x_true - 0.5) + rnorm(n, sd = 0.8),   # weakest
  g_noise_1 = rnorm(n), g_noise_2 = rnorm(n), g_noise_3 = rnorm(n),
  g_noise_4 = rnorm(n), g_noise_5 = rnorm(n), g_noise_6 = rnorm(n)
)
G <- as.matrix(scale(G))

causal_exposures <- c("g_causal_1", "g_causal_2", "g_causal_3")

# Exposure penalty used throughout. The continuous-outcome companion vignette
# shows what this value recovers and sweeps it, along with the omics penalty,
# separately.
RHO_G <- 0.05

# Covariates for G->X (CoG) and X->Y (CoY).
# Here we use age-related and sex covariates from phenotype.
CoG <- cbind(
  hs_child_age_yrs_None = as.numeric(ph$hs_child_age_yrs_None),
  sex_male = as.numeric(ph$e3_sex_None == "male")
)
CoY <- CoG

# Two outcomes on the SAME subjects, so the normal and binary results below are
# directly comparable: the only thing that changes between them is the outcome
# model, not the sample, the omics, or the injected missingness.
#
# Continuous outcome: the real CK-18 measurement, plus a subgroup effect so the
# cluster -> outcome arm of the model has something to estimate.
Y <- as.numeric(ph$ck18_scaled) + 1.2 * x_true

# Binary outcome: median split. The median is used rather than a higher
# threshold because it splits these 90 subjects 45/45, and a balanced outcome
# gives the K = 2 outcome model the most to work with at this sample size.
Y_binary <- as.integer(Y > median(Y))
cat("binary outcome balance:\n"); print(table(Y_binary))
## binary outcome balance:
## Y_binary
##  0  1 
## 45 45
# Construct three omics layers and standardize each, then plant the subgroup
# signal in the first three features of every layer. The remaining seven per
# layer are left as they came and act as omics noise.
meth <- scale(simulated_HELIX_data$methylome[idx, 1:10, drop = FALSE])
tran <- scale(simulated_HELIX_data$transcriptome[idx, 1:10, drop = FALSE])
mir  <- scale(simulated_HELIX_data$miRNA[idx, 1:10, drop = FALSE])

signal_features <- 1:3
omics_shift <- 3.0
meth[, signal_features] <- meth[, signal_features] + omics_shift * x_true
tran[, signal_features] <- tran[, signal_features] - omics_shift * x_true
mir[,  signal_features] <- mir[,  signal_features] + omics_shift * x_true

# Column positions of the signal features once the layers are stacked for the
# early model, so selection can be scored against them later.
signal_cols_early <- c(signal_features,
                       ncol(meth) + signal_features,
                       ncol(meth) + ncol(tran) + signal_features)

# Early model uses one combined Z matrix.
Z_early <- cbind(meth, tran, mir)

# Parallel model uses list-of-layers.
Z_parallel <- list(methylome = meth, transcriptome = tran, miRNA = mir)

# Inject listwise + sporadic missingness for demonstration.
Z_early_miss <- Z_early
Z_early_miss[1, ] <- NA      # listwise row
Z_early_miss[2:4, 1] <- NA   # sporadic block
Z_early_miss[5, 3] <- NA     # sporadic cell

Z_parallel_miss <- Z_parallel
Z_parallel_miss[[1]][1, ] <- NA  # listwise in layer 1
Z_parallel_miss[[2]][2, 2] <- NA # sporadic in layer 2
Z_parallel_miss[[3]][3, 1] <- NA # sporadic in layer 3

# Quick structural sanity check.
str(list(
  G = G,
  CoG = CoG,
  CoY = CoY,
  Y = Y,
  Z_early = Z_early_miss,
  Z_parallel = Z_parallel_miss
), max.level = 1)
## List of 6
##  $ G         : num [1:90, 1:9] 1.587 1.051 -0.744 0.168 0.834 ...
##   ..- attr(*, "dimnames")=List of 2
##   ..- attr(*, "scaled:center")= Named num [1:9] -0.0377 0.0866 0.0539 -0.0695 0.1933 ...
##   .. ..- attr(*, "names")= chr [1:9] "g_causal_1" "g_causal_2" "g_causal_3" "g_noise_1" ...
##   ..- attr(*, "scaled:scale")= Named num [1:9] 1.053 0.864 0.732 1.038 1.211 ...
##   .. ..- attr(*, "names")= chr [1:9] "g_causal_1" "g_causal_2" "g_causal_3" "g_noise_1" ...
##  $ CoG       : num [1:90, 1:2] 7.48 7.21 8.59 8.48 6.06 ...
##   ..- attr(*, "dimnames")=List of 2
##  $ CoY       : num [1:90, 1:2] 7.48 7.21 8.59 8.48 6.06 ...
##   ..- attr(*, "dimnames")=List of 2
##  $ Y         : num [1:90] 1.817 1.67 0.479 -0.281 0.394 ...
##  $ Z_early   : num [1:90, 1:30] NA NA NA NA 3.17 ...
##   ..- attr(*, "dimnames")=List of 2
##  $ Z_parallel:List of 3

6) Helper Functions for Selected-Feature Refit

These helpers implement a robust refit pipeline:

  1. Read feature-selection indicators from penalized fit.
  2. Build selected-only G/Z inputs.
  3. Refit with all penalties set to zero for bootstrap inference.
# get_selected_G()/get_selected_Z() (from the package itself) already return a
# well-shaped, aligned logical mask straight from the fitted object -- no
# length mismatch is possible, since they derive it from the model's own
# recorded fields. The one thing left for a tutorial to decide is what to do
# if a penalty happened to deselect EVERY feature: refitting on zero columns
# would fail, so this keeps everything instead in that one edge case.
keep_or_all <- function(mask) if (any(mask, na.rm = TRUE)) mask else rep(TRUE, length(mask))

# Build selected-only inputs for early model.
prepare_early_selected_inputs <- function(fit_pen, G, Z) {
  list(
    G = as.matrix(G[, keep_or_all(get_selected_G(fit_pen)), drop = FALSE]),
    Z = as.matrix(Z[, keep_or_all(get_selected_Z(fit_pen)), drop = FALSE])
  )
}

# Build selected-only inputs for parallel model.
prepare_parallel_selected_inputs <- function(fit_pen, G, Z) {
  keep_g <- keep_or_all(get_selected_G(fit_pen))
  Z_sel <- lapply(seq_along(Z), function(i) {
    zi <- as.matrix(Z[[i]])
    zi[, keep_or_all(get_selected_Z(fit_pen, layer = i)), drop = FALSE]
  })
  names(Z_sel) <- names(Z)
  list(
    G = as.matrix(G[, keep_g, drop = FALSE]),
    Z = Z_sel
  )
}

# Serial stage>1 uses latent-cluster-derived "G" internally.
# We therefore subset stage-1 original G and each stage's Z where applicable.
prepare_serial_selected_inputs <- function(fit_pen, G, Z) {
  G_refit <- as.matrix(G)
  keep_g1 <- get_selected_G(fit_pen)
  if (length(keep_g1) == ncol(G_refit)) {
    G_refit <- G_refit[, keep_or_all(keep_g1), drop = FALSE]
  }

  selected_z <- get_selected_Z(fit_pen)
  Z_refit <- Z
  for (i in seq_along(fit_pen$submodel)) {
    sm <- fit_pen$submodel[[i]]
    if (inherits(sm, "early_lucid")) {
      zi <- as.matrix(Z_refit[[i]])
      Z_refit[[i]] <- zi[, keep_or_all(selected_z[[i]]), drop = FALSE]
    } else if (inherits(sm, "lucid_parallel")) {
      zi_list <- Z_refit[[i]]
      for (j in seq_along(zi_list)) {
        zij <- as.matrix(zi_list[[j]])
        zi_list[[j]] <- zij[, keep_or_all(selected_z[[i]][[j]]), drop = FALSE]
      }
      Z_refit[[i]] <- zi_list
    }
  }

  list(G = G_refit, Z = Z_refit)
}

# Zero-penalty refit, for any model type.
#
# The three model types previously had three byte-identical wrappers differing
# only in `lucid_model` and whether `useY` was forwarded; they are one function
# here. Everything about the model -- family, K, initialization, EM controls --
# is carried over from the screening fit, so the ONLY difference between the
# screening fit and this one is that the penalties are zero. That is what makes
# the refit estimates unshrunk and therefore suitable for bootstrap inference.
refit_selected <- function(model_type, fit_pen, inputs, Y,
                           CoG = NULL, CoY = NULL, seed = 1, verbose = FALSE) {
  args <- list(
    lucid_model = model_type,
    G = inputs$G,
    Z = inputs$Z,
    Y = Y,
    CoG = CoG,
    CoY = CoY,
    family = fit_pen$family,
    K = fit_pen$K,
    init_omic.data.model = fit_pen$init_omic.data.model,
    init_impute = fit_pen$init_impute,
    init_par = fit_pen$init_par,
    Rho_G = 0,
    Rho_Z_Mu = 0,
    Rho_Z_Cov = 0,
    max_itr = fit_pen$em_control$max_itr,
    max_tot.itr = fit_pen$em_control$max_tot.itr,
    tol = fit_pen$em_control$tol,
    seed = seed,
    verbose = verbose
  )
  # Every fitted class records useY, so it is carried over for all three model
  # types. The original three wrappers omitted it on the early path, which meant
  # an unsupervised screening fit would have been silently refitted supervised.
  args$useY <- fit_pen$useY
  do.call(estimate_lucid, args)
}

# Dispatcher for the three input-preparation helpers above.
prepare_selected_inputs <- function(model_type, fit_pen, G, Z) {
  switch(model_type,
    early    = prepare_early_selected_inputs(fit_pen, G, Z),
    parallel = prepare_parallel_selected_inputs(fit_pen, G, Z),
    serial   = prepare_serial_selected_inputs(fit_pen, G, Z),
    stop("unknown model_type: ", model_type)
  )
}


# Compact stage-wise feature-selection report for serial fits, built entirely
# from get_selected_G()/get_selected_Z() -- no per-stage dispatch of its own.
serial_selection_report <- function(fit_serial_pen) {
  selected_z <- get_selected_Z(fit_serial_pen)
  out <- vector("list", length(fit_serial_pen$submodel))
  for (i in seq_along(fit_serial_pen$submodel)) {
    sm <- fit_serial_pen$submodel[[i]]
    if (inherits(sm, "early_lucid")) {
      out[[i]] <- list(
        stage = i,
        model = "early",
        selected_G = if (i == 1) sum(get_selected_G(fit_serial_pen)) else NA,
        total_G = if (i == 1) length(get_selected_G(fit_serial_pen)) else NA,
        selected_Z = sum(selected_z[[i]]),
        total_Z = length(selected_z[[i]])
      )
    } else {
      out[[i]] <- list(
        stage = i,
        model = "parallel",
        selected_G = if (i == 1) sum(get_selected_G(fit_serial_pen)) else NA,
        total_G = if (i == 1) length(get_selected_G(fit_serial_pen)) else NA,
        selected_Z_by_layer = sapply(selected_z[[i]], sum),
        total_Z_by_layer = sapply(selected_z[[i]], length)
      )
    }
  }
  out
}

7) Early Model Tutorial: Binary Outcome

Y_binary is the median split of the continuous outcome built in section 5, on the same subjects, with the same injected missingness. Fitting proceeds in the same three explicit steps as the continuous-outcome companion vignette: penalized screening fit, zero-penalty refit on the survivors, then bootstrap. Only the outcome model changes.

7.1 Penalized screening fit

set.seed(1105)

early_pen_bin <- estimate_lucid(
  lucid_model = "early",
  G = G,
  Z = Z_early_miss,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  family = "binary",
  K = 2,
  Rho_G = RHO_G,
  Rho_Z_Mu = 0,
  Rho_Z_Cov = 0,
  max_itr = 15,
  max_tot.itr = 40,
  tol = 1e-2,
  seed = 1105,
  verbose = FALSE
)
## Fitting LUCID early model (K = 2)...
## Finished LUCID early model. Selected G: 4/9; Selected Z: 30/30.
summary(early_pen_bin)
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 1 / 90 (1.1%)
##   Sporadic missing rows  : 4 / 90 (4.4%)
##   Missing cells total    : 34 / 2700 (1.3%)
## 
## Feature selection overview
##   G features selected    : 4 / 9 (44.4%)
##   Z features selected    : 30 / 30 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -2905.97
##   BIC                    : 10316.25
##   Number of parameters   : 1001
## 
## Regularization
##   Rho_G                  : 0.050
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): log odds of Y for cluster 1 (reference) and log OR for rest cluster (and log OR of covariate if included)
##                             gamma exp(gamma)
## (Intercept)           -0.12272266  0.8845089
## LC2                    0.46969468  1.5995058
## hs_child_age_yrs_None -0.04731969  0.9537824
## sex_male               0.20877813  1.2321716
## 
## (2) Z: mean of omics data for each latent cluster 
##                   mu_cluster1  mu_cluster2
## cg_GRHL3          1.070130027  1.467807192
## cg_BTF3L4         1.177269715  1.448826628
## cg_AL358472.7     1.111915428  1.503179122
## cg_HDGF           0.039776680 -0.007386556
## cg_TDRD5          0.060508379 -0.025499538
## cg_CSRNP3        -0.315770044  0.106570901
## cg_HSPD1          0.247344951 -0.091051585
## cg_EPM2AIP1       0.095823370 -0.031917066
## cg_AC025171.1    -0.115955949  0.016990794
## cg_VTRNA1_3       0.008624239 -0.017515088
## tc_TC01006069_nc -1.032187726 -1.500901334
## tc_SLC9A4        -1.107493413 -1.473018657
## tc_RAB6C_AS1     -1.041453398 -1.509522359
## tc_LOC100129029   0.268771608 -0.068905489
## tc_BRE           -0.173495627  0.059580555
## tc_TC03001220_nc  0.135402054 -0.075080553
## tc_TC04002114_nc  0.603537655 -0.187588817
## tc_TC04002369_nc -0.479441927  0.160477541
## tc_BEND4         -0.216113295  0.076788618
## tc_SLC9A3         0.169377726 -0.032648173
## miR.101.3p        0.028992443  1.824713207
## miR.125a.5p       0.785178492  1.588652258
## miR.125b.1.3p     1.448589102  1.366871494
## miR.127.3p       -0.032178676  0.021562673
## miR.140.5p       -0.804412403  0.265078938
## miR.142.3p       -1.186015202  0.381068666
## miR.144.5p       -1.122938565  0.360302094
## miR.19a.3p       -1.102814044  0.358147262
## miR.19b.3p       -1.087845497  0.355544328
## miR.21.5p        -0.882421524  0.294568136
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure 
##                                       beta         OR
## (Intercept).cluster2            2.42580914 11.3113782
## g_causal_2.cluster2            -0.04231731  0.9585656
## g_causal_3.cluster2            -0.13739149  0.8716289
## g_noise_3.cluster2              0.18897532  1.2080111
## g_noise_4.cluster2              0.32063240  1.3779989
## hs_child_age_yrs_None.cluster2 -0.13946483  0.8698236
## sex_male.cluster2              -0.51128754  0.5997229

7.2 Zero-penalty selected-only refit

set.seed(1106)

early_inputs_bin <- prepare_early_selected_inputs(early_pen_bin, G, Z_early_miss)

early_bin <- list(fit_pen = early_pen_bin, inputs = early_inputs_bin)
early_bin$fit_refit <- refit_selected(
  "early",
  fit_pen = early_pen_bin,
  inputs = early_inputs_bin,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  seed = 1106
)
## Fitting LUCID early model (K = 2)...
## Finished LUCID early model.
summary(early_bin$fit_refit)
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 1 / 90 (1.1%)
##   Sporadic missing rows  : 4 / 90 (4.4%)
##   Missing cells total    : 34 / 2700 (1.3%)
## 
## Feature selection overview
##   G features selected    : 4 / 4 (100.0%)
##   Z features selected    : 30 / 30 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -2884.76
##   BIC                    : 10273.84
##   Number of parameters   : 1001
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): log odds of Y for cluster 1 (reference) and log OR for rest cluster (and log OR of covariate if included)
##                             gamma exp(gamma)
## (Intercept)           -0.26156022  0.7698495
## LC2                    0.56721855  1.7633555
## hs_child_age_yrs_None -0.03726512  0.9634207
## sex_male               0.20745520  1.2305426
## 
## (2) Z: mean of omics data for each latent cluster 
##                  mu_cluster1  mu_cluster2
## cg_GRHL3          1.03142172  1.485669045
## cg_BTF3L4         1.13354093  1.468186310
## cg_AL358472.7     1.10522727  1.513235444
## cg_HDGF           0.08486240 -0.023816710
## cg_TDRD5          0.13378426 -0.052344620
## cg_CSRNP3        -0.22314677  0.080690254
## cg_HSPD1          0.31181222 -0.118653340
## cg_EPM2AIP1       0.04510631 -0.016176138
## cg_AC025171.1    -0.17656817  0.040133586
## cg_VTRNA1_3      -0.03825754 -0.001570394
## tc_TC01006069_nc -0.99865039 -1.519698204
## tc_SLC9A4        -1.00275913 -1.515067178
## tc_RAB6C_AS1     -1.04855340 -1.514144958
## tc_LOC100129029   0.26480149 -0.072641897
## tc_BRE           -0.16650572  0.060678315
## tc_TC03001220_nc  0.12780429 -0.075623850
## tc_TC04002114_nc  0.52738071 -0.173039573
## tc_TC04002369_nc -0.47160615  0.167449463
## tc_BEND4         -0.19013278  0.072174181
## tc_SLC9A3         0.13908256 -0.025151868
## miR.101.3p        0.02352182  1.853848697
## miR.125a.5p       0.78364918  1.601368451
## miR.125b.1.3p     1.35534448  1.398133617
## miR.127.3p       -0.07994581  0.039027202
## miR.140.5p       -0.77362299  0.270563806
## miR.142.3p       -1.13144910  0.385811002
## miR.144.5p       -1.05248948  0.358236957
## miR.19a.3p       -1.08798247  0.375130264
## miR.19b.3p       -1.06082542  0.368012483
## miR.21.5p        -0.83772616  0.296835971
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure 
##                                      beta         OR
## (Intercept).cluster2            4.1549935 63.7515487
## g_causal_2.cluster2            -0.6992095  0.4969780
## g_causal_3.cluster2            -0.6882463  0.5024565
## g_noise_3.cluster2              0.4955328  1.6413725
## g_noise_4.cluster2              0.9527485  2.5928261
## hs_child_age_yrs_None.cluster2 -0.3368294  0.7140306
## sex_male.cluster2              -0.5743772  0.5630554

7.3 Bootstrap CI + summary

set.seed(1107)

early_bin$boot <- boot_lucid(
  G = early_inputs_bin$G,
  Z = early_inputs_bin$Z,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  model = early_bin$fit_refit,
  R = 2,
  conf = 0.90
)

summary(early_bin$fit_refit, boot.se = early_bin$boot)
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 1 / 90 (1.1%)
##   Sporadic missing rows  : 4 / 90 (4.4%)
##   Missing cells total    : 34 / 2700 (1.3%)
## 
## Feature selection overview
##   G features selected    : 4 / 4 (100.0%)
##   Z features selected    : 30 / 30 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -2884.76
##   BIC                    : 10273.84
##   Number of parameters   : 1001
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): log odds of Y for cluster 1 (reference) and log OR for rest cluster (and log OR of covariate if included)
##                             gamma  norm_lower   norm_upper sig
## (Intercept)           -0.26156022 -0.01938759  5.415545450    
## LC2                    0.56721855 -0.60791793  0.772206375    
## hs_child_age_yrs_None -0.03726512 -0.85666607 -0.007585115   *
## sex_male               0.20745520  0.25787428  1.028603527   *
## 
## (2) Z: mean of omics data for each latent cluster 
##                               estimate  norm_lower  norm_upper sig
## cg_GRHL3.cluster1          1.031421718  0.84555851  2.41488335   *
## cg_BTF3L4.cluster1         1.133540932  0.76081585  3.06130968   *
## cg_AL358472.7.cluster1     1.105227268  1.82371458  2.22214076   *
## cg_HDGF.cluster1           0.084862399 -0.08273520  0.45402216    
## cg_TDRD5.cluster1          0.133784259 -0.29959513  1.11244467    
## cg_CSRNP3.cluster1        -0.223146771 -0.58703250 -0.13375416   *
## cg_HSPD1.cluster1          0.311812221  0.31710611  0.84440777   *
## cg_EPM2AIP1.cluster1       0.045106312 -0.42568353  0.96128167    
## cg_AC025171.1.cluster1    -0.176568168 -1.04879284 -0.01723998   *
## cg_VTRNA1_3.cluster1      -0.038257539 -0.48044672 -0.04438684   *
## tc_TC01006069_nc.cluster1 -0.998650387 -1.89157466 -1.48145049   *
## tc_SLC9A4.cluster1        -1.002759132 -1.87597747 -1.48341077   *
## tc_RAB6C_AS1.cluster1     -1.048553404 -1.72101989 -1.07768108   *
## tc_LOC100129029.cluster1   0.264801495  0.31203606  0.60233768   *
## tc_BRE.cluster1           -0.166505718 -1.09536894 -0.12369679   *
## tc_TC03001220_nc.cluster1  0.127804285  0.07781712  0.12054394   *
## tc_TC04002114_nc.cluster1  0.527380708  0.37996482  1.97262568   *
## tc_TC04002369_nc.cluster1 -0.471606146 -1.74793005  0.07203090    
## tc_BEND4.cluster1         -0.190132779 -0.88872806 -0.22958455   *
## tc_SLC9A3.cluster1         0.139082560 -0.47237161  1.01648893    
## miR.101.3p.cluster1        0.023521821 -1.12264251  1.25541137    
## miR.125a.5p.cluster1       0.783649178  0.83572639  1.59359668   *
## miR.125b.1.3p.cluster1     1.355344480  1.63371566  3.08187096   *
## miR.127.3p.cluster1       -0.079945810 -0.36522143  0.40942704    
## miR.140.5p.cluster1       -0.773622986 -2.68077888 -0.05584291   *
## miR.142.3p.cluster1       -1.131449104 -3.82856922 -0.25724237   *
## miR.144.5p.cluster1       -1.052489480 -3.58260602 -0.20046092   *
## miR.19a.3p.cluster1       -1.087982474 -4.02211085 -0.04422464   *
## miR.19b.3p.cluster1       -1.060825418 -4.12746141  0.07363572    
## miR.21.5p.cluster1        -0.837726158 -3.24149362  0.05931470    
## cg_GRHL3.cluster2          1.485669045  0.76440824  1.26799192   *
## cg_BTF3L4.cluster2         1.468186310  0.46171400  1.00344920   *
## cg_AL358472.7.cluster2     1.513235444  0.35461088  1.29454445   *
## cg_HDGF.cluster2          -0.023816710 -0.20621615  0.18663670    
## cg_TDRD5.cluster2         -0.052344620 -0.19152378 -0.01014356   *
## cg_CSRNP3.cluster2         0.080690254  0.22473551  0.27310763   *
## cg_HSPD1.cluster2         -0.118653340 -0.52860131 -0.12729541   *
## cg_EPM2AIP1.cluster2      -0.016176138 -0.29704811  0.08834767    
## cg_AC025171.1.cluster2     0.040133586 -0.11780049  0.17724575    
## cg_VTRNA1_3.cluster2      -0.001570394  0.04280355  0.11482614   *
## tc_TC01006069_nc.cluster2 -1.519698204 -0.92018788 -0.91589584   *
## tc_SLC9A4.cluster2        -1.515067178 -1.66503805 -0.12785236   *
## tc_RAB6C_AS1.cluster2     -1.514144958 -1.13485336 -1.04891383   *
## tc_LOC100129029.cluster2  -0.072641897 -0.44820314  0.11105883    
## tc_BRE.cluster2            0.060678315  0.17347542  0.33117251   *
## tc_TC03001220_nc.cluster2 -0.075623850 -0.37921882 -0.24988890   *
## tc_TC04002114_nc.cluster2 -0.173039573 -0.71799806  0.23826191    
## tc_TC04002369_nc.cluster2  0.167449463 -0.03564733  0.70546867    
## tc_BEND4.cluster2          0.072174181  0.14519795  0.36072712   *
## tc_SLC9A3.cluster2        -0.025151868 -0.74619552  0.17976286    
## miR.101.3p.cluster2        1.853848697  0.79195475  2.55847891   *
## miR.125a.5p.cluster2       1.601368451  0.49712491  1.70646883   *
## miR.125b.1.3p.cluster2     1.398133617  0.16738802  1.36746524   *
## miR.127.3p.cluster2        0.039027202 -0.32719821  0.12468470    
## miR.140.5p.cluster2        0.270563806 -0.11897285  1.16890992    
## miR.142.3p.cluster2        0.385811002  0.32862946  1.22584031   *
## miR.144.5p.cluster2        0.358236957  0.45413015  1.03464665   *
## miR.19a.3p.cluster2        0.375130264  0.28843872  1.22642992   *
## miR.19b.3p.cluster2        0.368012483  0.36621304  1.20354652   *
## miR.21.5p.cluster2         0.296835971  0.21625484  1.04483443   *
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure 
##                       estimate norm_lower norm_upper sig
## g_causal_2.cluster2 -0.6992095 -1.6367413   0.517465    
## g_causal_3.cluster2 -0.6882463 -1.7880611  -1.565861   *
## g_noise_3.cluster2   0.4955328 -0.1301882   2.013555    
## g_noise_4.cluster2   0.9527485  1.1980018   2.647867   *

Three differences are worth noting in that output whenever an outcome switches from continuous to binary – they are the whole payload of the outcome-family axis, and apply identically to the parallel and serial models fitted below:

8) Parallel Model Tutorial: Binary Outcome

8.1 Penalized screening fit

set.seed(1205)

parallel_pen_bin <- estimate_lucid(
  lucid_model = "parallel",
  G = G,
  Z = Z_parallel_miss,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  family = "binary",
  K = c(2, 2, 2),
  Rho_G = RHO_G,
  Rho_Z_Mu = 0,
  Rho_Z_Cov = 0,
  max_itr = 15,
  max_tot.itr = 40,
  tol = 1e-2,
  seed = 1205,
  verbose = FALSE
)
## Fitting LUCID parallel model (3 layers)...
## Finished LUCID parallel model. Selected G: 3/9; Selected Z by layer: 10/10, 10/10, 10/10.
summary(parallel_pen_bin)
## 
## ====================================================
## LUCID Parallel: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binomial
##   Number of observations : 90
##   Clusters per layer     : 2, 2, 2
## 
## Missing-data profile by layer
##   Layer 1 listwise rows : 1 / 90 (1.1%)
##   Layer 1 sporadic rows : 0 / 90 (0.0%)
##   Layer 1 missing cells : 10 / 900 (1.1%)
##   Layer 2 listwise rows : 0 / 90 (0.0%)
##   Layer 2 sporadic rows : 1 / 90 (1.1%)
##   Layer 2 missing cells : 1 / 900 (0.1%)
##   Layer 3 listwise rows : 0 / 90 (0.0%)
##   Layer 3 sporadic rows : 1 / 90 (1.1%)
##   Layer 3 missing cells : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 3 / 9 (33.3%)
##   G features by layer
##     Layer 1              : 3 / 9 (33.3%)
##     Layer 2              : 3 / 9 (33.3%)
##     Layer 3              : 3 / 9 (33.3%)
##   Z features
##     Layer 1 selected     : 10 / 10 (100.0%)
##     Layer 1 multi-cluster: 10
##     Layer 2 selected     : 10 / 10 (100.0%)
##     Layer 2 multi-cluster: 10
##     Layer 3 selected     : 10 / 10 (100.0%)
##     Layer 3 multi-cluster: 10
## 
## Model fit statistics
##   Log-likelihood         : -3344.66
##   BIC                    : 8552.24
##   Number of parameters   : 414
## 
## Regularization
##   Rho_G                  : 0.050
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): intercept and log OR for non-reference clusters for each layer (and log OR of covariate if included)
##                              gamma   exp(gamma)
## (Intercept)           -15.27511103 2.323291e-07
## Layer1_LC2              0.17476067 1.190961e+00
## Layer2_LC2             14.66202952 2.331517e+06
## Layer3_LC2             16.65458497 1.709993e+07
## hs_child_age_yrs_None  -0.03654061 9.641189e-01
## sex_male               -0.21552887 8.061150e-01
## 
## (2) Z: mean of omics data for each latent cluster of each layer 
## Layer 1
## 
##               mu_cluster1 mu_cluster2
## cg_GRHL3       0.01956752  2.78717791
## cg_BTF3L4      0.02555562  2.77153309
## cg_AL358472.7 -0.05854745  2.85130770
## cg_HDGF        0.19385174 -0.19001699
## cg_TDRD5      -0.13904990  0.13392791
## cg_CSRNP3     -0.07421134  0.08042597
## cg_HSPD1       0.18528278 -0.20485556
## cg_EPM2AIP1   -0.03945871  0.03975783
## cg_AC025171.1 -0.07344147  0.04311865
## cg_VTRNA1_3    0.15848085 -0.18480017
## 
## Layer 2
## 
##                   mu_cluster1  mu_cluster2
## tc_TC01006069_nc -2.761402720 -0.148980731
## tc_SLC9A4        -3.020385364  0.062508410
## tc_RAB6C_AS1     -2.734391777 -0.173801608
## tc_LOC100129029  -0.047094195  0.043275766
## tc_BRE           -0.073350292  0.067403001
## tc_TC03001220_nc  0.029037497 -0.026683118
## tc_TC04002114_nc -0.032967723  0.030294678
## tc_TC04002369_nc  0.133840868 -0.122988960
## tc_BEND4         -0.086472203  0.079460979
## tc_SLC9A3        -0.007464973  0.006859708
## 
## Layer 3
## 
##                mu_cluster1  mu_cluster2
## miR.101.3p    -0.006887544  3.007913039
## miR.125a.5p    0.187538247  2.785670575
## miR.125b.1.3p  0.027408679  2.968675795
## miR.127.3p    -0.107628701  0.123004230
## miR.140.5p     0.090862876 -0.103843286
## miR.142.3p    -0.004784527  0.005468031
## miR.144.5p     0.080101772 -0.091544883
## miR.19a.3p     0.060465036 -0.069102898
## miR.19b.3p     0.108156640 -0.123607589
## miR.21.5p      0.102286783 -0.116899180
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure for each layer 
## Layer 1
## 
##                             beta        OR
## (Intercept).cluster2 -0.23505568 0.7905268
## g_causal_1.cluster2   0.80835465 2.2442124
## g_causal_2.cluster2  -0.77185299 0.4621559
## g_causal_3.cluster2   0.05349673 1.0549535
## 
## Layer 2
## 
##                            beta        OR
## (Intercept).cluster2  0.8086360 2.2448439
## g_causal_1.cluster2  -0.9266784 0.3958665
## g_causal_2.cluster2   0.8938889 2.4446180
## g_causal_3.cluster2  -0.1997295 0.8189523
## 
## Layer 3
## 
##                            beta        OR
## (Intercept).cluster2 -1.3654847 0.2552569
## g_causal_1.cluster2   0.9934557 2.7005507
## g_causal_2.cluster2  -0.8493218 0.4277049
## g_causal_3.cluster2   0.1937766 1.2138251

8.2 Zero-penalty selected-only refit

set.seed(1206)

parallel_inputs_bin <- prepare_parallel_selected_inputs(parallel_pen_bin, G, Z_parallel_miss)

parallel_bin <- list(fit_pen = parallel_pen_bin, inputs = parallel_inputs_bin)
parallel_bin$fit_refit <- refit_selected(
  "parallel",
  fit_pen = parallel_pen_bin,
  inputs = parallel_inputs_bin,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  seed = 1206
)
## Fitting LUCID parallel model (3 layers)...
## Finished LUCID parallel model.
summary(parallel_bin$fit_refit)
## 
## ====================================================
## LUCID Parallel: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binomial
##   Number of observations : 90
##   Clusters per layer     : 2, 2, 2
## 
## Missing-data profile by layer
##   Layer 1 listwise rows : 1 / 90 (1.1%)
##   Layer 1 sporadic rows : 0 / 90 (0.0%)
##   Layer 1 missing cells : 10 / 900 (1.1%)
##   Layer 2 listwise rows : 0 / 90 (0.0%)
##   Layer 2 sporadic rows : 1 / 90 (1.1%)
##   Layer 2 missing cells : 1 / 900 (0.1%)
##   Layer 3 listwise rows : 0 / 90 (0.0%)
##   Layer 3 sporadic rows : 1 / 90 (1.1%)
##   Layer 3 missing cells : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 3 / 3 (100.0%)
##   G features by layer
##     Layer 1              : 3 / 3 (100.0%)
##     Layer 2              : 3 / 3 (100.0%)
##     Layer 3              : 3 / 3 (100.0%)
##   Z features
##     Layer 1 selected     : 10 / 10 (100.0%)
##     Layer 1 multi-cluster: 10
##     Layer 2 selected     : 10 / 10 (100.0%)
##     Layer 2 multi-cluster: 10
##     Layer 3 selected     : 10 / 10 (100.0%)
##     Layer 3 multi-cluster: 10
## 
## Model fit statistics
##   Log-likelihood         : -3330.30
##   BIC                    : 8523.51
##   Number of parameters   : 414
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): intercept and log OR for non-reference clusters for each layer (and log OR of covariate if included)
##                              gamma   exp(gamma)
## (Intercept)           -15.25206888 2.377446e-07
## Layer1_LC2              0.22785005 1.255897e+00
## Layer2_LC2             14.64253482 2.286505e+06
## Layer3_LC2             16.58300835 1.591875e+07
## hs_child_age_yrs_None  -0.03707901 9.636000e-01
## sex_male               -0.21751274 8.045174e-01
## 
## (2) Z: mean of omics data for each latent cluster of each layer 
## Layer 1
## 
##               mu_cluster1 mu_cluster2
## cg_GRHL3       0.01909800  2.78553563
## cg_BTF3L4      0.02480271  2.77019739
## cg_AL358472.7 -0.05985293  2.85041175
## cg_HDGF        0.19302043 -0.18887189
## cg_TDRD5      -0.13877077  0.13343290
## cg_CSRNP3     -0.07405841  0.08015087
## cg_HSPD1       0.18595283 -0.20524198
## cg_EPM2AIP1   -0.03855514  0.03877246
## cg_AC025171.1 -0.07290097  0.04247616
## cg_VTRNA1_3    0.15834764 -0.18440057
## 
## Layer 2
## 
##                   mu_cluster1  mu_cluster2
## tc_TC01006069_nc -2.760922332 -0.144990549
## tc_SLC9A4        -3.015209870  0.063240291
## tc_RAB6C_AS1     -2.732426038 -0.171269137
## tc_LOC100129029  -0.046840128  0.043194826
## tc_BRE           -0.072030770  0.066425023
## tc_TC03001220_nc  0.028025357 -0.025844302
## tc_TC04002114_nc -0.035638034  0.032864528
## tc_TC04002369_nc  0.134901102 -0.124402513
## tc_BEND4         -0.085317482  0.078677705
## tc_SLC9A3        -0.005786017  0.005335724
## 
## Layer 3
## 
##                mu_cluster1  mu_cluster2
## miR.101.3p    -0.006887544  3.007913039
## miR.125a.5p    0.187538247  2.785670575
## miR.125b.1.3p  0.027408679  2.968675795
## miR.127.3p    -0.107628701  0.123004230
## miR.140.5p     0.090862876 -0.103843286
## miR.142.3p    -0.004784527  0.005468031
## miR.144.5p     0.080101772 -0.091544883
## miR.19a.3p     0.060465036 -0.069102898
## miR.19b.3p     0.108156640 -0.123607589
## miR.21.5p      0.102286783 -0.116899180
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure for each layer 
## Layer 1
## 
##                            beta        OR
## (Intercept).cluster2 -1.1608772 0.3132113
## g_causal_1.cluster2   1.4623917 4.3162703
## g_causal_2.cluster2  -1.4797223 0.2277009
## g_causal_3.cluster2   0.4022659 1.4952088
## 
## Layer 2
## 
##                            beta        OR
## (Intercept).cluster2  2.1585328 8.6584247
## g_causal_1.cluster2  -1.8388979 0.1589926
## g_causal_2.cluster2   1.9681946 7.1577422
## g_causal_3.cluster2  -0.6638832 0.5148482
## 
## Layer 3
## 
##                            beta         OR
## (Intercept).cluster2 -3.2332090 0.03943076
## g_causal_1.cluster2   1.9557658 7.06933069
## g_causal_2.cluster2  -1.9426942 0.14331731
## g_causal_3.cluster2   0.6703573 1.95493576

8.3 Bootstrap CI + summary

set.seed(1207)

parallel_bin$boot <- boot_lucid(
  G = parallel_inputs_bin$G,
  Z = parallel_inputs_bin$Z,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  model = parallel_bin$fit_refit,
  R = 2,
  conf = 0.90
)

summary(parallel_bin$fit_refit, boot.se = parallel_bin$boot)
## 
## ====================================================
## LUCID Parallel: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binomial
##   Number of observations : 90
##   Clusters per layer     : 2, 2, 2
## 
## Missing-data profile by layer
##   Layer 1 listwise rows : 1 / 90 (1.1%)
##   Layer 1 sporadic rows : 0 / 90 (0.0%)
##   Layer 1 missing cells : 10 / 900 (1.1%)
##   Layer 2 listwise rows : 0 / 90 (0.0%)
##   Layer 2 sporadic rows : 1 / 90 (1.1%)
##   Layer 2 missing cells : 1 / 900 (0.1%)
##   Layer 3 listwise rows : 0 / 90 (0.0%)
##   Layer 3 sporadic rows : 1 / 90 (1.1%)
##   Layer 3 missing cells : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 3 / 3 (100.0%)
##   G features by layer
##     Layer 1              : 3 / 3 (100.0%)
##     Layer 2              : 3 / 3 (100.0%)
##     Layer 3              : 3 / 3 (100.0%)
##   Z features
##     Layer 1 selected     : 10 / 10 (100.0%)
##     Layer 1 multi-cluster: 10
##     Layer 2 selected     : 10 / 10 (100.0%)
##     Layer 2 multi-cluster: 10
##     Layer 3 selected     : 10 / 10 (100.0%)
##     Layer 3 multi-cluster: 10
## 
## Model fit statistics
##   Log-likelihood         : -3330.30
##   BIC                    : 8523.51
##   Number of parameters   : 414
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): intercept and log OR for non-reference clusters for each layer (and log OR of covariate if included)
##                              gamma  norm_lower norm_upper sig
## (Intercept)           -15.25206888 -43.5013144 -2.7184586   *
## Layer1_LC2              0.22785005  -0.9111796 -0.1120062   *
## Layer2_LC2             14.64253482   4.7087795 37.0098008   *
## Layer3_LC2             16.58300835   2.6436359 44.5867260   *
## hs_child_age_yrs_None  -0.03707901  -0.1637000  0.4450905    
## sex_male               -0.21751274  -0.1299947  0.2952572    
## 
## (2) Z: mean of omics data for each latent cluster of each layer 
## Layer 1
## 
##                                  estimate  norm_lower  norm_upper sig
## Layer1.cg_GRHL3.cluster1       0.01909800 -3.63761194 -0.01466919   *
## Layer1.cg_BTF3L4.cluster1      0.02480271 -3.47010608  0.19922044    
## Layer1.cg_AL358472.7.cluster1 -0.05985293 -4.09396125  0.55265989    
## Layer1.cg_HDGF.cluster1        0.19302043 -0.33678172  0.65885059    
## Layer1.cg_TDRD5.cluster1      -0.13877077 -0.45999113 -0.17759826   *
## Layer1.cg_CSRNP3.cluster1     -0.07405841 -0.15022524 -0.07645651   *
## Layer1.cg_HSPD1.cluster1       0.18595283  0.21520836  0.83044414   *
## Layer1.cg_EPM2AIP1.cluster1   -0.03855514 -0.51855787  0.18004922    
## Layer1.cg_AC025171.1.cluster1 -0.07290097 -0.37660277  0.33646852    
## Layer1.cg_VTRNA1_3.cluster1    0.15834764  0.05122069  0.75174884   *
## Layer1.cg_GRHL3.cluster2       2.78553563  1.86982812  7.15596671   *
## Layer1.cg_BTF3L4.cluster2      2.77019739  0.90486862  7.34841571   *
## Layer1.cg_AL358472.7.cluster2  2.85041175  0.45215503  8.25689706   *
## Layer1.cg_HDGF.cluster2       -0.18887189 -0.88786194  0.45945399    
## Layer1.cg_TDRD5.cluster2       0.13343290 -0.44585033  0.66594870    
## Layer1.cg_CSRNP3.cluster2      0.08015087 -0.06970684  0.20099536    
## Layer1.cg_HSPD1.cluster2      -0.20524198 -0.68657829 -0.32787890   *
## Layer1.cg_EPM2AIP1.cluster2    0.03877246  0.08402300  0.08958159   *
## Layer1.cg_AC025171.1.cluster2  0.04247616 -0.39222534  0.67990814    
## Layer1.cg_VTRNA1_3.cluster2   -0.18440057 -1.02780152  0.23153863    
## 
## Layer 2
## 
##                                      estimate  norm_lower  norm_upper sig
## Layer2.tc_TC01006069_nc.cluster1 -2.760922332 -6.09119747 -1.28330978   *
## Layer2.tc_SLC9A4.cluster1        -3.015209870 -6.14808947 -1.58410753   *
## Layer2.tc_RAB6C_AS1.cluster1     -2.732426038 -5.55494664 -1.31432649   *
## Layer2.tc_LOC100129029.cluster1  -0.046840128 -0.75497844  0.30679425    
## Layer2.tc_BRE.cluster1           -0.072030770 -0.50456828 -0.03057753   *
## Layer2.tc_TC03001220_nc.cluster1  0.028025357  0.02911272  0.18443042   *
## Layer2.tc_TC04002114_nc.cluster1 -0.035638034 -0.68810739  0.45827569    
## Layer2.tc_TC04002369_nc.cluster1  0.134901102 -0.26057566  0.89019098    
## Layer2.tc_BEND4.cluster1         -0.085317482 -0.18907583 -0.13619379   *
## Layer2.tc_SLC9A3.cluster1        -0.005786017 -0.11581659  0.27507811    
## Layer2.tc_TC01006069_nc.cluster2 -0.144990549 -1.02719420  2.79569293    
## Layer2.tc_SLC9A4.cluster2         0.063240291 -1.32796825  3.32092748    
## Layer2.tc_RAB6C_AS1.cluster2     -0.171269137 -1.09791435  2.14717831    
## Layer2.tc_LOC100129029.cluster2   0.043194826  0.15939529  0.29955526   *
## Layer2.tc_BRE.cluster2            0.066425023  0.12157620  0.47437878   *
## Layer2.tc_TC03001220_nc.cluster2 -0.025844302 -0.39399481  0.03923286    
## Layer2.tc_TC04002114_nc.cluster2  0.032864528  0.04836820  0.24202336   *
## Layer2.tc_TC04002369_nc.cluster2 -0.124402513 -0.58692385 -0.02185336   *
## Layer2.tc_BEND4.cluster2          0.078677705 -0.32359240  0.16474707    
## Layer2.tc_SLC9A3.cluster2         0.005335724  0.03466449  0.34674978   *
## 
## Layer 3
## 
##                                   estimate norm_lower norm_upper sig
## Layer3.miR.101.3p.cluster1    -0.006887544 -1.1672076 0.14692826    
## Layer3.miR.125a.5p.cluster1    0.187538247 -1.2613077 0.64533412    
## Layer3.miR.125b.1.3p.cluster1  0.027408679 -2.7272639 1.08219609    
## Layer3.miR.127.3p.cluster1    -0.107628701 -0.1238395 0.20139482    
## Layer3.miR.140.5p.cluster1     0.090862876 -0.4858587 1.60592077    
## Layer3.miR.142.3p.cluster1    -0.004784527 -0.9170645 1.68706957    
## Layer3.miR.144.5p.cluster1     0.080101772 -0.8861933 1.98596841    
## Layer3.miR.19a.3p.cluster1     0.060465036 -0.7908822 1.46249256    
## Layer3.miR.19b.3p.cluster1     0.108156640 -0.9511673 2.10653058    
## Layer3.miR.21.5p.cluster1      0.102286783 -0.8183850 2.06306041    
## Layer3.miR.101.3p.cluster2     3.007913039  2.6326486 4.35508183   *
## Layer3.miR.125a.5p.cluster2    2.785670575  1.8392145 4.48917937   *
## Layer3.miR.125b.1.3p.cluster2  2.968675795  2.1548826 5.15819371   *
## Layer3.miR.127.3p.cluster2     0.123004230 -0.6521996 0.33281074    
## Layer3.miR.140.5p.cluster2    -0.103843286 -0.8670132 0.16293285    
## Layer3.miR.142.3p.cluster2     0.005468031 -0.7670781 0.35147560    
## Layer3.miR.144.5p.cluster2    -0.091544883 -0.7232061 0.01013771    
## Layer3.miR.19a.3p.cluster2    -0.069102898 -0.6947970 0.06209677    
## Layer3.miR.19b.3p.cluster2    -0.123607589 -1.1174273 0.20278101    
## Layer3.miR.21.5p.cluster2     -0.116899180 -1.1476837 0.35100555    
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure for each layer 
## Layer 1
## 
##                        estimate norm_lower norm_upper sig
## (Intercept).cluster2 -1.1608772  -2.071876  1.3633131    
## g_causal_1.cluster2   1.4623917   2.084766  5.2416994   *
## g_causal_2.cluster2  -1.4797223  -5.253690 -0.4568312   *
## g_causal_3.cluster2   0.4022659   0.746633  1.2905459   *
## 
## Layer 2
## 
##                        estimate norm_lower norm_upper sig
## (Intercept).cluster2  2.1585328   4.232697  5.4398598   *
## g_causal_1.cluster2  -1.8388979  -6.514051  0.4553255    
## g_causal_2.cluster2   1.9681946   2.003519  4.2368986   *
## g_causal_3.cluster2  -0.6638832  -2.592880  0.6842472    
## 
## Layer 3
## 
##                        estimate  norm_lower norm_upper sig
## (Intercept).cluster2 -3.2332090 -10.7815322  -5.617127   *
## g_causal_1.cluster2   1.9557658  -0.5721802   6.430527    
## g_causal_2.cluster2  -1.9426942  -4.1060943  -1.442991   *
## g_causal_3.cluster2   0.6703573  -0.1136640   2.229143

One structural difference between the families is specific to the parallel model and easy to miss. For a normal outcome, the early model estimates a per-cluster residual standard deviation – one value per latent cluster – whereas the parallel model estimates a single pooled standard deviation across the joint cluster configuration. For a binary outcome neither exists. So if you are comparing dispersion across model types, compare like with like.

9) Serial Model Tutorial A (All-Early Stages): Binary Outcome

9.1 Penalized screening fit

# Serial structure: list of early-stage matrices.
Z_serial_all_early <- list(
  methylome = Z_parallel_miss[[1]],
  transcriptome = Z_parallel_miss[[2]],
  miRNA = Z_parallel_miss[[3]]
)

set.seed(1305)

serial_ae_pen_bin <- estimate_lucid(
  lucid_model = "serial",
  G = G,
  Z = Z_serial_all_early,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  family = "binary",
  K = list(2, 2, 2),
  Rho_G = RHO_G,
  Rho_Z_Mu = 0,
  Rho_Z_Cov = 0,
  max_itr = 15,
  max_tot.itr = 40,
  tol = 1e-2,
  seed = 1305,
  verbose = FALSE
)
## Fitting LUCID serial model (3 stages)...
##   Stage 1/3 (early) finished: log-likelihood = -1202.568. Selected G: 3/9; Selected Z: 10/10.
##   Stage 2/3 (early) finished: log-likelihood = -1217.744.
##   Stage 3/3 (early) finished: log-likelihood = -865.953.
## Finished LUCID serial model.
summary(serial_ae_pen_bin)
## 
## ====================================================
## LUCID Serial: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of stages       : 3
##   Stage 1               : early (K = 2)
##   Stage 2               : early (K = 2)
##   Stage 3               : early (K = 2)
## 
## Missing-data profile by stage
##   Stage 1
##     Listwise rows         : 1
##     Sporadic rows         : 0
##   Stage 2
##     Listwise rows         : 0
##     Sporadic rows         : 1
##   Stage 3
##     Listwise rows         : 0
##     Sporadic rows         : 1
## 
## Model fit statistics
##   Log-likelihood         : -3286.26
##   BIC                    : 8390.45
## 
## Regularization
##   Rho_G                  : 0.050
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Stage-wise detailed parameter estimates
## 
## --- Stage 1 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : normal
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 1 / 90 (1.1%)
##   Sporadic missing rows  : 0 / 90 (0.0%)
##   Missing cells total    : 10 / 900 (1.1%)
## 
## Feature selection overview
##   G features selected    : 3 / 9 (33.3%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -1202.57
##   BIC                    : 3017.11
##   Number of parameters   : 136
## 
## Regularization
##   Rho_G                  : 0.050
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Z: mean of omics data for each latent cluster 
##               mu_cluster1 mu_cluster2
## cg_GRHL3       0.01961961  2.78734239
## cg_BTF3L4      0.02563074  2.77167224
## cg_AL358472.7 -0.05842807  2.85141440
## cg_HDGF        0.19392675 -0.19012409
## cg_TDRD5      -0.13906521  0.13396510
## cg_CSRNP3     -0.07423022  0.08045749
## cg_HSPD1       0.18520055 -0.20480198
## cg_EPM2AIP1   -0.03954436  0.03985187
## cg_AC025171.1 -0.07348909  0.04317664
## cg_VTRNA1_3    0.15848766 -0.18483418
## 
## (2) E: intercept and odds ratio of being assigned to each latent cluster for each exposure 
##                                        beta        OR
## (Intercept).cluster2           -0.233840269 0.7914882
## g_causal_1.cluster2             0.807430367 2.2421391
## g_causal_2.cluster2            -0.771389025 0.4623704
## g_causal_3.cluster2             0.053240477 1.0546832
## hs_child_age_yrs_None.cluster2 -0.004675195 0.9953357
## sex_male.cluster2               0.628334926 1.8744868
## 
## --- Stage 2 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : normal
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 0 / 90 (0.0%)
##   Sporadic missing rows  : 1 / 90 (1.1%)
##   Missing cells total    : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 1 / 1 (100.0%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -1217.74
##   BIC                    : 3029.46
##   Number of parameters   : 132
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Z: mean of omics data for each latent cluster 
##                   mu_cluster1   mu_cluster2
## tc_TC01006069_nc -2.770982314 -0.2004046469
## tc_SLC9A4        -3.075746204  0.0375636749
## tc_RAB6C_AS1     -2.807651591 -0.1683194085
## tc_LOC100129029  -0.052357560  0.0458123240
## tc_BRE           -0.112174125  0.0981512000
## tc_TC03001220_nc  0.060238428 -0.0527080015
## tc_TC04002114_nc -0.036339642  0.0317968112
## tc_TC04002369_nc  0.078990827 -0.0691161574
## tc_BEND4         -0.063253000  0.0553457209
## tc_SLC9A3        -0.001011716  0.0008852407
## 
## (2) E: intercept and odds ratio of being assigned to each latent cluster for each cluster from previous serial stage
##                                beta           OR
## (Intercept).cluster2       7.781379 2.395576e+03
## Stage1.cluster2.cluster2 -10.462010 2.860268e-05
## 
## --- Stage 3 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 0 / 90 (0.0%)
##   Sporadic missing rows  : 1 / 90 (1.1%)
##   Missing cells total    : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 1 / 1 (100.0%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -865.95
##   BIC                    : 2343.88
##   Number of parameters   : 136
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): log odds of Y for cluster 1 (reference) and log OR for rest cluster (and log OR of covariate if included)
##                             gamma exp(gamma)
## (Intercept)           -0.73799854  0.4780698
## LC2                    2.18407289  8.8824098
## hs_child_age_yrs_None -0.02423243  0.9760588
## sex_male              -0.18297636  0.8327878
## 
## (2) Z: mean of omics data for each latent cluster 
##                mu_cluster1  mu_cluster2
## miR.101.3p    -0.006887544  3.007913039
## miR.125a.5p    0.187538247  2.785670575
## miR.125b.1.3p  0.027408679  2.968675795
## miR.127.3p    -0.107628701  0.123004230
## miR.140.5p     0.090862876 -0.103843286
## miR.142.3p    -0.004784527  0.005468031
## miR.144.5p     0.080101772 -0.091544883
## miR.19a.3p     0.060465036 -0.069102898
## miR.19b.3p     0.108156640 -0.123607589
## miR.21.5p      0.102286783 -0.116899180
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each cluster from previous serial stage
##                               beta           OR
## (Intercept).cluster2      20.28449 6.448268e+08
## Stage2.cluster2.cluster2 -33.53361 2.732364e-15

9.2 Zero-penalty selected-input refit

set.seed(1306)

serial_ae_inputs_bin <- prepare_serial_selected_inputs(serial_ae_pen_bin, G, Z_serial_all_early)

serial_ae_bin <- list(fit_pen = serial_ae_pen_bin, inputs = serial_ae_inputs_bin)
serial_ae_bin$fit_refit <- refit_selected(
  "serial",
  fit_pen = serial_ae_pen_bin,
  inputs = serial_ae_inputs_bin,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  seed = 1306
)
## Fitting LUCID serial model (3 stages)...
##   Stage 1/3 (early) finished: log-likelihood = -1198.662.
##   Stage 2/3 (early) finished: log-likelihood = -1217.524.
##   Stage 3/3 (early) finished: log-likelihood = -865.953.
## Finished LUCID serial model.
summary(serial_ae_bin$fit_refit)
## 
## ====================================================
## LUCID Serial: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of stages       : 3
##   Stage 1               : early (K = 2)
##   Stage 2               : early (K = 2)
##   Stage 3               : early (K = 2)
## 
## Missing-data profile by stage
##   Stage 1
##     Listwise rows         : 1
##     Sporadic rows         : 0
##   Stage 2
##     Listwise rows         : 0
##     Sporadic rows         : 1
##   Stage 3
##     Listwise rows         : 0
##     Sporadic rows         : 1
## 
## Model fit statistics
##   Log-likelihood         : -3282.14
##   BIC                    : 8382.20
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Stage-wise detailed parameter estimates
## 
## --- Stage 1 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : normal
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 1 / 90 (1.1%)
##   Sporadic missing rows  : 0 / 90 (0.0%)
##   Missing cells total    : 10 / 900 (1.1%)
## 
## Feature selection overview
##   G features selected    : 3 / 3 (100.0%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -1198.66
##   BIC                    : 3009.30
##   Number of parameters   : 136
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Z: mean of omics data for each latent cluster 
##               mu_cluster1 mu_cluster2
## cg_GRHL3       0.01915163  2.78569034
## cg_BTF3L4      0.02487628  2.77033010
## cg_AL358472.7 -0.05975885  2.85053597
## cg_HDGF        0.19307580 -0.18895749
## cg_TDRD5      -0.13876555  0.13344819
## cg_CSRNP3     -0.07408559  0.08019037
## cg_HSPD1       0.18585702 -0.20517357
## cg_EPM2AIP1   -0.03863075  0.03885569
## cg_AC025171.1 -0.07293798  0.04252278
## cg_VTRNA1_3    0.15834584 -0.18442470
## 
## (2) E: intercept and odds ratio of being assigned to each latent cluster for each exposure 
##                                      beta        OR
## (Intercept).cluster2           -1.1615344 0.3130055
## g_causal_1.cluster2             1.4615136 4.3124821
## g_causal_2.cluster2            -1.4789346 0.2278803
## g_causal_3.cluster2             0.4021607 1.4950515
## hs_child_age_yrs_None.cluster2  0.1162000 1.1232205
## sex_male.cluster2               1.0246400 2.7860923
## 
## --- Stage 2 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : normal
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 0 / 90 (0.0%)
##   Sporadic missing rows  : 1 / 90 (1.1%)
##   Missing cells total    : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 1 / 1 (100.0%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -1217.52
##   BIC                    : 3029.02
##   Number of parameters   : 132
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Z: mean of omics data for each latent cluster 
##                   mu_cluster1   mu_cluster2
## tc_TC01006069_nc -2.770982428 -0.2004050510
## tc_SLC9A4        -3.075854117  0.0375632833
## tc_RAB6C_AS1     -2.807651890 -0.1683196643
## tc_LOC100129029  -0.052357769  0.0458124875
## tc_BRE           -0.112174379  0.0981513805
## tc_TC03001220_nc  0.060238499 -0.0527080414
## tc_TC04002114_nc -0.036339702  0.0317968501
## tc_TC04002369_nc  0.078990703 -0.0691160197
## tc_BEND4         -0.063252955  0.0553456585
## tc_SLC9A3        -0.001011683  0.0008852121
## 
## (2) E: intercept and odds ratio of being assigned to each latent cluster for each cluster from previous serial stage
##                               beta           OR
## (Intercept).cluster2      35.88858 3.856685e+15
## Stage1.cluster2.cluster2 -38.54939 1.812230e-17
## 
## --- Stage 3 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 0 / 90 (0.0%)
##   Sporadic missing rows  : 1 / 90 (1.1%)
##   Missing cells total    : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 1 / 1 (100.0%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -865.95
##   BIC                    : 2343.88
##   Number of parameters   : 136
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): log odds of Y for cluster 1 (reference) and log OR for rest cluster (and log OR of covariate if included)
##                             gamma exp(gamma)
## (Intercept)           -0.73799854  0.4780698
## LC2                    2.18407289  8.8824098
## hs_child_age_yrs_None -0.02423243  0.9760588
## sex_male              -0.18297636  0.8327878
## 
## (2) Z: mean of omics data for each latent cluster 
##                mu_cluster1  mu_cluster2
## miR.101.3p    -0.006887544  3.007913039
## miR.125a.5p    0.187538247  2.785670575
## miR.125b.1.3p  0.027408679  2.968675795
## miR.127.3p    -0.107628701  0.123004230
## miR.140.5p     0.090862876 -0.103843286
## miR.142.3p    -0.004784527  0.005468031
## miR.144.5p     0.080101772 -0.091544883
## miR.19a.3p     0.060465036 -0.069102898
## miR.19b.3p     0.108156640 -0.123607589
## miR.21.5p      0.102286783 -0.116899180
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each cluster from previous serial stage
##                               beta           OR
## (Intercept).cluster2      20.28449 6.448276e+08
## Stage2.cluster2.cluster2 -33.53361 2.732356e-15

9.3 Bootstrap CI + summary

set.seed(1307)

serial_ae_bin$boot <- boot_lucid(
  G = serial_ae_inputs_bin$G,
  Z = serial_ae_inputs_bin$Z,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  model = serial_ae_bin$fit_refit,
  R = 2,
  conf = 0.90
)

summary(serial_ae_bin$fit_refit, boot.se = serial_ae_bin$boot)
## 
## ====================================================
## LUCID Serial: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of stages       : 3
##   Stage 1               : early (K = 2)
##   Stage 2               : early (K = 2)
##   Stage 3               : early (K = 2)
## 
## Missing-data profile by stage
##   Stage 1
##     Listwise rows         : 1
##     Sporadic rows         : 0
##   Stage 2
##     Listwise rows         : 0
##     Sporadic rows         : 1
##   Stage 3
##     Listwise rows         : 0
##     Sporadic rows         : 1
## 
## Model fit statistics
##   Log-likelihood         : -3282.14
##   BIC                    : 8382.20
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Stage-wise detailed parameter estimates
## 
## --- Stage 1 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : normal
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 1 / 90 (1.1%)
##   Sporadic missing rows  : 0 / 90 (0.0%)
##   Missing cells total    : 10 / 900 (1.1%)
## 
## Feature selection overview
##   G features selected    : 3 / 3 (100.0%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -1198.66
##   BIC                    : 3009.30
##   Number of parameters   : 136
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Z: mean of omics data for each latent cluster 
##                           estimate  norm_lower  norm_upper sig
## cg_GRHL3.cluster1       0.01917728 -0.96654644  0.50139420    
## cg_BTF3L4.cluster1      0.02491138 -0.95657970  0.51892493    
## cg_AL358472.7.cluster1 -0.05970852 -0.37192308  0.25697318    
## cg_HDGF.cluster1        0.19310271  0.21952877  0.35817331   *
## cg_TDRD5.cluster1      -0.13876194 -0.60562596  0.26519851    
## cg_CSRNP3.cluster1     -0.07409829 -0.41327861  0.17269079    
## cg_HSPD1.cluster1       0.18582648 -0.07993323  0.52249752    
## cg_EPM2AIP1.cluster1   -0.03866306 -0.36887099  0.11433660    
## cg_AC025171.1.cluster1 -0.07295478 -0.27580899  0.07076267    
## cg_VTRNA1_3.cluster1    0.15834595  0.07400669  0.46978305   *
## cg_GRHL3.cluster2       2.78575022  2.69010441  3.02228016   *
## cg_BTF3L4.cluster2      2.77037965  2.70409714  3.08696990   *
## cg_AL358472.7.cluster2  2.85057507  1.86927605  3.65281300   *
## cg_HDGF.cluster2       -0.18899692 -0.60739930  0.47650620    
## cg_TDRD5.cluster2       0.13345297 -0.21419581  1.01437460    
## cg_CSRNP3.cluster2      0.08020818 -0.74025766  0.60724877    
## cg_HSPD1.cluster2      -0.20515448 -0.41595584 -0.13476374   *
## cg_EPM2AIP1.cluster2    0.03889117 -0.43194128  0.76482567    
## cg_AC025171.1.cluster2  0.04254357 -0.61832827  0.36165471    
## cg_VTRNA1_3.cluster2   -0.18443547 -0.53504894  0.28329170    
## 
## (2) E: intercept and odds ratio of being assigned to each latent cluster for each exposure 
##                                  estimate  norm_lower norm_upper sig
## (Intercept).cluster2           -1.1625544 -10.0340911  0.6650835    
## g_causal_1.cluster2             1.4617398   1.9813890  2.3535550   *
## g_causal_2.cluster2            -1.4783342  -2.0696527 -0.2703055   *
## g_causal_3.cluster2             0.4023752   0.5064620  0.5597854   *
## hs_child_age_yrs_None.cluster2  0.1163518   0.1168535  1.2250367   *
## sex_male.cluster2               1.0242235   0.1431386  2.5826317   *
## 
## --- Stage 2 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : normal
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 0 / 90 (0.0%)
##   Sporadic missing rows  : 1 / 90 (1.1%)
##   Missing cells total    : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 1 / 1 (100.0%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -1217.52
##   BIC                    : 3029.02
##   Number of parameters   : 132
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Z: mean of omics data for each latent cluster 
##                                estimate  norm_lower   norm_upper sig
## tc_TC01006069_nc.cluster1 -2.7709824245 -3.24358476 -2.341476416   *
## tc_SLC9A4.cluster1        -3.0758530162 -3.90288594 -2.337454095   *
## tc_RAB6C_AS1.cluster1     -2.8076518845 -3.37213733 -2.409452470   *
## tc_LOC100129029.cluster1  -0.0523577630 -0.09636837  0.203224575    
## tc_BRE.cluster1           -0.1121743733 -0.98863571  0.048147427    
## tc_TC03001220_nc.cluster1  0.0602384985  0.04594864  0.334863496   *
## tc_TC04002114_nc.cluster1 -0.0363396983 -0.12764037  0.266652191    
## tc_TC04002369_nc.cluster1  0.0789907038 -0.29561604  0.616434019    
## tc_BEND4.cluster1         -0.0632529557  0.17253274  0.264150246   *
## tc_SLC9A3.cluster1        -0.0010116849 -0.22522138  0.389240534    
## tc_TC01006069_nc.cluster2 -0.2004050421 -0.35580846  0.069519677    
## tc_SLC9A4.cluster2         0.0375632946 -0.30485248  0.812729289    
## tc_RAB6C_AS1.cluster2     -0.1683196568 -0.37278903  0.124363661    
## tc_LOC100129029.cluster2   0.0458124826 -0.25111611  0.632997242    
## tc_BRE.cluster2            0.0981513768  0.06163978  0.361769291   *
## tc_TC03001220_nc.cluster2 -0.0527080419 -0.04192909  0.214395752    
## tc_TC04002114_nc.cluster2  0.0317968473 -0.19775684  0.210824673    
## tc_TC04002369_nc.cluster2 -0.0691160210 -0.17147289  0.029720774    
## tc_BEND4.cluster2          0.0553456597 -0.14056184 -0.003466741   *
## tc_SLC9A3.cluster2         0.0008852135 -0.14722423 -0.121734913   *
## 
## (2) E: intercept and odds ratio of being assigned to each latent cluster for each cluster from previous serial stage
##                           estimate norm_lower norm_upper sig
## (Intercept).cluster2      36.77489   70.38279   71.95230   *
## Stage1.cluster2.cluster2 -39.43640  -77.13531  -71.08961   *
## 
## --- Stage 3 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 0 / 90 (0.0%)
##   Sporadic missing rows  : 1 / 90 (1.1%)
##   Missing cells total    : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 1 / 1 (100.0%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -865.95
##   BIC                    : 2343.88
##   Number of parameters   : 136
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): log odds of Y for cluster 1 (reference) and log OR for rest cluster (and log OR of covariate if included)
##                             gamma norm_lower norm_upper sig
## (Intercept)           -0.73799854 -3.9300110  1.3890763    
## LC2                    2.18407289  0.8406138  4.7103806   *
## hs_child_age_yrs_None -0.02423243 -0.1238731  0.2549249    
## sex_male              -0.18297636 -0.9417870  0.9307660    
## 
## (2) Z: mean of omics data for each latent cluster 
##                            estimate  norm_lower  norm_upper sig
## miR.101.3p.cluster1    -0.006887544 -0.06834413  0.65870909    
## miR.125a.5p.cluster1    0.187538247 -0.25814193  0.65633205    
## miR.125b.1.3p.cluster1  0.027408679 -0.56289915  0.40865692    
## miR.127.3p.cluster1    -0.107628701 -0.12945705  0.07013801    
## miR.140.5p.cluster1     0.090862876  0.15257055  1.40859256   *
## miR.142.3p.cluster1    -0.004784527 -0.25718268  1.31770791    
## miR.144.5p.cluster1     0.080101772 -0.40980312  1.15135290    
## miR.19a.3p.cluster1     0.060465036 -0.01999336  1.14441789    
## miR.19b.3p.cluster1     0.108156640 -0.07744843  1.40507741    
## miR.21.5p.cluster1      0.102286783 -0.20778220  1.55714803    
## miR.101.3p.cluster2     3.007913039  2.97696361  3.91470334   *
## miR.125a.5p.cluster2    2.785670575  2.82303715  3.93810877   *
## miR.125b.1.3p.cluster2  2.968675795  2.90390997  4.65310599   *
## miR.127.3p.cluster2     0.123004230 -0.32697650  0.19925203    
## miR.140.5p.cluster2    -0.103843286 -0.83996106 -0.41887735   *
## miR.142.3p.cluster2     0.005468031 -0.48369531 -0.29304308   *
## miR.144.5p.cluster2    -0.091544883 -0.74767828 -0.25134839   *
## miR.19a.3p.cluster2    -0.069102898 -0.71571927 -0.08661989   *
## miR.19b.3p.cluster2    -0.123607589 -0.81449511 -0.24474757   *
## miR.21.5p.cluster2     -0.116899180 -0.59530853 -0.47712506   *
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each cluster from previous serial stage
##                           estimate norm_lower norm_upper sig
## (Intercept).cluster2      20.28449   35.79814   40.21430   *
## Stage2.cluster2.cluster2 -33.53361  -66.95671  -59.51429   *

For a serial model the outcome family applies to the final stage only. Upstream stages are fitted unsupervised regardless of what you pass, because the outcome enters the chain once, at the end – so the binary/normal distinction shows up in the last stage’s report and nowhere else.

10) Serial Model Tutorial B (Mixed Parallel + Early): Binary Outcome

10.1 Penalized screening fit

# Nested list signals a parallel submodel at stage 1, followed by early stage 2.
Z_serial_mixed <- list(
  list(
    methylome = Z_parallel_miss[[1]],
    transcriptome = Z_parallel_miss[[2]]
  ),
  miRNA = Z_parallel_miss[[3]]
)

set.seed(1405)

serial_mixed_pen_bin <- estimate_lucid(
  lucid_model = "serial",
  G = G,
  Z = Z_serial_mixed,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  family = "binary",
  K = list(list(2, 2), 2),
  Rho_G = RHO_G,
  Rho_Z_Mu = 0,
  Rho_Z_Cov = 0,
  max_itr = 15,
  max_tot.itr = 40,
  tol = 1e-2,
  seed = 1405,
  verbose = FALSE
)
## Fitting LUCID serial model (2 stages)...
##   Stage 1/2 (parallel) finished: log-likelihood = -2444.642. Selected G: 3/9; Selected Z by layer: 10/10, 10/10.
##   Stage 2/2 (early) finished: log-likelihood = -865.953.
## Finished LUCID serial model.
summary(serial_mixed_pen_bin)
## 
## ====================================================
## LUCID Serial: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of stages       : 2
##   Stage 1               : parallel (K = 2,2)
##   Stage 2               : early (K = 2)
## 
## Missing-data profile by stage
##   Stage 1
##     Layer 1 listwise/sporadic rows : 1 / 0
##     Layer 2 listwise/sporadic rows : 0 / 1
##   Stage 2
##     Listwise rows         : 0
##     Sporadic rows         : 1
## 
## Model fit statistics
##   Log-likelihood         : -3310.59
##   BIC                    : 8461.61
## 
## Regularization
##   Rho_G                  : 0.050
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Stage-wise detailed parameter estimates
## 
## --- Stage 1 (parallel) ---
## 
## ====================================================
## LUCID Parallel: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : gaussian
##   Number of observations : 90
##   Clusters per layer     : 2, 2
## 
## Missing-data profile by layer
##   Layer 1 listwise rows : 1 / 90 (1.1%)
##   Layer 1 sporadic rows : 0 / 90 (0.0%)
##   Layer 1 missing cells : 10 / 900 (1.1%)
##   Layer 2 listwise rows : 0 / 90 (0.0%)
##   Layer 2 sporadic rows : 1 / 90 (1.1%)
##   Layer 2 missing cells : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 3 / 9 (33.3%)
##   G features by layer
##     Layer 1              : 3 / 9 (33.3%)
##     Layer 2              : 3 / 9 (33.3%)
##   Z features
##     Layer 1 selected     : 10 / 10 (100.0%)
##     Layer 1 multi-cluster: 10
##     Layer 2 selected     : 10 / 10 (100.0%)
##     Layer 2 multi-cluster: 10
## 
## Model fit statistics
##   Log-likelihood         : -2444.64
##   BIC                    : 6113.23
##   Number of parameters   : 272
## 
## Regularization
##   Rho_G                  : 0.050
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Z: mean of omics data for each latent cluster of each layer 
## Layer 1
## 
##               mu_cluster1 mu_cluster2
## cg_GRHL3       0.01960667  2.78730293
## cg_BTF3L4      0.02561217  2.77163898
## cg_AL358472.7 -0.05845803  2.85138969
## cg_HDGF        0.19390770 -0.19009725
## cg_TDRD5      -0.13906115  0.13395573
## cg_CSRNP3     -0.07422600  0.08045023
## cg_HSPD1       0.18521867 -0.20481312
## cg_EPM2AIP1   -0.03952313  0.03982859
## cg_AC025171.1 -0.07347676  0.04316178
## cg_VTRNA1_3    0.15848563 -0.18482556
## 
## Layer 2
## 
##                  mu_cluster1  mu_cluster2
## tc_TC01006069_nc -2.76171772 -0.151431669
## tc_SLC9A4        -3.02348512  0.061983212
## tc_RAB6C_AS1     -2.73581445 -0.175182553
## tc_LOC100129029  -0.04727047  0.043342620
## tc_BRE           -0.07428108  0.068108837
## tc_TC03001220_nc  0.02974284 -0.027271414
## tc_TC04002114_nc -0.03156595  0.028943037
## tc_TC04002369_nc  0.13303025 -0.121976355
## tc_BEND4         -0.08701134  0.079781296
## tc_SLC9A3        -0.00834082  0.007647755
## 
## (2) E: intercept and odds ratio of being assigned to each latent cluster for each exposure for each layer 
## Layer 1
## 
##                             beta        OR
## (Intercept).cluster2 -0.23378217 0.7915342
## g_causal_1.cluster2   0.80753183 2.2423666
## g_causal_2.cluster2  -0.77145952 0.4623378
## g_causal_3.cluster2   0.05322757 1.0546696
## 
## Layer 2
## 
##                            beta        OR
## (Intercept).cluster2  0.8377907 2.3112551
## g_causal_1.cluster2  -0.9250451 0.3965136
## g_causal_2.cluster2   0.8918749 2.4396996
## g_causal_3.cluster2  -0.2028611 0.8163916
## 
## 
## --- Stage 2 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 0 / 90 (0.0%)
##   Sporadic missing rows  : 1 / 90 (1.1%)
##   Missing cells total    : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 2 / 2 (100.0%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -865.95
##   BIC                    : 2348.38
##   Number of parameters   : 137
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): log odds of Y for cluster 1 (reference) and log OR for rest cluster (and log OR of covariate if included)
##                             gamma exp(gamma)
## (Intercept)           -0.73799854  0.4780698
## LC2                    2.18407289  8.8824098
## hs_child_age_yrs_None -0.02423243  0.9760588
## sex_male              -0.18297636  0.8327878
## 
## (2) Z: mean of omics data for each latent cluster 
##                mu_cluster1  mu_cluster2
## miR.101.3p    -0.006887544  3.007913039
## miR.125a.5p    0.187538247  2.785670575
## miR.125b.1.3p  0.027408679  2.968675795
## miR.127.3p    -0.107628701  0.123004230
## miR.140.5p     0.090862876 -0.103843286
## miR.142.3p    -0.004784527  0.005468031
## miR.144.5p     0.080101772 -0.091544883
## miR.19a.3p     0.060465036 -0.069102898
## miR.19b.3p     0.108156640 -0.123607589
## miR.21.5p      0.102286783 -0.116899180
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each cluster from previous serial stage
##                                      beta           OR
## (Intercept).cluster2             -9.92645 4.886498e-05
## Stage1.Layer1.cluster2.cluster2  24.14059 3.048780e+10
## Stage1.Layer2.cluster2.cluster2 -26.02694 4.973275e-12

10.2 Zero-penalty selected-input refit

set.seed(1406)

serial_mixed_inputs_bin <- prepare_serial_selected_inputs(serial_mixed_pen_bin, G, Z_serial_mixed)

serial_mixed_bin <- list(fit_pen = serial_mixed_pen_bin, inputs = serial_mixed_inputs_bin)
serial_mixed_bin$fit_refit <- refit_selected(
  "serial",
  fit_pen = serial_mixed_pen_bin,
  inputs = serial_mixed_inputs_bin,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  seed = 1406
)
## Fitting LUCID serial model (2 stages)...
##   Stage 1/2 (parallel) finished: log-likelihood = -2435.579.
##   Stage 2/2 (early) finished: log-likelihood = -865.953.
## Finished LUCID serial model.
summary(serial_mixed_bin$fit_refit)
## 
## ====================================================
## LUCID Serial: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of stages       : 2
##   Stage 1               : parallel (K = 2,2)
##   Stage 2               : early (K = 2)
## 
## Missing-data profile by stage
##   Stage 1
##     Layer 1 listwise/sporadic rows : 1 / 0
##     Layer 2 listwise/sporadic rows : 0 / 1
##   Stage 2
##     Listwise rows         : 0
##     Sporadic rows         : 1
## 
## Model fit statistics
##   Log-likelihood         : -3301.53
##   BIC                    : 8443.49
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Stage-wise detailed parameter estimates
## 
## --- Stage 1 (parallel) ---
## 
## ====================================================
## LUCID Parallel: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : gaussian
##   Number of observations : 90
##   Clusters per layer     : 2, 2
## 
## Missing-data profile by layer
##   Layer 1 listwise rows : 1 / 90 (1.1%)
##   Layer 1 sporadic rows : 0 / 90 (0.0%)
##   Layer 1 missing cells : 10 / 900 (1.1%)
##   Layer 2 listwise rows : 0 / 90 (0.0%)
##   Layer 2 sporadic rows : 1 / 90 (1.1%)
##   Layer 2 missing cells : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 3 / 3 (100.0%)
##   G features by layer
##     Layer 1              : 3 / 3 (100.0%)
##     Layer 2              : 3 / 3 (100.0%)
##   Z features
##     Layer 1 selected     : 10 / 10 (100.0%)
##     Layer 1 multi-cluster: 10
##     Layer 2 selected     : 10 / 10 (100.0%)
##     Layer 2 multi-cluster: 10
## 
## Model fit statistics
##   Log-likelihood         : -2435.58
##   BIC                    : 6095.11
##   Number of parameters   : 272
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Z: mean of omics data for each latent cluster of each layer 
## Layer 1
## 
##               mu_cluster1 mu_cluster2
## cg_GRHL3       0.01912123  2.78560954
## cg_BTF3L4      0.02483414  2.77026216
## cg_AL358472.7 -0.05982001  2.85048084
## cg_HDGF        0.19304072 -0.18890613
## cg_TDRD5      -0.13876550  0.13343713
## cg_CSRNP3     -0.07407272  0.08017097
## cg_HSPD1       0.18589857 -0.20520028
## cg_EPM2AIP1   -0.03858839  0.03880921
## cg_AC025171.1 -0.07291502  0.04249461
## cg_VTRNA1_3    0.15834446 -0.18440941
## 
## Layer 2
## 
##                   mu_cluster1  mu_cluster2
## tc_TC01006069_nc -2.761572789 -0.150145472
## tc_SLC9A4        -3.021703480  0.062198525
## tc_RAB6C_AS1     -2.735259250 -0.174299961
## tc_LOC100129029  -0.047193030  0.043320800
## tc_BRE           -0.073892278  0.067829351
## tc_TC03001220_nc  0.029444093 -0.027028179
## tc_TC04002114_nc -0.032501823  0.029835020
## tc_TC04002369_nc  0.133308600 -0.122370510
## tc_BEND4         -0.086573422  0.079469996
## tc_SLC9A3        -0.007742854  0.007107546
## 
## (2) E: intercept and odds ratio of being assigned to each latent cluster for each exposure for each layer 
## Layer 1
## 
##                            beta        OR
## (Intercept).cluster2 -1.1612442 0.3130964
## g_causal_1.cluster2   1.4619640 4.3144246
## g_causal_2.cluster2  -1.4793132 0.2277941
## g_causal_3.cluster2   0.4022122 1.4951286
## 
## Layer 2
## 
##                            beta        OR
## (Intercept).cluster2  2.2583148 9.5669530
## g_causal_1.cluster2  -1.8289379 0.1605840
## g_causal_2.cluster2   1.9595122 7.0958651
## g_causal_3.cluster2  -0.6722936 0.5105363
## 
## 
## --- Stage 2 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 0 / 90 (0.0%)
##   Sporadic missing rows  : 1 / 90 (1.1%)
##   Missing cells total    : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 2 / 2 (100.0%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -865.95
##   BIC                    : 2348.38
##   Number of parameters   : 137
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): log odds of Y for cluster 1 (reference) and log OR for rest cluster (and log OR of covariate if included)
##                             gamma exp(gamma)
## (Intercept)           -0.73799854  0.4780698
## LC2                    2.18407289  8.8824098
## hs_child_age_yrs_None -0.02423243  0.9760588
## sex_male              -0.18297636  0.8327878
## 
## (2) Z: mean of omics data for each latent cluster 
##                mu_cluster1  mu_cluster2
## miR.101.3p    -0.006887544  3.007913039
## miR.125a.5p    0.187538247  2.785670575
## miR.125b.1.3p  0.027408679  2.968675795
## miR.127.3p    -0.107628701  0.123004230
## miR.140.5p     0.090862876 -0.103843286
## miR.142.3p    -0.004784527  0.005468031
## miR.144.5p     0.080101772 -0.091544883
## miR.19a.3p     0.060465036 -0.069102898
## miR.19b.3p     0.108156640 -0.123607589
## miR.21.5p      0.102286783 -0.116899180
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each cluster from previous serial stage
##                                      beta           OR
## (Intercept).cluster2            -10.07034 4.231622e-05
## Stage1.Layer1.cluster2.cluster2  23.72705 2.016173e+10
## Stage1.Layer2.cluster2.cluster2 -25.85493 5.906727e-12

10.3 Bootstrap CI + summary

set.seed(1407)

serial_mixed_bin$boot <- boot_lucid(
  G = serial_mixed_inputs_bin$G,
  Z = serial_mixed_inputs_bin$Z,
  Y = Y_binary,
  CoG = CoG,
  CoY = CoY,
  model = serial_mixed_bin$fit_refit,
  R = 2,
  conf = 0.90
)

summary(serial_mixed_bin$fit_refit, boot.se = serial_mixed_bin$boot)
## 
## ====================================================
## LUCID Serial: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of stages       : 2
##   Stage 1               : parallel (K = 2,2)
##   Stage 2               : early (K = 2)
## 
## Missing-data profile by stage
##   Stage 1
##     Layer 1 listwise/sporadic rows : 1 / 0
##     Layer 2 listwise/sporadic rows : 0 / 1
##   Stage 2
##     Listwise rows         : 0
##     Sporadic rows         : 1
## 
## Model fit statistics
##   Log-likelihood         : -3301.53
##   BIC                    : 8443.49
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Stage-wise detailed parameter estimates
## 
## --- Stage 1 (parallel) ---
## 
## ====================================================
## LUCID Parallel: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : gaussian
##   Number of observations : 90
##   Clusters per layer     : 2, 2
## 
## Missing-data profile by layer
##   Layer 1 listwise rows : 1 / 90 (1.1%)
##   Layer 1 sporadic rows : 0 / 90 (0.0%)
##   Layer 1 missing cells : 10 / 900 (1.1%)
##   Layer 2 listwise rows : 0 / 90 (0.0%)
##   Layer 2 sporadic rows : 1 / 90 (1.1%)
##   Layer 2 missing cells : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 3 / 3 (100.0%)
##   G features by layer
##     Layer 1              : 3 / 3 (100.0%)
##     Layer 2              : 3 / 3 (100.0%)
##   Z features
##     Layer 1 selected     : 10 / 10 (100.0%)
##     Layer 1 multi-cluster: 10
##     Layer 2 selected     : 10 / 10 (100.0%)
##     Layer 2 multi-cluster: 10
## 
## Model fit statistics
##   Log-likelihood         : -2435.58
##   BIC                    : 6095.11
##   Number of parameters   : 272
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Z: mean of omics data for each latent cluster of each layer 
## Layer 1
## 
##                                  estimate  norm_lower  norm_upper sig
## Layer1.cg_GRHL3.cluster1       0.01912123 -0.40529135  0.37831601    
## Layer1.cg_BTF3L4.cluster1      0.02483414 -0.84282012  0.33802948    
## Layer1.cg_AL358472.7.cluster1 -0.05982001 -0.54493733  0.65638079    
## Layer1.cg_HDGF.cluster1        0.19304072 -0.57217715  0.54234897    
## Layer1.cg_TDRD5.cluster1      -0.13876550 -0.46072088  0.11506437    
## Layer1.cg_CSRNP3.cluster1     -0.07407272 -0.38505903 -0.05540362   *
## Layer1.cg_HSPD1.cluster1       0.18589857 -0.03807418  0.39941641    
## Layer1.cg_EPM2AIP1.cluster1   -0.03858839 -0.73394880  1.24231509    
## Layer1.cg_AC025171.1.cluster1 -0.07291502 -0.04604864  0.42825110    
## Layer1.cg_VTRNA1_3.cluster1    0.15834446  0.02372599  0.56521730   *
## Layer1.cg_GRHL3.cluster2       2.78560954  2.44151193  4.18122514   *
## Layer1.cg_BTF3L4.cluster2      2.77026216  2.88327749  4.11201808   *
## Layer1.cg_AL358472.7.cluster2  2.85048084  2.67344422  4.04445646   *
## Layer1.cg_HDGF.cluster2       -0.18890613 -0.72858828  0.15269813    
## Layer1.cg_TDRD5.cluster2       0.13343713 -0.37070135  0.80082923    
## Layer1.cg_CSRNP3.cluster2      0.08017097  0.09727878  0.43152702   *
## Layer1.cg_HSPD1.cluster2      -0.20520028 -0.37004680 -0.17239242   *
## Layer1.cg_EPM2AIP1.cluster2    0.03880921 -0.60938194  0.12635912    
## Layer1.cg_AC025171.1.cluster2  0.04249461 -0.32508065  0.29220864    
## Layer1.cg_VTRNA1_3.cluster2   -0.18440941 -0.74654954 -0.11037622   *
## 
## Layer 2
## 
##                                      estimate    norm_lower  norm_upper sig
## Layer2.tc_TC01006069_nc.cluster1 -2.761572789 -4.6619097124 -1.76255653   *
## Layer2.tc_SLC9A4.cluster1        -3.021703480 -5.2052357438 -1.93884815   *
## Layer2.tc_RAB6C_AS1.cluster1     -2.735259250 -5.0618569867 -1.40502571   *
## Layer2.tc_LOC100129029.cluster1  -0.047193030 -0.4357488656  0.31453017    
## Layer2.tc_BRE.cluster1           -0.073892278 -1.0661973387  0.98141302    
## Layer2.tc_TC03001220_nc.cluster1  0.029444093 -0.1464439817 -0.07323575   *
## Layer2.tc_TC04002114_nc.cluster1 -0.032501823 -0.7211652030  0.67833866    
## Layer2.tc_TC04002369_nc.cluster1  0.133308600  0.2049254741  0.45294577   *
## Layer2.tc_BEND4.cluster1         -0.086573422 -0.1298605214  0.04567261    
## Layer2.tc_SLC9A3.cluster1        -0.007742854 -1.1453296568  0.38343681    
## Layer2.tc_TC01006069_nc.cluster2 -0.150145472 -0.0001158781  0.55169218    
## Layer2.tc_SLC9A4.cluster2         0.062198525  0.1117282292  0.69021809   *
## Layer2.tc_RAB6C_AS1.cluster2     -0.174299961 -0.5681566500  0.27904444    
## Layer2.tc_LOC100129029.cluster2   0.043320800 -0.1018327560  0.19589746    
## Layer2.tc_BRE.cluster2            0.067829351 -0.5161712756  0.82963968    
## Layer2.tc_TC03001220_nc.cluster2 -0.027028179 -0.0652984637  0.45670741    
## Layer2.tc_TC04002114_nc.cluster2  0.029835020 -1.2082675552  0.92546117    
## Layer2.tc_TC04002369_nc.cluster2 -0.122370510 -0.5759879681 -0.05441673   *
## Layer2.tc_BEND4.cluster2          0.079469996 -0.3037237349  0.22939867    
## Layer2.tc_SLC9A3.cluster2         0.007107546  0.0645591481  0.20386746   *
## 
## (2) E: intercept and odds ratio of being assigned to each latent cluster for each exposure for each layer 
## Layer 1
## 
##                        estimate norm_lower norm_upper sig
## (Intercept).cluster2 -1.1612442 -13.700090   4.605863    
## g_causal_1.cluster2   1.4619640  -1.068428   4.017602    
## g_causal_2.cluster2  -1.4793132  -3.006403  -1.486334   *
## g_causal_3.cluster2   0.4022122  -0.745642   1.166268    
## 
## Layer 2
## 
##                        estimate norm_lower norm_upper sig
## (Intercept).cluster2  2.2583148  -6.984457  6.7352423    
## g_causal_1.cluster2  -1.8289379  -3.303785 -1.5706809   *
## g_causal_2.cluster2   1.9595122   2.476332  3.8982477   *
## g_causal_3.cluster2  -0.6722936  -1.226931 -0.7362678   *
## 
## 
## --- Stage 2 (early) ---
## 
## ====================================================
## LUCID Early Integration: Model Summary
## ====================================================
## 
## Model specification
##   Family                 : binary
##   Number of observations : 90
##   Number of clusters (K) : 2
## 
## Missing-data profile
##   Listwise missing rows  : 0 / 90 (0.0%)
##   Sporadic missing rows  : 1 / 90 (1.1%)
##   Missing cells total    : 1 / 900 (0.1%)
## 
## Feature selection overview
##   G features selected    : 2 / 2 (100.0%)
##   Z features selected    : 10 / 10 (100.0%)
## 
## Model fit statistics
##   Log-likelihood         : -865.95
##   BIC                    : 2348.38
##   Number of parameters   : 137
## 
## Regularization
##   Rho_G                  : 0.000
##   Rho_Z_Mu               : 0.000
##   Rho_Z_Cov              : 0.000
## 
## Detailed parameter estimates
## (1) Y (binary outcome): log odds of Y for cluster 1 (reference) and log OR for rest cluster (and log OR of covariate if included)
##                             gamma  norm_lower norm_upper sig
## (Intercept)           -0.73799854 -2.13769738 -1.5019852   *
## LC2                    2.18407289  2.32927120  4.1359708   *
## hs_child_age_yrs_None -0.02423243  0.04222547  0.1147943   *
## sex_male              -0.18297636 -0.89239360 -0.1094076   *
## 
## (2) Z: mean of omics data for each latent cluster 
##                            estimate norm_lower norm_upper sig
## miR.101.3p.cluster1    -0.006887544 -1.0637950 -0.5405671   *
## miR.125a.5p.cluster1    0.187538247 -0.6164715 -0.1664977   *
## miR.125b.1.3p.cluster1  0.027408679 -0.9294142  0.4586921    
## miR.127.3p.cluster1    -0.107628701 -0.7281524  0.2487127    
## miR.140.5p.cluster1     0.090862876 -0.6344672  0.1968797    
## miR.142.3p.cluster1    -0.004784527 -1.1217346  0.3121149    
## miR.144.5p.cluster1     0.080101772 -0.8057870  0.2574643    
## miR.19a.3p.cluster1     0.060465036 -1.1537490  0.4747618    
## miR.19b.3p.cluster1     0.108156640 -0.8969251  0.5233145    
## miR.21.5p.cluster1      0.102286783 -0.5398389  0.3570562    
## miR.101.3p.cluster2     3.007913039  1.3299884  6.1202330   *
## miR.125a.5p.cluster2    2.785670575  1.8737230  4.4753317   *
## miR.125b.1.3p.cluster2  2.968675795  1.2566966  4.8839244   *
## miR.127.3p.cluster2     0.123004230 -1.3809174  0.5945605    
## miR.140.5p.cluster2    -0.103843286 -0.6119507  0.6883734    
## miR.142.3p.cluster2     0.005468031 -0.7145894  1.1790086    
## miR.144.5p.cluster2    -0.091544883 -0.5760967  1.0163960    
## miR.19a.3p.cluster2    -0.069102898 -0.7963561  1.0453231    
## miR.19b.3p.cluster2    -0.123607589 -0.8534212  0.8443237    
## miR.21.5p.cluster2     -0.116899180 -0.7071151  0.6450535    
## 
## (3) E: intercept and odds ratio of being assigned to each latent cluster for each cluster from previous serial stage
##                                  estimate norm_lower norm_upper sig
## (Intercept).cluster2            -10.07034  -21.09962  -16.63774   *
## Stage1.Layer1.cluster2.cluster2  23.72705   44.75119   46.65391   *
## Stage1.Layer2.cluster2.cluster2 -25.85493  -51.14782  -49.96942   *

This is the most general configuration the package supports: a serial chain whose first stage is itself a parallel model over two omics layers, fitted to a binary outcome, with both listwise and sporadic missingness present. If this runs and reports sane estimates, the combination space is covered.

11) Visualization: Sankey Diagram and Cluster Omics Profiles

plot() renders an early-integration fit as a Sankey diagram: exposures flow into the latent clusters, and the clusters flow on into the omics features and the outcome. Everything about how to read it – node colour, link width and sign – is exactly as in the continuous-outcome case (see the companion vignette’s section 14); a binary outcome only changes what the final cluster -> outcome link represents (a log-odds effect rather than a mean difference).

sankey_early_bin <- plot(early_bin$fit_refit)
sankey_early_bin

As in the continuous case, plot() on a parallel or serial fit currently raises an error by design – not implemented yet.

plot_cluster_omic_profile() – which shows what the clusters are, via their fitted omics means – is unaffected by outcome family entirely: res_Mu and the separation/range/sd ranking behind it describe the X -> Z arm of the model, which a binary Y never touches. Every panel, per architecture:

prof_early_bin <- plot_cluster_omic_profile(early_bin$fit_refit, top_n = 10)
prof_early_bin[[1]]

prof_par_bin <- plot_cluster_omic_profile(
  parallel_bin$fit_refit,
  layer_names = c("methylome", "transcriptome", "miRNA"),
  top_n = 8
)
for (nm in names(prof_par_bin)) print(prof_par_bin[[nm]])

prof_ser_bin <- plot_cluster_omic_profile(serial_ae_bin$fit_refit, top_n = 8)
for (nm in names(prof_ser_bin)) print(prof_ser_bin[[nm]])

The parallel model’s methylome panel and the serial model’s stage-1 panel above look almost identical, and that isn’t a rendering glitch: stage 1 of an all-early serial chain and the corresponding layer of a parallel fit are both fit as an early-integration model on the same methylome matrix, with the same K and the same (default, mclust-based) initialization. The exposure/outcome coupling that distinguishes them is comparatively weak, so both converge to nearly the same cluster solution – which is itself a useful sanity check that the methylation clustering is robust to which architecture surfaces it.

For the full discussion of what the importance argument measures and why (separation vs. range vs. sd), and how to pull the ranking behind a plot out as a plain table, see lucid_3models_normal_outcome.Rmd’s section 15 – none of that changes here, so it isn’t repeated.

12) Prediction: Labels or Probabilities

predict_lucid()’s response argument controls whether a binary outcome comes back as class labels or as probabilities. Both draw on the early-model fit from section 7.

pred_lab <- predict_lucid(model = early_bin$fit_refit,
                          G = early_bin$inputs$G, Z = early_bin$inputs$Z,
                          CoG = CoG, CoY = CoY, response = TRUE)
pred_prob <- predict_lucid(model = early_bin$fit_refit,
                           G = early_bin$inputs$G, Z = early_bin$inputs$Z,
                           CoG = CoG, CoY = CoY, response = FALSE)

cat("response = TRUE  ->", paste(head(pred_lab$pred.y, 8), collapse = " "), "(class labels)\n")
## response = TRUE  -> 1 1 1 0 1 1 0 0 (class labels)
cat("response = FALSE ->", paste(round(head(pred_prob$pred.y, 8), 3), collapse = " "), "(probabilities)\n")
## response = FALSE -> 0.511 0.509 0.548 0.409 0.571 0.565 0.38 0.421 (probabilities)

The two rows describe the same underlying prediction at different granularities: each label in the first row is simply the second row’s probability rounded to whichever side of 0.5 it falls on. Use response = FALSE when the downstream use needs the actual predicted risk (e.g. computing a mean predicted probability, or a classification threshold other than 0.5), and response = TRUE when a hard label is what’s needed.

13) Closing Notes and Session Info

Every model this vignette fits is registered below. This is the document checking itself: if a fit failed, its object would be missing or of the wrong class, and it would be listed here rather than passing unnoticed.

check_obj("early_bin",             "list", "7 early binary")
check_obj("parallel_bin",          "list", "8 parallel binary")
check_obj("serial_ae_bin",         "list", "9 serial all-early binary")
check_obj("serial_mixed_bin",      "list", "10 serial mixed binary")
check_obj("prof_early_bin",        "list", "11 omics profile (early)")
check_obj("prof_par_bin",          "list", "11 omics profile (parallel)")
check_obj("prof_ser_bin",          "list", "11 omics profile (serial)")
check_obj("pred_lab",              "list", "12 predict (labels)")
check_obj("pred_prob",             "list", "12 predict (probabilities)")

status <- do.call(rbind, .reg$rows)
print(status, row.names = FALSE)
##                      section           object class status
##               7 early binary        early_bin  list     ok
##            8 parallel binary     parallel_bin  list     ok
##    9 serial all-early binary    serial_ae_bin  list     ok
##       10 serial mixed binary serial_mixed_bin  list     ok
##     11 omics profile (early)   prof_early_bin  list     ok
##  11 omics profile (parallel)     prof_par_bin  list     ok
##    11 omics profile (serial)     prof_ser_bin  list     ok
##          12 predict (labels)         pred_lab  list     ok
##   12 predict (probabilities)        pred_prob  list     ok
cat(sprintf("\n%d of %d registered steps ok; %d not ok\n",
            sum(status$status == "ok"), nrow(status),
            sum(status$status != "ok")))
## 
## 9 of 9 registered steps ok; 0 not ok

Session Info

sessionInfo()
## R version 4.4.0 (2024-04-24)
## Platform: aarch64-apple-darwin20
## Running under: macOS 26.6.2
## 
## Matrix products: default
## BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib 
## LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
## 
## locale:
## [1] C/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## time zone: America/Los_Angeles
## tzcode source: internal
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] LUCIDus_3.2.0
## 
## loaded via a namespace (and not attached):
##  [1] sass_0.4.10        generics_0.1.4     shape_1.4.6.1      stringi_1.8.7     
##  [5] lattice_0.22-7     hms_1.1.4          digest_0.6.39      magrittr_2.0.4    
##  [9] evaluate_1.0.5     grid_4.4.0         RColorBrewer_1.1-3 iterators_1.0.14  
## [13] fastmap_1.2.0      foreach_1.5.2      jsonlite_2.0.0     glmnet_4.1-10     
## [17] Matrix_1.7-4       progress_1.2.3     nnet_7.3-20        survival_3.8-3    
## [21] mclust_6.1.2       scales_1.4.0       codetools_0.2-20   networkD3_0.4.1   
## [25] jquerylib_0.1.4    cli_3.6.5          rlang_1.1.6        crayon_1.5.3      
## [29] splines_4.4.0      withr_3.0.2        cachem_1.1.0       yaml_2.3.11       
## [33] tools_4.4.0        dplyr_1.1.4        ggplot2_4.0.2      boot_1.3-32       
## [37] vctrs_0.6.5        R6_2.6.1           lifecycle_1.0.4    htmlwidgets_1.6.4 
## [41] pkgconfig_2.0.3    glasso_1.11        bslib_0.9.0        pillar_1.11.1     
## [45] gtable_0.3.6       glue_1.8.0         Rcpp_1.1.0         tidyselect_1.2.1  
## [49] xfun_0.54          tibble_3.3.0       data.tree_1.2.0    knitr_1.50        
## [53] dichromat_2.0-0.1  farver_2.1.2       htmltools_0.5.9    igraph_2.2.1      
## [57] labeling_0.4.3     rmarkdown_2.30     compiler_4.4.0     prettyunits_1.2.0 
## [61] S7_0.2.1