# LUCIDus Comprehensive Guide: Models, APIs, and End-to-End Functionality

## 1) Purpose of This Document

LUCID (Latent Unknown Clusters by Integrating multi-omics Data) finds latent subgroups of subjects that are simultaneously (1) predictable from a set of exposures `G`, (2) characterized by distinct omics profiles `Z`, and (3) associated with a health outcome `Y`. This document is a comprehensive, user-facing tour of the package’s public API: everything a user needs to fit, tune, summarize, predict from, bootstrap, and visualize a LUCID model, worked through end to end on a small simulated dataset. It intentionally does not describe internal implementation details – only functions you are meant to call directly.

It covers:

- the three model architectures and what each one assumes:
  - early integration – one joint set of clusters from all omics layers pooled together
  - parallel integration – one cluster variable per omics layer, fit jointly
  - serial integration – a chain of cluster stages, each built on the previous stage’s cluster assignment
- fitting APIs: `estimate_lucid()`, `tune_lucid()`, `lucid()`
- post-fit APIs: `summary()`, `predict_lucid()` (including `g_computation` mode), `boot_lucid()`, `plot()`, `plot_cluster_omic_profile()`
- extractor functions for a fitted model’s output: `get_selected_G()`, `get_selected_Z()`, `get_cluster_assignment()`, `get_top_omics_features()`
- missing-data diagnostics and imputation: `check_na()`, `analyze_missing_pattern()`, `safe_impute()`, `check_imputation_quality()`

Two things intentionally stay out of view. First, several internal EM building blocks (a low-level data-filling routine, and a handful of numerical-stability safeguards for the optimizer) support the functions above but are not exported and are not part of the API this guide teaches; section 7 explains what they do for you conceptually where it matters. Second, this is a breadth guide – runtime is kept small throughout so the whole document fits together as one example. Section 3-model tutorials (`lucid_3models_normal_outcome.Rmd`, `lucid_3models_binary_outcome.Rmd`) go deeper on each architecture, including the two-step penalized-screen-then- refit workflow used for real feature selection.

## 2) Quick Functionality Map

Functionality | Main API
--- | ---
Fit early/parallel/serial model directly | `estimate_lucid()`
Grid search for `K` and penalties | `tune_lucid()`
One-step wrapper (fit or tune+fit) | `lucid()`
Structured model summary | `summary()`
Prediction and cluster assignment on new/held-out data | `predict_lucid()`
Bootstrap CI inference | `boot_lucid()`
Sankey-style path visualization | `plot()`
Per-cluster omics profile visualization | `plot_cluster_omic_profile()`
Extract selected exposures / omics features from a fit | `get_selected_G()`, `get_selected_Z()`
Extract hard cluster assignment from a fit | `get_cluster_assignment()`
Extract top-N most important omics features from a fit | `get_top_omics_features()`
Missingness diagnostics | `check_na()`, `analyze_missing_pattern()`
Robust imputation helpers | `safe_impute()`, `check_imputation_quality()`

The four `get_*()` extractors share one design point worth calling out up front: every one of them takes only the fitted model object and figures out on its own whether it is looking at an early, parallel, or serial fit. You never pass a `lucid_model` argument to them, and (since this release) `predict_lucid()` and `boot_lucid()` no longer require one either – they detect it from `class(model)`. The only place you still name the model type explicitly is when fitting one, since that is the argument that decides which architecture gets fit in the first place.

## 3) Setup

    library(LUCIDus)

## 4) Build Associated Demo Data (Not Independent Sampling)

A LUCID model only has something interesting to find if `G`, the latent cluster, `Z`, and `Y` are actually related – fitting it to independent noise would produce arbitrary clusters and make every example below meaningless. This simulation deliberately wires in that structure: a subset of exposures in `G` drive a binary latent state `x`, `x` in turn shifts each omics layer’s means apart (so the layers actually separate by cluster) and shifts the outcome `Y`. That known ground truth is also what lets the fitted models’ recovered clusters and selected features be checked for sanity throughout this document.

    make_demo_data <- function(n = 80, pG = 6, pZ = 4, seed = 20260309) {
      set.seed(seed)

      # Exposures
      G <- matrix(rnorm(n * pG), nrow = n, ncol = pG)
      colnames(G) <- paste0("G", seq_len(pG))

      # Covariates associated with exposures
      CoG <- cbind(
        cov_g1 = G[, 1] + 0.2 * G[, 2] + rnorm(n, sd = 0.2),
        cov_g2 = -0.3 * G[, 3] + 0.4 * G[, 4] + rnorm(n, sd = 0.25)
      )
      CoY <- CoG

      # Latent cluster driver
      lin <- 1.0 * G[, 1] - 0.8 * G[, 2] + 0.4 * CoG[, 1]
      prob_x <- plogis(lin)
      x <- rbinom(n, size = 1, prob = prob_x)

      # Layer 1 and 2 for parallel stage
      Z1 <- cbind(
        1.2 * x + 0.5 * G[, 1] + rnorm(n, sd = 0.5),
        1.0 * x - 0.4 * G[, 2] + rnorm(n, sd = 0.5),
        0.8 * x + 0.3 * G[, 3] + rnorm(n, sd = 0.5),
        0.6 * x + 0.2 * G[, 4] + rnorm(n, sd = 0.5)
      )

      Z2 <- cbind(
        -1.1 * x + 0.45 * G[, 2] + rnorm(n, sd = 0.5),
        -0.9 * x - 0.35 * G[, 1] + rnorm(n, sd = 0.5),
        -0.7 * x + 0.25 * G[, 5] + rnorm(n, sd = 0.5),
        -0.5 * x + 0.20 * G[, 6] + rnorm(n, sd = 0.5)
      )

      # Layer 3 for serial second stage (early stage)
      Z3 <- cbind(
        0.9 * x + 0.30 * G[, 1] + rnorm(n, sd = 0.55),
        0.7 * x - 0.25 * G[, 3] + rnorm(n, sd = 0.55),
        -0.8 * x + 0.20 * G[, 4] + rnorm(n, sd = 0.55),
        -0.6 * x + 0.15 * G[, 6] + rnorm(n, sd = 0.55)
      )

      colnames(Z1) <- paste0("Z1_f", seq_len(pZ))
      colnames(Z2) <- paste0("Z2_f", seq_len(pZ))
      colnames(Z3) <- paste0("Z3_f", seq_len(pZ))

      # Outcomes
      Y_normal <- 1.1 * x + 0.5 * G[, 1] - 0.25 * G[, 3] + 0.35 * CoY[, 2] + rnorm(n, sd = 0.7)
      Y_binary <- rbinom(n, size = 1, prob = plogis(-0.2 + 1.0 * x + 0.35 * G[, 1] - 0.2 * CoY[, 1]))

      # Structures for each model
      Z_parallel <- list(layer1 = Z1, layer2 = Z2)
      Z_early <- cbind(Z1, Z2)
      Z_serial_mixed <- list(list(layer1 = Z1, layer2 = Z2), Z3)

      list(
        G = G,
        CoG = CoG,
        CoY = CoY,
        Y_normal = as.numeric(Y_normal),
        Y_binary = as.numeric(Y_binary),
        Z1 = Z1,
        Z2 = Z2,
        Z3 = Z3,
        Z_parallel = Z_parallel,
        Z_early = Z_early,
        Z_serial_mixed = Z_serial_mixed
      )
    }

    d <- make_demo_data()

The three `Z_*` structures at the bottom – `Z_early` (one concatenated matrix), `Z_parallel` (a named list of layer matrices), `Z_serial_mixed` (a list whose first element is itself a parallel-style list) – are exactly the three shapes `estimate_lucid()` expects for early, parallel, and serial fitting respectively. Building all three up front from the same underlying `Z1`/`Z2`/`Z3` layers means every model type below is fit on data that differ only in how the layers are packaged, not in what they contain.

## 5) Add Missingness for Diagnostics and Missing-Data Examples

Multi-omics studies rarely have complete data: whole layers can be missing for a subject (e.g. a blood sample was never collected – “listwise” missingness within that layer), or a handful of individual features can be missing sporadically. LUCID’s EM fitting handles both patterns natively – the missing-data helpers in section 6 exist to let you check that handling, not to do it manually. This chunk injects both patterns into copies of the data built above, purely so the diagnostics in section 6 have something to report on.

    # Early matrix missingness
    Z_early_miss <- d$Z_early
    Z_early_miss[1, ] <- NA          # listwise
    Z_early_miss[2:4, 1] <- NA       # sporadic block
    Z_early_miss[5, 3] <- NA         # sporadic cell

    # Parallel list missingness
    Z_parallel_miss <- d$Z_parallel
    Z_parallel_miss[[1]][1, ] <- NA  # listwise on layer1
    Z_parallel_miss[[2]][2, 2] <- NA # sporadic on layer2

    # Serial mixed missingness (stage1 parallel + stage2 early)
    Z_serial_miss <- list(
      list(
        layer1 = Z_parallel_miss[[1]],
        layer2 = Z_parallel_miss[[2]]
      ),
      {tmp <- d$Z3; tmp[3, ] <- NA; tmp}
    )

## 6) Missing-Data Diagnostics and Imputation

`estimate_lucid()` imputes missing omics values internally, conditional on the current model fit, as part of its EM loop – there is no separate “impute first, then fit” step for you to run. What the two functions below give you is visibility into that process: whether the missingness pattern looks the way you expect, and whether a quick standalone imputation of the same data is directionally sane.

### 6.1 `analyze_missing_pattern()` and `check_na()`

`analyze_missing_pattern()` reports how much data is missing and how it is distributed; `check_na()` additionally classifies, per subject and per omics layer, which missingness pattern applies (fully observed, listwise missing, or sporadically missing) – the same three-way classification the EM loop itself uses internally to decide how to handle each subject.

    miss_info_early <- analyze_missing_pattern(Z_early_miss)
    na_early <- check_na(Z_early_miss, lucid_model = "early")
    na_parallel <- check_na(Z_parallel_miss, lucid_model = "parallel")

    miss_info_early$total_missing

    ## [1] 0.01875

    table(na_early$indicator_na)

    ##
    ##  1  2  3
    ## 75  4  1

    na_parallel$cross_layer_summary

    ## $n_layers
    ## [1] 2
    ##
    ## $n_observations
    ## [1] 80
    ##
    ## $features_per_layer
    ## layer1 layer2
    ##      4      4
    ##
    ## $missing_pattern_counts
    ##
    ## 1 1 1 2 3 1
    ##  78   1   1
    ##
    ## $total_missing_prop
    ## [1] 0.0078125

`miss_info_early$total_missing` should match the number of `NA`s injected above (6, from the 1 listwise row, the 3-cell sporadic block, and the 1 sporadic cell). `table(na_early$indicator_na)` cross-tabulates subjects by pattern; `na_parallel$cross_layer_summary` shows the same thing per layer for the parallel structure, which is the number to check when a specific layer is suspected of being a bigger driver of missingness than the others.

### 6.2 `safe_impute()` and `check_imputation_quality()`

`safe_impute()` is a standalone imputation you can run outside of model fitting – useful for a quick look at the data, or for a workflow that needs a complete matrix for some other purpose before ever calling `estimate_lucid()`. `check_imputation_quality()` then scores how plausible the imputed values are relative to the observed distribution of each column.

    # Use a sub-matrix for concise display
    orig_sub <- Z_early_miss[, 1:4, drop = FALSE]
    imputed_sub <- safe_impute(orig_sub, method = "mean")
    quality_sub <- check_imputation_quality(orig_sub, imputed_sub)

    quality_sub$overall_quality

    ## NULL

`overall_quality` summarizes, across all imputed cells, how close the filled values landed to each column’s observed range and variance – a low score here would flag an imputation that looks implausible (e.g. a filled value far outside the observed range), which is worth knowing about a data set even before it reaches `estimate_lucid()`. `estimate_lucid()`’s own internal imputation is more informed than this standalone version – it fills a missing value conditional on the fitted cluster structure and covariance, re-estimated at every EM iteration, rather than with a single fixed column statistic – but the two are checking the same basic question: are the filled-in values sane?

## 7) Core Model Fitting: `estimate_lucid()`

`estimate_lucid()` is the single entry point for fitting any of the three architectures at one fixed `K` (number of clusters) and one fixed penalty. `lucid_model` picks the architecture; everything else (`G`, `Z`, `Y`, `family`, `K`, the `Rho_*` penalties) has the same meaning across all three. `Rho_G` penalizes the `G -> cluster` coefficients toward zero (an exposure whose coefficient is driven to exactly zero is “not selected”); `Rho_Z_Mu` and `Rho_Z_Cov` do the analogous thing for the omics mean/covariance structure, so weakly-differentiating features can be zeroed out of the cluster definition. This guide fits at `Rho_* = 0` throughout for simplicity and speed; the 3-model tutorials walk through the penalized-screen-then- zero-penalty-refit workflow that real feature selection uses.

### 7.1 Early model

Early integration concatenates every omics layer into one matrix and fits one cluster variable from the pooled features – the right choice when you expect a single latent state to manifest across all your omics layers at once.

    set.seed(101)
    fit_early <- estimate_lucid(
      lucid_model = "early",
      G = d$G,
      Z = Z_early_miss,
      Y = d$Y_normal,
      CoG = d$CoG,
      CoY = d$CoY,
      family = "normal",
      K = 2,
      Rho_G = 0,
      Rho_Z_Mu = 0,
      Rho_Z_Cov = 0,
      max_itr = 8,
      max_tot.itr = 30,
      tol = 1e-2,
      seed = 101,
      verbose = FALSE
    )

    ## Fitting LUCID early model (K = 2)...

    ## Finished LUCID early model.

    class(fit_early)

    ## [1] "early_lucid"

Every field below is documented on `?estimate_lucid`’s `@return`; the same field names and shapes are used across all three model types where a field applies to more than one. Rather than reading `fit_early$select$selectG` directly, use `get_selected_G()`/`get_selected_Z()` – they return the same information as a named vector and auto-detect the model type, so the same call works unchanged if `fit_early` were swapped for a parallel or serial fit later in a script.

    names(fit_early)

    ##  [1] "res_Beta"             "res_Mu"               "res_Sigma"
    ##  [4] "res_Gamma"            "K"                    "var.names"
    ##  [7] "init_omic.data.model" "likelihood"           "inclusion.p"
    ## [10] "family"               "select"               "useY"
    ## [13] "Z"                    "init_impute"          "init_par"
    ## [16] "Rho"                  "missing_summary"      "em_control"

    cat("likelihood:", fit_early$likelihood, "\n")

    ## likelihood: -650.09

    cat("selected G:", sum(get_selected_G(fit_early)), "of",
        length(get_selected_G(fit_early)), "\n")

    ## selected G: 6 of 6

    cat("selected Z:", sum(get_selected_Z(fit_early)), "of",
        length(get_selected_Z(fit_early)), "\n")

    ## selected Z: 8 of 8

At `Rho_G = Rho_Z_Mu = 0` nothing is actually penalized out, so every exposure and every omics feature is reported “selected” here – the counts become informative once a positive penalty is applied, as in the 3-model tutorials’ screening step.

### 7.2 Parallel model

Parallel integration fits a separate cluster variable per omics layer, jointly, rather than pooling the layers into one. Use this when the layers plausibly reflect different underlying processes – e.g. a metabolomic subtype and a methylation subtype need not coincide – and you want each to be discovered on its own terms rather than forced into a single joint cluster.

    set.seed(102)
    fit_parallel <- estimate_lucid(
      lucid_model = "parallel",
      G = d$G,
      Z = Z_parallel_miss,
      Y = d$Y_normal,
      CoG = d$CoG,
      CoY = d$CoY,
      family = "normal",
      K = c(2, 2),
      Rho_G = 0,
      Rho_Z_Mu = 0,
      Rho_Z_Cov = 0,
      max_itr = 8,
      max_tot.itr = 30,
      tol = 1e-2,
      seed = 102,
      verbose = FALSE
    )

    ## Fitting LUCID parallel model (2 layers)...

    ## Finished LUCID parallel model.

    class(fit_parallel)

    ## [1] "lucid_parallel"

Two fields differ from the early fit above: `N` (sample size) is present here but not for early, and `z` (the joint E-step responsibility array across layers, before it is marginalized into `inclusion.p`) appears only for parallel. `get_selected_G()` takes an optional `layer` argument here, since a parallel fit can select a different exposure subset per layer.

    names(fit_parallel)

    ##  [1] "res_Beta"             "res_Mu"               "res_Sigma"
    ##  [4] "res_Gamma"            "K"                    "N"
    ##  [7] "var.names"            "init_omic.data.model" "likelihood"
    ## [10] "inclusion.p"          "family"               "select"
    ## [13] "useY"                 "Z"                    "z"
    ## [16] "init_impute"          "init_par"             "Rho"
    ## [19] "missing_summary"      "em_control"

    cat("N:", fit_parallel$N, "\n")

    ## N: 80

    cat("selected G per layer:", paste(sapply(seq_along(fit_parallel$K), function(i)
                                         sum(get_selected_G(fit_parallel, layer = i))),
                                       collapse = ", "), "\n")

    ## selected G per layer: 6, 6

    sel_z_parallel <- get_selected_Z(fit_parallel)
    cat("selected Z per layer:", paste(sapply(sel_z_parallel, sum), collapse = ", "), "\n")

    ## selected Z per layer: 4, 4

`get_selected_Z()` returns a list here, one logical vector per layer, since “which omics features are selected” is itself a per-layer question for a parallel fit.

### 7.3 Serial model (mixed topology)

Serial integration chains stages together: stage 1’s cluster assignment becomes (part of) stage 2’s input, and so on. It suits a mediation-like hypothesis – e.g. exposures act on an early biological layer, which in turn shapes a later one, which in turn shapes the outcome. Any stage can itself be early or parallel; the fit below mixes a parallel first stage (two omics layers) with an early second stage (one layer), which is the most general configuration the package supports.

    set.seed(103)
    fit_serial <- estimate_lucid(
      lucid_model = "serial",
      G = d$G,
      Z = Z_serial_miss,
      Y = d$Y_normal,
      CoG = d$CoG,
      CoY = d$CoY,
      family = "normal",
      K = list(list(2, 2), 2),
      Rho_G = 0,
      Rho_Z_Mu = 0,
      Rho_Z_Cov = 0,
      max_itr = 8,
      max_tot.itr = 36,
      tol = 1e-2,
      seed = 103,
      verbose = FALSE
    )

    ## Fitting LUCID serial model (2 stages)...

    ##   Stage 1/2 (parallel) finished: log-likelihood = -607.523.

    ##   Stage 2/2 (early) finished: log-likelihood = -399.220.
    ## Finished LUCID serial model.

    class(fit_serial)

    ## [1] "lucid_serial"

    length(fit_serial$submodel)

    ## [1] 2

Serial adds `submodel` (the fitted stage models, each a complete `early_lucid`/`lucid_parallel` object) and `res_Delta` (between-stage transition coefficients). `likelihood` and `select` are present, same as early/parallel, but are aggregates: `likelihood` sums each stage’s own log-likelihood (no single joint EM loop exists to report one from), and `select` is stage 1’s own selection only – the one stage whose `G` is the cohort’s real exposures, not a previous stage’s cluster probabilities. `get_selected_G()` on a serial fit always returns stage 1’s selection for the same reason; `get_selected_Z()` returns a list, one element per stage, shaped like whatever that stage’s own architecture is (a vector for an early stage, a list of layers for a parallel stage).

    names(fit_serial)

    ##  [1] "res_Beta"             "res_Mu"               "res_Sigma"
    ##  [4] "res_Delta"            "res_Gamma"            "K"
    ##  [7] "N"                    "var.names"            "init_omic.data.model"
    ## [10] "inclusion.p"          "family"               "useY"
    ## [13] "Z"                    "init_impute"          "init_par"
    ## [16] "submodel"             "missing_summary"      "Rho"
    ## [19] "em_control"           "likelihood"           "select"

    cat("top-level likelihood (sum over stages):", fit_serial$likelihood, "\n")

    ## top-level likelihood (sum over stages): -1006.743

    cat("top-level select is stage 1's select:",
        identical(fit_serial$select, fit_serial$submodel[[1]]$select), "\n")

    ## top-level select is stage 1's select: TRUE

    sel_z_serial <- get_selected_Z(fit_serial)
    cat("stage 1 (parallel) selected Z per layer:",
        paste(sapply(sel_z_serial[[1]], sum), collapse = ", "), "\n")

    ## stage 1 (parallel) selected Z per layer: 4, 4

    cat("stage 2 (early) selected Z:", sum(sel_z_serial[[2]]), "\n")

    ## stage 2 (early) selected Z: 4

### 7.4 Verbose Logging Demos (`verbose = TRUE`)

`verbose = TRUE` prints one line per EM iteration – the current log-likelihood and, where relevant, the change since the last iteration – which is the fastest way to confirm a fit is actually converging (steadily increasing, then flattening) rather than diverging or oscillating. The fits below are intentionally lightweight (a 30-subject subsample, tiny iteration caps) and only serve to show what that log looks like for each architecture.

    n_demo <- 30
    G_demo <- d$G[1:n_demo, , drop = FALSE]
    Y_demo <- d$Y_normal[1:n_demo]
    CoG_demo <- d$CoG[1:n_demo, , drop = FALSE]
    CoY_demo <- d$CoY[1:n_demo, , drop = FALSE]
    Z_early_demo <- d$Z_early[1:n_demo, , drop = FALSE]
    Z_parallel_demo <- lapply(d$Z_parallel, function(z) z[1:n_demo, , drop = FALSE])
    Z_serial_demo <- list(
      lapply(d$Z_parallel, function(z) z[1:n_demo, , drop = FALSE]),
      d$Z3[1:n_demo, , drop = FALSE]
    )

    set.seed(111)
    fit_early_verbose <- estimate_lucid(
      lucid_model = "early",
      G = G_demo,
      Z = Z_early_demo,
      Y = Y_demo,
      CoG = CoG_demo,
      CoY = CoY_demo,
      family = "normal",
      K = 2,
      Rho_G = 0,
      Rho_Z_Mu = 0,
      Rho_Z_Cov = 0,
      max_itr = 2,
      max_tot.itr = 8,
      tol = 1e-2,
      seed = 111,
      verbose = TRUE
    )

    ## Initialize LUCID with mclust based on inclusion probabilities given by mclust
    ## Fitting Early Integration LUCID model (K = 2, Rho_G = 0, Rho_Z_Mu = 0, Rho_Z_Cov = 0)
    ## iteration 1 : E-step finished.
    ## iteration 1: log-likelihood = -211.863
    ## iteration 2 : E-step finished.
    ## iteration 2: log-likelihood = -207.327
    ## iteration 3 : E-step finished.
    ## iteration 3: log-likelihood = -207.288

    ## Finished LUCID early model: log-likelihood = -207.288.

    set.seed(112)
    fit_parallel_verbose <- estimate_lucid(
      lucid_model = "parallel",
      G = G_demo,
      Z = Z_parallel_demo,
      Y = Y_demo,
      CoG = CoG_demo,
      CoY = CoY_demo,
      family = "normal",
      K = c(2, 2),
      Rho_G = 0,
      Rho_Z_Mu = 0,
      Rho_Z_Cov = 0,
      max_itr = 2,
      max_tot.itr = 8,
      tol = 1e-2,
      seed = 112,
      verbose = TRUE
    )

    ## Fitting LUCID in Parallel model (K = 2, Rho_G = 0, Rho_Z_Mu = 0, Rho_Z_Cov = 0) (K = 2, Rho_G = 0, Rho_Z_Mu = 0, Rho_Z_Cov = 0)
    ## iteration 1 : E-step finished.
    ## iteration 1: log-likelihood = -241.834
    ## iteration 2 : E-step finished.
    ## iteration 2: log-likelihood = -241.139

    ## Finished LUCID parallel model: log-likelihood = -241.139.

    set.seed(113)
    fit_serial_verbose <- estimate_lucid(
      lucid_model = "serial",
      G = G_demo,
      Z = Z_serial_demo,
      Y = Y_demo,
      CoG = CoG_demo,
      CoY = CoY_demo,
      family = "normal",
      K = list(list(2, 2), 2),
      Rho_G = 0,
      Rho_Z_Mu = 0,
      Rho_Z_Cov = 0,
      max_itr = 2,
      max_tot.itr = 10,
      tol = 1e-2,
      seed = 113,
      verbose = TRUE
    )

    ## Fitting LUCID serial model (Stage 1/2)...
    ## Fitting LUCID in Parallel model (K = 2, Rho_G = 0, Rho_Z_Mu = 0, Rho_Z_Cov = 0) (K = 2, Rho_G = 0, Rho_Z_Mu = 0, Rho_Z_Cov = 0)
    ## iteration 1 : E-step finished.
    ## iteration 1: log-likelihood = -206.521
    ## iteration 2 : E-step finished.
    ## iteration 2: log-likelihood = -205.875

    ## Finished LUCID parallel model: log-likelihood = -205.875.
    ##
    ## Fitting LUCID serial model (Stage 2/2)...
    ## Initialize LUCID with mclust based on inclusion probabilities given by mclust
    ## Fitting Early Integration LUCID model (K = 2, Rho_G = 0, Rho_Z_Mu = 0, Rho_Z_Cov = 0)
    ## iteration 1 : E-step finished.
    ## iteration 1: log-likelihood = -156.096
    ## iteration 2 : E-step finished.
    ## iteration 2: log-likelihood = -153.735
    ## iteration 3 : E-step finished.
    ## iteration 3: log-likelihood = -152.525

    ## Finished LUCID early model: log-likelihood = -152.525.
    ##
    ## Success: LUCID serial model constructed!

## 8) Tuning and Wrapper APIs: `tune_lucid()` and `lucid()`

Choosing `K` and the penalty strengths by hand, as section 7 did, only works when you already have a good guess. `tune_lucid()` instead fits a whole grid of `K`/`Rho_*` combinations and reports each one’s BIC, so the combination that best balances fit against model complexity can be selected systematically rather than by trial and error. `lucid()` goes one step further: it runs the same tuning search, picks the BIC-best candidate, and – importantly – refits that winner at zero penalty, so the returned model’s coefficients are not shrunk by the same penalty that was used only to decide which features to keep. That two-step logic (penalized search for selection, zero-penalty refit for estimation) is worth remembering; the 3-model tutorials walk through doing it by hand for cases where more control over the refit is needed than `lucid()`’s default gives.

### 8.1 `tune_lucid()` with a small early-model grid

    set.seed(104)

    tune_early <- tune_lucid(
      G = d$G,
      Z = d$Z_early,
      Y = d$Y_normal,
      CoG = d$CoG,
      CoY = d$CoY,
      family = "normal",
      lucid_model = "early",
      K = 2:3,
      Rho_G = 0,
      Rho_Z_Mu = 0,
      Rho_Z_Cov = 0,
      max_itr = 6,
      max_tot.itr = 24,
      seed = 104
    )

    ## Fitting LUCID early model (K = 2)...

    ## Finished LUCID early model.
    ##
    ## ====================================================
    ## LUCID Early Integration: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : normal
    ##   Number of observations : 80
    ##   Number of clusters (K) : 2
    ##
    ## Missing-data profile
    ##   Listwise missing rows  : 0 / 80 (0.0%)
    ##   Sporadic missing rows  : 0 / 80 (0.0%)
    ##   Missing cells total    : 0 / 640 (0.0%)
    ##
    ## Feature selection overview
    ##   G features selected    : 6 / 6 (100.0%)
    ##   Z features selected    : 8 / 8 (100.0%)
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -704.46
    ##   BIC                    : 1860.28
    ##   Number of parameters   : 103
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Detailed parameter estimates
    ## (1) Y (continuous outcome): cluster 1 mean (intercept), mean differences for other clusters, and covariate effects
    ##                 Gamma
    ## (Intercept) 0.3125382
    ## cluster2    0.2619192
    ## cov_g1      0.6175223
    ## cov_g2      0.5320559
    ##
    ## (2) Z: mean of omics data for each latent cluster
    ##       mu_cluster1 mu_cluster2
    ## Z1_f1 -0.56900590   0.5170944
    ## Z1_f2  0.15183098   0.5103446
    ## Z1_f3 -0.12642173   0.4027654
    ## Z1_f4 -0.01756913   0.3191024
    ## Z2_f1 -0.09641616  -0.4994498
    ## Z2_f2  0.23868326  -0.4993390
    ## Z2_f3  0.25290360  -0.5092025
    ## Z2_f4  0.08254873  -0.3134526
    ##
    ## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure
    ##                            beta         OR
    ## (Intercept).cluster2  2.3141047 10.1158624
    ## G1.cluster2           0.6683651  1.9510450
    ## G2.cluster2          -0.2712207  0.7624482
    ## G3.cluster2          -0.3940201  0.6743405
    ## G4.cluster2           0.9889927  2.6885250
    ## G5.cluster2          -0.6029099  0.5472170
    ## G6.cluster2          -0.1097568  0.8960520
    ## cov_g1.cluster2       0.2329658  1.2623383
    ## cov_g2.cluster2      -0.8264530  0.4375987
    ## Fitting LUCID early model (K = 3)...

    ## Finished LUCID early model.
    ##
    ## ====================================================
    ## LUCID Early Integration: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : normal
    ##   Number of observations : 80
    ##   Number of clusters (K) : 3
    ##
    ## Missing-data profile
    ##   Listwise missing rows  : 0 / 80 (0.0%)
    ##   Sporadic missing rows  : 0 / 80 (0.0%)
    ##   Missing cells total    : 0 / 640 (0.0%)
    ##
    ## Feature selection overview
    ##   G features selected    : 6 / 6 (100.0%)
    ##   Z features selected    : 8 / 8 (100.0%)
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -667.52
    ##   BIC                    : 2027.40
    ##   Number of parameters   : 158
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Detailed parameter estimates
    ## (1) Y (continuous outcome): cluster 1 mean (intercept), mean differences for other clusters, and covariate effects
    ##                  Gamma
    ## (Intercept) -0.3562758
    ## cluster2     0.6505660
    ## cluster3     1.2050978
    ## cov_g1       0.5174619
    ## cov_g2       0.5870936
    ##
    ## (2) Z: mean of omics data for each latent cluster
    ##       mu_cluster1 mu_cluster2 mu_cluster3
    ## Z1_f1 -0.38770100 -0.59861975   0.7271796
    ## Z1_f2 -0.35709134  0.11509707   0.7676527
    ## Z1_f3 -0.07327279 -0.08045505   0.5112009
    ## Z1_f4 -0.07748138  0.26794562   0.3957297
    ## Z2_f1  0.44010129  0.08223693  -0.7893889
    ## Z2_f2 -0.07577175  0.61037027  -0.6111670
    ## Z2_f3 -0.17081122  0.38396071  -0.5697624
    ## Z2_f4  0.01014765 -0.10728092  -0.3700419
    ##
    ## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure
    ##                            beta          OR
    ## (Intercept).cluster2 -0.6811935  0.50601271
    ## G1.cluster2           4.0580956 57.86401080
    ## G2.cluster2          -1.9687675  0.13962885
    ## G3.cluster2          -0.7790547  0.45883954
    ## G4.cluster2           1.8085998  6.10189747
    ## G5.cluster2           1.3757889  3.95819813
    ## G6.cluster2          -1.9227067  0.14621067
    ## cov_g1.cluster2      -3.6739561  0.02537588
    ## cov_g2.cluster2      -0.8862642  0.41219276
    ## (Intercept).cluster3  3.3408897 28.24424309
    ## G1.cluster3           3.7131381 40.98221005
    ## G2.cluster3          -2.4721863  0.08440014
    ## G3.cluster3          -1.0152803  0.36230087
    ## G4.cluster3           0.3603281  1.43379981
    ## G5.cluster3           0.5800672  1.78615851
    ## G6.cluster3          -1.3117669  0.26934374
    ## cov_g1.cluster3      -1.7567726  0.17260102
    ## cov_g2.cluster3       0.4749526  1.60793799

    head(tune_early$tune_list)

    ##   K Rho_G Rho_Z_Mu Rho_Z_Cov      BIC
    ## 1 2     0        0         0 1860.275
    ## 2 3     0        0         0 2027.397

`tune_list` has one row per grid candidate, with its fitted `K`/penalty combination and its BIC; the lowest BIC in this table is the candidate `lucid()` (below) would have selected automatically.

### 8.2 `lucid()` wrapper (auto-tune over K for early model)

    set.seed(105)

    fit_lucid_wrapper <- lucid(
      G = d$G,
      Z = d$Z_early,
      Y = d$Y_normal,
      CoG = d$CoG,
      CoY = d$CoY,
      family = "normal",
      lucid_model = "early",
      K = 2:3,
      Rho_G = 0,
      Rho_Z_Mu = 0,
      Rho_Z_Cov = 0,
      max_itr = 6,
      max_tot.itr = 24,
      seed = 105
    )

    class(fit_lucid_wrapper)

    ## [1] "early_lucid"

`fit_lucid_wrapper` is a complete `early_lucid` object – the same class `estimate_lucid()` would have returned had `K = 2:3`’s BIC-winner been fit directly – so it works with every function in this guide (`summary()`, `predict_lucid()`, the `get_*()` extractors) exactly like any other fit.

## 9) Summaries: `summary()`

`summary()` is the primary way to read a fitted model’s results without digging through its raw list structure. For every model type it prints, in order: the model specification (family, sample size, `K`); a feature- selection overview (how many exposures/omics features survived, if a penalty was used); the `G -> cluster` coefficients (with odds ratios); the cluster-specific omics means; and the `cluster -> Y` coefficients. When a `boot.se`/bootstrap object is supplied (section 11), every one of those coefficient tables additionally gets a `sig` column marking rows whose 95%-normal-theory confidence interval excludes 0 with `"*"` – a quick visual scan for which effects are distinguishable from no effect at all, without reading every interval by eye.

    summary(fit_early)

    ##
    ## ====================================================
    ## LUCID Early Integration: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : normal
    ##   Number of observations : 80
    ##   Number of clusters (K) : 2
    ##
    ## Missing-data profile
    ##   Listwise missing rows  : 1 / 80 (1.2%)
    ##   Sporadic missing rows  : 4 / 80 (5.0%)
    ##   Missing cells total    : 12 / 640 (1.9%)
    ##
    ## Feature selection overview
    ##   G features selected    : 6 / 6 (100.0%)
    ##   Z features selected    : 8 / 8 (100.0%)
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -650.09
    ##   BIC                    : 1751.53
    ##   Number of parameters   : 103
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Detailed parameter estimates
    ## (1) Y (continuous outcome): cluster 1 mean (intercept), mean differences for other clusters, and covariate effects
    ##                 Gamma
    ## (Intercept) 0.1765857
    ## cluster2    1.0449481
    ## cov_g1      0.4929783
    ## cov_g2      0.6931914
    ##
    ## (2) Z: mean of omics data for each latent cluster
    ##         mu_cluster1 mu_cluster2
    ## Z1_f1 -0.0777814604   1.3323119
    ## Z1_f2  0.0840691217   1.1873927
    ## Z1_f3  0.0237814722   0.9834228
    ## Z1_f4  0.1617553558   0.5274073
    ## Z2_f1 -0.0021631579  -1.2547393
    ## Z2_f2  0.0001776934  -1.1894463
    ## Z2_f3 -0.1742793904  -0.8918661
    ## Z2_f4 -0.1083855071  -0.5296839
    ##
    ## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure
    ##                             beta        OR
    ## (Intercept).cluster2 -0.86505853 0.4210269
    ## G1.cluster2           1.49321813 4.4513977
    ## G2.cluster2          -1.12634744 0.3242153
    ## G3.cluster2           0.19971412 1.2210536
    ## G4.cluster2          -0.55799151 0.5723575
    ## G5.cluster2          -0.27541536 0.7592567
    ## G6.cluster2           0.09558619 1.1003037
    ## cov_g1.cluster2       0.08560300 1.0893738
    ## cov_g2.cluster2       0.64058435 1.8975894

    summary(fit_parallel)

    ##
    ## ====================================================
    ## LUCID Parallel: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : gaussian
    ##   Number of observations : 80
    ##   Clusters per layer     : 2, 2
    ##
    ## Missing-data profile by layer
    ##   Layer 1 listwise rows : 1 / 80 (1.2%)
    ##   Layer 1 sporadic rows : 0 / 80 (0.0%)
    ##   Layer 1 missing cells : 4 / 320 (1.2%)
    ##   Layer 2 listwise rows : 0 / 80 (0.0%)
    ##   Layer 2 sporadic rows : 1 / 80 (1.2%)
    ##   Layer 2 missing cells : 1 / 320 (0.3%)
    ##
    ## Feature selection overview
    ##   G features selected    : 6 / 6 (100.0%)
    ##   G features by layer
    ##     Layer 1              : 6 / 6 (100.0%)
    ##     Layer 2              : 6 / 6 (100.0%)
    ##   Z features
    ##     Layer 1 selected     : 4 / 4 (100.0%)
    ##     Layer 1 multi-cluster: 4
    ##     Layer 2 selected     : 4 / 4 (100.0%)
    ##     Layer 2 multi-cluster: 4
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -703.20
    ##   BIC                    : 1756.96
    ##   Number of parameters   : 80
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Detailed parameter estimates
    ## (1) Y (continuous outcome): intercept, effects of each non-reference latent cluster for each layer of Y (and effect of covariates if included)
    ##                  Gamma
    ## (Intercept) 0.04248756
    ## Layer1_LC2  0.01765860
    ## Layer2_LC2  1.06757644
    ## cov_g1      0.49345900
    ## cov_g2      0.65852863
    ##
    ## (2) Z: mean of omics data for each latent cluster of each layer
    ## Layer 1
    ##
    ##        mu_cluster1 mu_cluster2
    ## Z1_f1 -0.349168270   1.2391929
    ## Z1_f2  0.004260061   0.9934904
    ## Z1_f3 -0.154181517   0.9239008
    ## Z1_f4  0.116945108   0.4850517
    ##
    ## Layer 2
    ##
    ##       mu_cluster1 mu_cluster2
    ## Z2_f1  0.21574635  -1.2562624
    ## Z2_f2  0.08358451  -1.0078702
    ## Z2_f3 -0.12054320  -0.7762626
    ## Z2_f4 -0.08469553  -0.4845957
    ##
    ## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure for each layer
    ## Layer 1
    ##
    ##                             beta        OR
    ## (Intercept).cluster2 -0.08685128 0.9168134
    ## G1.cluster2           1.20608960 3.3403968
    ## G2.cluster2          -0.92554313 0.3963161
    ## G3.cluster2           0.63658188 1.8900095
    ## G4.cluster2          -0.73551173 0.4792601
    ## G5.cluster2          -0.04677656 0.9543006
    ## G6.cluster2          -0.09518064 0.9092087
    ##
    ## Layer 2
    ##
    ##                             beta         OR
    ## (Intercept).cluster2 -0.16662615  0.8465160
    ## G1.cluster2           3.59813736 36.5301286
    ## G2.cluster2          -1.28668079  0.2761860
    ## G3.cluster2           0.06073117  1.0626132
    ## G4.cluster2          -0.13557162  0.8732166
    ## G5.cluster2          -0.16105933  0.8512416
    ## G6.cluster2           0.05468855  1.0562116

    summary(fit_serial)

    ##
    ## ====================================================
    ## LUCID Serial: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : normal
    ##   Number of observations : 80
    ##   Number of stages       : 2
    ##   Stage 1               : parallel (K = 2,2)
    ##   Stage 2               : early (K = 2)
    ##
    ## Missing-data profile by stage
    ##   Stage 1
    ##     Layer 1 listwise/sporadic rows : 1 / 0
    ##     Layer 2 listwise/sporadic rows : 0 / 1
    ##   Stage 2
    ##     Listwise rows         : 1
    ##     Sporadic rows         : 0
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -1006.74
    ##   BIC                    : 2499.89
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Stage-wise detailed parameter estimates
    ##
    ## --- Stage 1 (parallel) ---
    ##
    ## ====================================================
    ## LUCID Parallel: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : gaussian
    ##   Number of observations : 80
    ##   Clusters per layer     : 2, 2
    ##
    ## Missing-data profile by layer
    ##   Layer 1 listwise rows : 1 / 80 (1.2%)
    ##   Layer 1 sporadic rows : 0 / 80 (0.0%)
    ##   Layer 1 missing cells : 4 / 320 (1.2%)
    ##   Layer 2 listwise rows : 0 / 80 (0.0%)
    ##   Layer 2 sporadic rows : 1 / 80 (1.2%)
    ##   Layer 2 missing cells : 1 / 320 (0.3%)
    ##
    ## Feature selection overview
    ##   G features selected    : 6 / 6 (100.0%)
    ##   G features by layer
    ##     Layer 1              : 6 / 6 (100.0%)
    ##     Layer 2              : 6 / 6 (100.0%)
    ##   Z features
    ##     Layer 1 selected     : 4 / 4 (100.0%)
    ##     Layer 1 multi-cluster: 4
    ##     Layer 2 selected     : 4 / 4 (100.0%)
    ##     Layer 2 multi-cluster: 4
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -607.52
    ##   BIC                    : 1539.32
    ##   Number of parameters   : 74
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Detailed parameter estimates
    ## (1) Z: mean of omics data for each latent cluster of each layer
    ## Layer 1
    ##
    ##        mu_cluster1 mu_cluster2
    ## Z1_f1 -0.352615720   1.2292345
    ## Z1_f2  0.002360677   0.9869976
    ## Z1_f3 -0.156731519   0.9173883
    ## Z1_f4  0.116972344   0.4817741
    ##
    ## Layer 2
    ##
    ##       mu_cluster1 mu_cluster2
    ## Z2_f1  -1.2489062  0.22761663
    ## Z2_f2  -1.0010499  0.09162188
    ## Z2_f3  -0.7529336 -0.13230863
    ## Z2_f4  -0.4929066 -0.07270329
    ##
    ## (2) E: intercept and odds ratio of being assigned to each latent cluster for each exposure for each layer
    ## Layer 1
    ##
    ##                             beta        OR
    ## (Intercept).cluster2 -0.05763017 0.9439990
    ## G1.cluster2           1.21618905 3.3743039
    ## G2.cluster2          -0.95080523 0.3864297
    ## G3.cluster2           0.67635032 1.9666868
    ## G4.cluster2          -0.74690763 0.4738295
    ## G5.cluster2          -0.06497218 0.9370935
    ## G6.cluster2          -0.11292848 0.8932145
    ##
    ## Layer 2
    ##
    ##                             beta         OR
    ## (Intercept).cluster2  0.10180517 1.10716774
    ## G1.cluster2          -3.76996919 0.02305277
    ## G2.cluster2           1.30981565 3.70549056
    ## G3.cluster2          -0.20510303 0.81456338
    ## G4.cluster2           0.05465422 1.05617534
    ## G5.cluster2           0.16024275 1.17379577
    ## G6.cluster2          -0.05478440 0.94668923
    ##
    ##
    ## --- Stage 2 (early) ---
    ##
    ## ====================================================
    ## LUCID Early Integration: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : normal
    ##   Number of observations : 80
    ##   Number of clusters (K) : 2
    ##
    ## Missing-data profile
    ##   Listwise missing rows  : 1 / 80 (1.2%)
    ##   Sporadic missing rows  : 0 / 80 (0.0%)
    ##   Missing cells total    : 4 / 320 (1.2%)
    ##
    ## Feature selection overview
    ##   G features selected    : 2 / 2 (100.0%)
    ##   Z features selected    : 4 / 4 (100.0%)
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -399.22
    ##   BIC                    : 960.58
    ##   Number of parameters   : 37
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Detailed parameter estimates
    ## (1) Y (continuous outcome): cluster 1 mean (intercept), mean differences for other clusters, and covariate effects
    ##                  Gamma
    ## (Intercept) -0.2702597
    ## cluster2     1.0434643
    ## cov_g1       0.4964385
    ## cov_g2       0.4379221
    ##
    ## (2) Z: mean of omics data for each latent cluster
    ##       mu_cluster1 mu_cluster2
    ## Z3_f1 -0.42148130   0.5295890
    ## Z3_f2  0.25828699   0.3402714
    ## Z3_f3  0.26462045  -0.5481375
    ## Z3_f4 -0.07877362  -0.3163739
    ##
    ## (3) E: intercept and odds ratio of being assigned to each latent cluster for each cluster from previous serial stage
    ##                                      beta           OR
    ## (Intercept).cluster2             42.07607 1.876740e+18
    ## Stage1.Layer1.cluster2.cluster2  59.87371 1.006518e+26
    ## Stage1.Layer2.cluster2.cluster2 -42.99601 2.123585e-19

At `Rho_* = 0` the selection overview above reports everything selected (as noted in section 7); with a positive penalty, this is the table that shows how much was screened out and at what rate.

## 10) Prediction: `predict_lucid()`

`predict_lucid()` runs a fitted model’s E-step on new (or the same) `G`/`Z` data to obtain posterior cluster probabilities and, from those, a predicted outcome. It is what you use to score subjects who were not part of the fitting sample, or to double-check a model’s assignments on its own training data.

### 10.1 Standard prediction

    # Use lightweight no-covariate fits for robust prediction demo.
    set.seed(205)
    fit_early_pred <- estimate_lucid(
      lucid_model = "early",
      G = d$G,
      Z = d$Z_early,
      Y = d$Y_normal,
      family = "normal",
      K = 2,
      max_itr = 6,
      max_tot.itr = 20,
      tol = 1e-2,
      seed = 205
    )

    ## Fitting LUCID early model (K = 2)...

    ## Finished LUCID early model.

    set.seed(206)
    fit_parallel_pred <- estimate_lucid(
      lucid_model = "parallel",
      G = d$G,
      Z = d$Z_parallel,
      Y = d$Y_normal,
      family = "normal",
      K = c(2, 2),
      max_itr = 6,
      max_tot.itr = 20,
      tol = 1e-2,
      seed = 206
    )

    ## Fitting LUCID parallel model (2 layers)...

    ## Finished LUCID parallel model.

    set.seed(207)
    fit_serial_pred <- estimate_lucid(
      lucid_model = "serial",
      G = d$G,
      Z = d$Z_serial_mixed,
      Y = d$Y_normal,
      family = "normal",
      K = list(list(2, 2), 2),
      max_itr = 6,
      max_tot.itr = 24,
      tol = 1e-2,
      seed = 207
    )

    ## Fitting LUCID serial model (2 stages)...

    ##   Stage 1/2 (parallel) finished: log-likelihood = -611.077.

    ##   Stage 2/2 (early) finished: log-likelihood = -383.375.
    ## Finished LUCID serial model.

    pred_early <- predict_lucid(
      model = fit_early_pred,
      G = d$G,
      Z = d$Z_early,
      Y = d$Y_normal
    )

    pred_parallel <- predict_lucid(
      model = fit_parallel_pred,
      G = d$G,
      Z = d$Z_parallel,
      Y = d$Y_normal
    )

    pred_serial <- predict_lucid(
      model = fit_serial_pred,
      G = d$G,
      Z = d$Z_serial_mixed,
      Y = d$Y_normal
    )

    # Cluster assignment. pred.x is a vector for early and a list -- by layer for
    # parallel, by stage for serial -- so the blocks are summarised in one place.
    # pred.x nests: a vector for early, a list by layer for parallel, and for
    # serial a list by stage whose elements are themselves lists when that stage is
    # a parallel submodel. Flatten to the leaves so each cluster variable is counted
    # on its own -- pooling a parallel stage's layers would report twice as many
    # assignments as there are subjects.
    flatten_blocks <- function(x, path = "") {
      if (!is.list(x)) return(stats::setNames(list(as.numeric(x)), path))
      out <- list()
      for (i in seq_along(x)) {
        nm <- if (nzchar(path)) paste0(path, ".", i) else as.character(i)
        out <- c(out, flatten_blocks(x[[i]], nm))
      }
      out
    }

    cluster_sizes <- function(pred_x, label) {
      blocks <- flatten_blocks(pred_x)
      nms <- names(blocks)
      # index by position: the single block of an early model is named "", and
      # blocks[[""]] does not select anything.
      do.call(rbind, lapply(seq_along(blocks), function(i) {
        tb <- table(factor(blocks[[i]]))
        data.frame(model = label,
                   block = if (!nzchar(nms[i])) "-" else nms[i],
                   cluster = names(tb), n = as.integer(tb), row.names = NULL)
      }))
    }

    rbind(
      cluster_sizes(pred_early$pred.x,    "early"),
      cluster_sizes(pred_parallel$pred.x, "parallel"),
      cluster_sizes(pred_serial$pred.x,   "serial")
    )

    ##       model block cluster  n
    ## 1     early     -       1 32
    ## 2     early     -       2 48
    ## 3  parallel     1       1 44
    ## 4  parallel     1       2 36
    ## 5  parallel     2       1 43
    ## 6  parallel     2       2 37
    ## 7    serial   1.1       1 44
    ## 8    serial   1.1       2 36
    ## 9    serial   1.2       1 36
    ## 10   serial   1.2       2 44
    ## 11   serial     2       1 42
    ## 12   serial     2       2 38

The table above reports how many subjects fall in each cluster, per model and (for parallel/serial) per layer/stage block. Roughly balanced counts here are expected for this simulation’s roughly 50/50 latent split; a wildly unbalanced split can be a sign that `K` is larger than the data actually supports.

`predict_lucid()`’s `pred.x` above required re-running the E-step on `G`/ `Z`. When the same fitted-model hard assignment is wanted directly, without supplying data again, `get_cluster_assignment()` reads it straight off the fitted object’s own posterior (`inclusion.p`):

    identical(as.numeric(get_cluster_assignment(fit_early_pred)),
              as.numeric(pred_early$pred.x))

    ## [1] TRUE

The two agree here because `pred_early` was computed on the same `G`/`Z` the model was fit on. `get_cluster_assignment()` is the right tool whenever the question is simply “what did this fitted model assign its subjects to,” while `predict_lucid()` is for scoring genuinely new data or getting a predicted outcome as well as an assignment.

Predicted outcomes, on the scale of the outcome that was modelled:

    outcome_summary <- function(pred_y, label) {
      v <- as.numeric(unlist(pred_y))
      data.frame(
        model  = label,
        n      = length(v),
        mean   = round(mean(v), 3),
        sd     = round(stats::sd(v), 3),
        min    = round(min(v), 3),
        median = round(stats::median(v), 3),
        max    = round(max(v), 3),
        row.names = NULL
      )
    }

    rbind(
      outcome_summary(pred_early$pred.y,    "early"),
      outcome_summary(pred_parallel$pred.y, "parallel"),
      outcome_summary(pred_serial$pred.y,   "serial")
    )

    ##      model  n  mean    sd    min median   max
    ## 1    early 80 0.511 0.702 -0.356  1.114 1.122
    ## 2 parallel 80 0.506 0.691 -0.180  0.372 1.337
    ## 3   serial 80 0.512 0.673 -0.123 -0.090 1.252

The predicted outcome is a posterior-weighted average of the cluster-specific outcome levels, so its spread is narrower than the observed outcome’s: it carries no residual variation, only the between-cluster differences. Comparing its range against `summary(d$Y_normal)` shows how much of the outcome the latent structure accounts for.

    rbind(
      outcome_summary(pred_early$pred.y, "predicted (early)"),
      outcome_summary(d$Y_normal,        "observed")
    )

    ##               model  n  mean    sd    min median   max
    ## 1 predicted (early) 80 0.511 0.702 -0.356  1.114 1.122
    ## 2          observed 80 0.504 1.186 -2.024  0.306 3.470

### 10.2 G-computation mode (early, parallel, and serial)

`g_computation = TRUE` switches to a different question: instead of scoring observed `(G, Z)` pairs, it asks what the outcome distribution would be for a given exposure profile, marginalizing over the fitted cluster and omics model – a simple causal “what if this subject’s exposure had been X” calculation built on top of the fitted parameters. This is why it needs only `G` (no `Z`, no `Y`) and why it alone returns `pred.z`, the implied omics profile under that hypothetical exposure.

    pred_early_g <- predict_lucid(
      model = fit_early_pred,
      G = d$G,
      Z = NULL,
      Y = NULL,
      g_computation = TRUE
    )

    pred_parallel_g <- try(
      predict_lucid(
        model = fit_parallel_pred,
        G = d$G,
        Z = NULL,
        Y = NULL,
        g_computation = TRUE
      ),
      silent = TRUE
    )

    names(pred_early_g)

    ## [1] "inclusion.p" "pred.x"      "pred.z"      "pred.y"

    if (inherits(pred_parallel_g, "try-error")) {
      "parallel g_computation returned a try-error on this demo object; code pattern is shown above."
    } else {
      names(pred_parallel_g)
    }

    ## [1] "inclusion.p" "pred.x"      "pred.z"      "pred.y"

    pred_serial_g <- predict_lucid(
      model = fit_serial_pred,
      G = d$G,
      Z = NULL,
      Y = NULL,
      g_computation = TRUE
    )

    names(pred_serial_g)

    ## [1] "inclusion.p" "pred.x"      "pred.z"      "pred.y"

    length(pred_serial_g$pred.z)

    ## [1] 2

### 10.3 Argument Modes: What to Pass and What You Get Back

`Z` is required for every model type. Only `g_computation = TRUE` relaxes it. `Y` is always optional; omitting it makes the prediction unsupervised.

`Z` | `Y` | `g_computation` | Result
--- | --- | --- | ---
supplied | supplied | `FALSE` | Supervised: outcome informs the posterior
supplied | omitted | `FALSE` | Unsupervised: clusters from `G` and `Z` only
omitted | either | `FALSE` | Error naming `Z` and `g_computation`
omitted | omitted | `TRUE` | Counterfactual prediction from `G` alone

The posterior is formed from the exposure, omics and outcome likelihood terms. Dropping `Y` removes one term and leaves a well-defined posterior over the rest. Dropping `Z` removes the term the clusters are defined by, leaving nothing to condition on. `g_computation = TRUE` is a separate estimator that uses only the exposure path, which is why it accepts `Z = NULL` and why it alone returns `pred.z`; supplied `Z` and `Y` are ignored in that mode.

Returned components, and their shape by model type:

Component | Meaning | early | parallel | serial
--- | --- | --- | --- | ---
`inclusion.p` | Posterior cluster probabilities | `N` x `K` matrix | list by layer | list by stage
`pred.x` | Cluster labels, `1..K` since 3.1.0 | vector | list by layer | list by stage
`pred.y` | Predicted outcome | vector | vector | vector
`pred.z` | Implied omics profile, `g_computation` only | matrix | list by layer | list by stage

Reusing the fits from 10.1, with no refitting:

    pred_unsup <- predict_lucid(
      model = fit_early_pred,
      G = d$G,
      Z = d$Z_early
    )

    missing_z <- try(
      predict_lucid(
        model = fit_early_pred,
        G = d$G,
        Z = NULL,
        Y = d$Y_normal
      ),
      silent = TRUE
    )

    data.frame(
      mode = c("Y omitted (unsupervised)", "Z omitted, no g-computation"),
      result = c(
        paste(names(pred_unsup), collapse = ", "),
        if (inherits(missing_z, "try-error")) "error, as documented" else "unexpectedly succeeded"
      )
    )

    ##                          mode                      result
    ## 1    Y omitted (unsupervised) inclusion.p, pred.x, pred.y
    ## 2 Z omitted, no g-computation        error, as documented

    cat(if (inherits(missing_z, "try-error")) attr(missing_z, "condition")$message else "")

    ## Input data 'Z' is required for prediction. Omit it only with g_computation = TRUE, which predicts from the exposures alone.

A serial model must have at least two stages to be predicted. A single-stage serial model is an equivalent early or parallel model and should be fitted as one; prediction declines it with a message to that effect.

## 11) Bootstrap Inference: `boot_lucid()`

Point estimates from `estimate_lucid()` come with no standard errors – `boot_lucid()` supplies them by nonparametric bootstrap: it resamples subjects with replacement, refits the model on each resample, and forms confidence intervals from the resulting distribution of estimates. Because that resampling-and-refitting only makes sense for stable, unpenalized estimates, `boot_lucid()` is meant to be run on a zero-penalty refit (the model whose coefficients are not shrunk by a selection penalty), not on the penalized screening fit itself – the 3-model tutorials’ screen-then-refit workflow produces exactly that kind of model. This guide bootstraps `fit_early`/`fit_parallel`/`fit_serial` directly since they were already fit at `Rho_* = 0`.

Runtime note: this section uses small `R` for tutorial speed; a real analysis needs enough bootstrap replicates (typically several hundred) for the resulting intervals to be stable.

    set.seed(106)

    boot_early <- boot_lucid(
      G = d$G,
      Z = Z_early_miss,
      Y = d$Y_normal,
      CoG = d$CoG,
      CoY = d$CoY,
      model = fit_early,
      R = 2,
      conf = 0.9
    )

    boot_parallel <- boot_lucid(
      G = d$G,
      Z = Z_parallel_miss,
      Y = d$Y_normal,
      CoG = d$CoG,
      CoY = d$CoY,
      model = fit_parallel,
      R = 2,
      conf = 0.9
    )

    boot_serial <- boot_lucid(
      G = d$G,
      Z = Z_serial_miss,
      Y = d$Y_normal,
      CoG = d$CoG,
      CoY = d$CoY,
      model = fit_serial,
      R = 2,
      conf = 0.9
    )

    summary(fit_early, boot.se = boot_early)

    ##
    ## ====================================================
    ## LUCID Early Integration: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : normal
    ##   Number of observations : 80
    ##   Number of clusters (K) : 2
    ##
    ## Missing-data profile
    ##   Listwise missing rows  : 1 / 80 (1.2%)
    ##   Sporadic missing rows  : 4 / 80 (5.0%)
    ##   Missing cells total    : 12 / 640 (1.9%)
    ##
    ## Feature selection overview
    ##   G features selected    : 6 / 6 (100.0%)
    ##   Z features selected    : 8 / 8 (100.0%)
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -650.09
    ##   BIC                    : 1751.53
    ##   Number of parameters   : 103
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Detailed parameter estimates
    ## (1) Y (continuous outcome): cluster 1 mean (intercept), mean differences for other clusters, and covariate effects
    ##                 Gamma norm_lower norm_upper sig
    ## (Intercept) 0.1765857 -0.1554741  0.3696930
    ## cluster2    1.0449481  0.3266079  2.2174532   *
    ## cov_g1      0.4929783  0.3155569  0.4159780   *
    ## cov_g2      0.6931914  0.5442827  0.9564068   *
    ##
    ## (2) Z: mean of omics data for each latent cluster
    ##                     estimate  norm_lower  norm_upper sig
    ## Z1_f1.cluster1 -0.0777814604 -1.27060679  0.78210567
    ## Z1_f2.cluster1  0.0840691217 -0.31275081  0.45248549
    ## Z1_f3.cluster1  0.0237814722 -0.66989312  0.67949470
    ## Z1_f4.cluster1  0.1617553558 -0.00526694  0.43843630
    ## Z2_f1.cluster1 -0.0021631579 -0.37326097  0.37273828
    ## Z2_f2.cluster1  0.0001776934 -0.48536139  0.66187172
    ## Z2_f3.cluster1 -0.1742793904 -0.21050324 -0.01656484   *
    ## Z2_f4.cluster1 -0.1083855071 -0.29955253  0.19077650
    ## Z1_f1.cluster2  1.3323118535  1.64221913  1.75838999   *
    ## Z1_f2.cluster2  1.1873926665  0.87516998  1.43197636   *
    ## Z1_f3.cluster2  0.9834227794  1.04521973  1.14914391   *
    ## Z1_f4.cluster2  0.5274072880  0.42942852  0.73374710   *
    ## Z2_f1.cluster2 -1.2547392707 -1.22486296 -0.99003002   *
    ## Z2_f2.cluster2 -1.1894463302 -1.54766882 -0.97333226   *
    ## Z2_f3.cluster2 -0.8918660581 -1.03894922 -0.25573482   *
    ## Z2_f4.cluster2 -0.5296838742 -0.71607080 -0.42147673   *
    ##
    ## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure
    ##                estimate norm_lower norm_upper sig
    ## G1.cluster2  1.49321813 -99.095033 43.7094324
    ## G2.cluster2 -1.12634744 -10.923986 26.6655522
    ## G3.cluster2  0.19971412 -14.716472 36.1019104
    ## G4.cluster2 -0.55799151  -8.092353 20.7389403
    ## G5.cluster2 -0.27541536  -2.929511  0.3435928
    ## G6.cluster2  0.09558619  -6.773340  2.4031418

    summary(fit_parallel, boot.se = boot_parallel)

    ##
    ## ====================================================
    ## LUCID Parallel: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : gaussian
    ##   Number of observations : 80
    ##   Clusters per layer     : 2, 2
    ##
    ## Missing-data profile by layer
    ##   Layer 1 listwise rows : 1 / 80 (1.2%)
    ##   Layer 1 sporadic rows : 0 / 80 (0.0%)
    ##   Layer 1 missing cells : 4 / 320 (1.2%)
    ##   Layer 2 listwise rows : 0 / 80 (0.0%)
    ##   Layer 2 sporadic rows : 1 / 80 (1.2%)
    ##   Layer 2 missing cells : 1 / 320 (0.3%)
    ##
    ## Feature selection overview
    ##   G features selected    : 6 / 6 (100.0%)
    ##   G features by layer
    ##     Layer 1              : 6 / 6 (100.0%)
    ##     Layer 2              : 6 / 6 (100.0%)
    ##   Z features
    ##     Layer 1 selected     : 4 / 4 (100.0%)
    ##     Layer 1 multi-cluster: 4
    ##     Layer 2 selected     : 4 / 4 (100.0%)
    ##     Layer 2 multi-cluster: 4
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -703.20
    ##   BIC                    : 1756.96
    ##   Number of parameters   : 80
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Detailed parameter estimates
    ## (1) Y (continuous outcome): intercept, effects of each non-reference latent cluster for each layer of Y (and effect of covariates if included)
    ##                  Gamma norm_lower norm_upper sig
    ## (Intercept) 0.04248756 -0.2335635  1.6167264
    ## Layer1_LC2  0.01765860 -1.3983514 -0.8411190   *
    ## Layer2_LC2  1.06757644 -1.1379911  3.3594512
    ## cov_g1      0.49345900  0.4024804  0.6025283   *
    ## cov_g2      0.65852863  0.4985023  0.9913395   *
    ##
    ## (2) Z: mean of omics data for each latent cluster of each layer
    ## Layer 1
    ##
    ##                           estimate   norm_lower norm_upper sig
    ## Layer1.Z1_f1.cluster1 -0.349168270 -2.868118306  0.7943782
    ## Layer1.Z1_f2.cluster1  0.004260061 -1.689039338  1.0268665
    ## Layer1.Z1_f3.cluster1 -0.154181517 -1.995086642  0.7518654
    ## Layer1.Z1_f4.cluster1  0.116945108 -0.471836130  0.5499805
    ## Layer1.Z1_f1.cluster2  1.239192854  0.390903419  3.8073366   *
    ## Layer1.Z1_f2.cluster2  0.993490420 -0.180375357  3.0510505
    ## Layer1.Z1_f3.cluster2  0.923900842  0.158957738  2.8085184   *
    ## Layer1.Z1_f4.cluster2  0.485051731 -0.009303011  1.2034653
    ##
    ## Layer 2
    ##
    ##                          estimate norm_lower norm_upper sig
    ## Layer2.Z2_f1.cluster1  0.21574635 -0.8937422  2.6711224
    ## Layer2.Z2_f2.cluster1  0.08358451 -0.8599423  2.0976992
    ## Layer2.Z2_f3.cluster1 -0.12054320 -0.7353785  1.1385857
    ## Layer2.Z2_f4.cluster1 -0.08469553 -0.2994629  0.6436793
    ## Layer2.Z2_f1.cluster2 -1.25626235 -3.6510921 -0.5765041   *
    ## Layer2.Z2_f2.cluster2 -1.00787017 -2.7292775 -0.1937855   *
    ## Layer2.Z2_f3.cluster2 -0.77626259 -1.7728773 -0.5921201   *
    ## Layer2.Z2_f4.cluster2 -0.48459575 -1.0609297 -0.2359717   *
    ##
    ## (3) E: intercept and odds ratio of being assigned to each latent cluster for each exposure for each layer
    ## Layer 1
    ##
    ##                         estimate norm_lower norm_upper sig
    ## (Intercept).cluster2 -0.08685128 -0.9525508  0.1498558
    ## G1.cluster2           1.20608960  0.6348916  5.8628468   *
    ## G2.cluster2          -0.92554313 -9.4325454  2.8921008
    ## G3.cluster2           0.63658188 -0.4922965  2.9857411
    ## G4.cluster2          -0.73551173 -4.3690734  0.3514864
    ## G5.cluster2          -0.04677656 -0.6017770  0.9175276
    ## G6.cluster2          -0.09518064 -1.0195382 -0.1738533   *
    ##
    ## Layer 2
    ##
    ##                         estimate  norm_lower norm_upper sig
    ## (Intercept).cluster2 -0.16662615 -0.20054088  0.1413125
    ## G1.cluster2           3.59813736  1.44079793 11.4055114   *
    ## G2.cluster2          -1.28668079 -6.40877990  3.0738119
    ## G3.cluster2           0.06073117  0.39768118  1.4758727   *
    ## G4.cluster2          -0.13557162 -0.32549805 -0.1897790   *
    ## G5.cluster2          -0.16105933 -1.03964599 -0.2352342   *
    ## G6.cluster2           0.05468855  0.05274401  0.5019673   *

    summary(fit_serial, boot.se = boot_serial)

    ##
    ## ====================================================
    ## LUCID Serial: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : normal
    ##   Number of observations : 80
    ##   Number of stages       : 2
    ##   Stage 1               : parallel (K = 2,2)
    ##   Stage 2               : early (K = 2)
    ##
    ## Missing-data profile by stage
    ##   Stage 1
    ##     Layer 1 listwise/sporadic rows : 1 / 0
    ##     Layer 2 listwise/sporadic rows : 0 / 1
    ##   Stage 2
    ##     Listwise rows         : 1
    ##     Sporadic rows         : 0
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -1006.74
    ##   BIC                    : 2499.89
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Stage-wise detailed parameter estimates
    ##
    ## --- Stage 1 (parallel) ---
    ##
    ## ====================================================
    ## LUCID Parallel: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : gaussian
    ##   Number of observations : 80
    ##   Clusters per layer     : 2, 2
    ##
    ## Missing-data profile by layer
    ##   Layer 1 listwise rows : 1 / 80 (1.2%)
    ##   Layer 1 sporadic rows : 0 / 80 (0.0%)
    ##   Layer 1 missing cells : 4 / 320 (1.2%)
    ##   Layer 2 listwise rows : 0 / 80 (0.0%)
    ##   Layer 2 sporadic rows : 1 / 80 (1.2%)
    ##   Layer 2 missing cells : 1 / 320 (0.3%)
    ##
    ## Feature selection overview
    ##   G features selected    : 6 / 6 (100.0%)
    ##   G features by layer
    ##     Layer 1              : 6 / 6 (100.0%)
    ##     Layer 2              : 6 / 6 (100.0%)
    ##   Z features
    ##     Layer 1 selected     : 4 / 4 (100.0%)
    ##     Layer 1 multi-cluster: 4
    ##     Layer 2 selected     : 4 / 4 (100.0%)
    ##     Layer 2 multi-cluster: 4
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -607.52
    ##   BIC                    : 1539.32
    ##   Number of parameters   : 74
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Detailed parameter estimates
    ## (1) Z: mean of omics data for each latent cluster of each layer
    ## Layer 1
    ##
    ##                           estimate  norm_lower  norm_upper sig
    ## Layer1.Z1_f1.cluster1 -0.352615720 -0.45377172 -0.34679383   *
    ## Layer1.Z1_f2.cluster1  0.002360677 -0.26013555  0.12172757
    ## Layer1.Z1_f3.cluster1 -0.156731519 -0.27504322  0.03886876
    ## Layer1.Z1_f4.cluster1  0.116972344 -0.07059789  0.19801677
    ## Layer1.Z1_f1.cluster2  1.229234465  1.00239553  1.24669512   *
    ## Layer1.Z1_f2.cluster2  0.986997627  0.83689718  0.96036872   *
    ## Layer1.Z1_f3.cluster2  0.917388278  0.70548033  1.00872966   *
    ## Layer1.Z1_f4.cluster2  0.481774129  0.19942295  0.44226875   *
    ##
    ## Layer 2
    ##
    ##                          estimate  norm_lower norm_upper sig
    ## Layer2.Z2_f1.cluster1 -1.24890623 -1.73171537 -0.8888058   *
    ## Layer2.Z2_f2.cluster1 -1.00104991 -1.41631230 -0.7704177   *
    ## Layer2.Z2_f3.cluster1 -0.75293355 -1.32158035 -0.3408488   *
    ## Layer2.Z2_f4.cluster1 -0.49290664 -0.61839700 -0.4914322   *
    ## Layer2.Z2_f1.cluster2  0.22761663  0.13434208  0.4651878   *
    ## Layer2.Z2_f2.cluster2  0.09162188 -0.16300910  0.2254955
    ## Layer2.Z2_f3.cluster2 -0.13230863 -0.19697712 -0.1377000   *
    ## Layer2.Z2_f4.cluster2 -0.07270329 -0.04715225  0.1038765
    ##
    ## (2) E: intercept and odds ratio of being assigned to each latent cluster for each exposure for each layer
    ## Layer 1
    ##
    ##                         estimate norm_lower norm_upper sig
    ## (Intercept).cluster2 -0.05763017 -0.3825221 0.45491231
    ## G1.cluster2           1.21618905 -6.1926124 9.64249027
    ## G2.cluster2          -0.95080523 -2.7214376 0.37461190
    ## G3.cluster2           0.67635032 -0.1060819 1.16397874
    ## G4.cluster2          -0.74690763 -1.4055136 0.21946525
    ## G5.cluster2          -0.06497218 -1.7541373 0.48710583
    ## G6.cluster2          -0.11292848 -1.0948580 0.06842552
    ##
    ## Layer 2
    ##
    ##                         estimate norm_lower norm_upper sig
    ## (Intercept).cluster2  0.10180517 -0.5130093 0.77644869
    ## G1.cluster2          -3.76996919 -0.8508330 0.08091987
    ## G2.cluster2           1.30981565  0.9150835 1.38148959   *
    ## G3.cluster2          -0.20510303 -1.8846199 1.07459406
    ## G4.cluster2           0.05465422 -2.3962603 4.88049047
    ## G5.cluster2           0.16024275 -1.5595252 2.10773916
    ## G6.cluster2          -0.05478440 -0.7339486 1.80736755
    ##
    ##
    ## --- Stage 2 (early) ---
    ##
    ## ====================================================
    ## LUCID Early Integration: Model Summary
    ## ====================================================
    ##
    ## Model specification
    ##   Family                 : normal
    ##   Number of observations : 80
    ##   Number of clusters (K) : 2
    ##
    ## Missing-data profile
    ##   Listwise missing rows  : 1 / 80 (1.2%)
    ##   Sporadic missing rows  : 0 / 80 (0.0%)
    ##   Missing cells total    : 4 / 320 (1.2%)
    ##
    ## Feature selection overview
    ##   G features selected    : 2 / 2 (100.0%)
    ##   Z features selected    : 4 / 4 (100.0%)
    ##
    ## Model fit statistics
    ##   Log-likelihood         : -399.22
    ##   BIC                    : 960.58
    ##   Number of parameters   : 37
    ##
    ## Regularization
    ##   Rho_G                  : 0.000
    ##   Rho_Z_Mu               : 0.000
    ##   Rho_Z_Cov              : 0.000
    ##
    ## Detailed parameter estimates
    ## (1) Y (continuous outcome): cluster 1 mean (intercept), mean differences for other clusters, and covariate effects
    ##                    Gamma norm_lower norm_upper sig
    ## Y.(Intercept) 0.02634618 0.04726623  0.2062116   *
    ## Y.LC2         1.06038828 0.52343619  1.0432476   *
    ## Y.cov_g1      0.44718681 0.25030684  0.7754435   *
    ## Y.cov_g2      0.65188473 0.79847213  0.9498793   *
    ##
    ## (2) Z: mean of omics data for each latent cluster
    ##                   estimate  norm_lower norm_upper sig
    ## Z3_f1.cluster1 -0.20456753 -0.61785787  0.1340400
    ## Z3_f2.cluster1  0.04858672 -0.11717991  0.2980835
    ## Z3_f3.cluster1  0.12805861 -0.02130632  0.1571153
    ## Z3_f4.cluster1 -0.05519954 -0.31251881  0.1650792
    ## Z3_f1.cluster2  0.88264257  0.66248903  1.1549322   *
    ## Z3_f2.cluster2  0.61612295  0.27766870  0.7637610   *
    ## Z3_f3.cluster2 -0.90229053 -1.12607961 -0.8493843   *
    ## Z3_f4.cluster2 -0.48813409 -0.66526079 -0.2885527   *
    ##
    ## (3) E: intercept and odds ratio of being assigned to each latent cluster for each cluster from previous serial stage
    ##                                  estimate norm_lower norm_upper sig
    ## (Intercept).cluster2             1.046768   1.666096   3.685129   *
    ## Stage1.Layer1.cluster2.cluster2  5.857494   4.095938   9.688545   *
    ## Stage1.Layer2.cluster2.cluster2 -6.549153 -13.457644  -6.318979   *

Each coefficient table above now shows a normal-theory confidence interval alongside the point estimate, plus the `sig` column described in section 9. With only `R = 2` replicates here the interval is not meaningful – this chunk exists to show the output format, not to draw real conclusions; a real bootstrap needs `R` in the hundreds.

## 12) Visualization

### 12.1 Path Diagram: `plot()`

`plot()` draws a Sankey diagram of the fitted model’s structure: flows from each exposure, through the latent cluster(s), to the outcome, with flow width proportional to the estimated association strength. It is a structural view – what connects to what, and how strongly – complementary to the omics-profile view in the next section, which shows what the clusters actually look like in the omics data.

    plot_early <- plot(fit_early)
    plot_parallel <- try(plot(fit_parallel), silent = TRUE)
    plot_serial <- try(plot(fit_serial), silent = TRUE)

    class(plot_early)

    ## [1] "sankeyNetwork" "htmlwidget"

    if (inherits(plot_parallel, "try-error")) {
      "plot(fit_parallel) returned try-error in this build (parallel plot is under development)."
    } else {
      class(plot_parallel)
    }

    ## [1] "plot(fit_parallel) returned try-error in this build (parallel plot is under development)."

    if (inherits(plot_serial, "try-error")) {
      "plot(fit_serial) returned try-error in this build (serial plot is under development)."
    } else {
      class(plot_serial)
    }

    ## [1] "plot(fit_serial) returned try-error in this build (serial plot is under development)."

### 12.2 Cluster Omics Profiles: `plot_cluster_omic_profile()`

`plot()` draws the path structure. `plot_cluster_omic_profile()` draws what the clusters actually are: which omics features separate them, and in which direction. It returns a named list of `ggplot` objects, one per omics layer, so a parallel or serial fit gives one figure per layer rather than one crowded one.

Features are ranked by `importance`. The default, `"separation"`, is the spread of the cluster means divided by the typical within-cluster spread; `"range"` and `"sd"` use the means alone. Only the standardized measure can distinguish a feature that separates the clusters from one that is merely noisy.

Argument | Effect
--- | ---
`type` | `"heatmap"` (default) or `"bar"`
`top_n` | Features per panel, default 10; a layer with fewer shows all
`importance` | `"separation"` (default), `"range"`, `"sd"`
`layer_names` | Subtitles; defaults to the names of the omics list
`layer_colors` | One hue per layer
`scale` | `TRUE` (default) fills with a per-feature z-score; `FALSE` with the cluster mean

    prof_early <- plot_cluster_omic_profile(fit_early, top_n = 8)
    names(prof_early)

    ## [1] "Omics"

    prof_early[[1]]

Read the heatmap by row: a feature whose color alternates sharply between clusters (e.g. dark for cluster 1, light for cluster 2) is one that strongly distinguishes them; a feature with similar shading across clusters is barely contributing to the separation despite appearing in the top-`n` list.

The bar rendering shows the same features and ordering, with clusters as shades of the layer’s colour:

    plot_cluster_omic_profile(fit_early, type = "bar", top_n = 8)[[1]]

A parallel fit returns one plot per layer:

    prof_parallel <- plot_cluster_omic_profile(fit_parallel, top_n = 8)
    names(prof_parallel)

    ## [1] "layer1" "layer2"

    prof_parallel[[1]]

    plot_cluster_omic_profile(fit_parallel, type = "bar", top_n = 8)[[2]]

A serial fit gives one plot per stage, and one per layer within a stage that is itself a parallel sub-model:

    prof_serial <- plot_cluster_omic_profile(fit_serial, top_n = 8)
    names(prof_serial)

    ## [1] "Stage 1 - layer1" "Stage 1 - layer2" "Stage 2"

    prof_serial[[1]]

    plot_cluster_omic_profile(fit_serial, type = "bar", top_n = 8)[[2]]

### 12.3 Extracting the Ranking Directly: `get_top_omics_features()`

The importance ranking behind every panel above is also available as plain data, without generating a plot – useful for a report table, or for feeding the top features into a downstream analysis. `get_top_omics_features()` uses the exact same ranking criterion as `plot_cluster_omic_profile()` (so its output for a given `top_n` matches that plot’s panel ordering), and, like the other extractors, auto-detects the model type and returns one named numeric vector per layer/stage.

    top_early <- get_top_omics_features(fit_early, top_n = 5)
    top_early

    ## $Omics
    ##    Z2_f2    Z1_f1    Z1_f2    Z1_f3    Z2_f1
    ## 1.487498 1.459852 1.246496 1.164370 1.099256

The names are the features, in descending order of importance score; the values are the score itself (by default, the between-cluster separation described above). This is the same top-5 that would appear in a `plot_cluster_omic_profile(fit_early, top_n = 5)` panel, just as a plain vector rather than a figure.

## 13) Binary Outcome Example (Early Model)

Everything above used a continuous outcome (`family = "normal"`). Switching to `family = "binary"` changes only the outcome model: the cluster effects on `Y` become log-odds (with odds ratios reported alongside), there is no residual variance to estimate, and `predict_lucid()`’s `response` argument lets you choose class-label or probability output. This short chunk shows that fit and a probability-scale prediction.

    set.seed(107)
    fit_early_binary <- estimate_lucid(
      lucid_model = "early",
      G = d$G,
      Z = d$Z_early,
      Y = d$Y_binary,
      CoG = d$CoG,
      CoY = d$CoY,
      family = "binary",
      K = 2,
      max_itr = 8,
      max_tot.itr = 30,
      tol = 1e-2,
      seed = 107
    )

    ## Fitting LUCID early model (K = 2)...
    ## Finished LUCID early model.

    pred_binary_prob <- predict_lucid(
      model = fit_early_binary,
      G = d$G,
      Z = d$Z_early,
      Y = d$Y_binary,
      CoG = d$CoG,
      CoY = d$CoY,
      response = FALSE
    )

    range(pred_binary_prob$pred.y)

    ## [1] 0.1039360 0.9491689

`pred.y` here is a probability in `[0, 1]` because `response = FALSE`; setting `response = TRUE` instead would return hard `0`/`1` class labels.

## 14) End Notes

- This guide focuses on breadth (all major exported APIs) while keeping runtime practical.
- For production analysis, you should usually increase:
  - `max_itr`, `max_tot.itr`
  - bootstrap `R`
  - tuning grid size in `tune_lucid()`
- For serial pipelines, keep a close eye on stage-wise missingness summaries.
- Under the hood, the EM optimizer that every fit in this guide runs on applies standard numerical-stability safeguards – computing sums of probabilities in log space to avoid underflow, regularizing near-singular covariance matrices, and monitoring the log-likelihood at every iteration to catch non-convergence – automatically, with no action needed from the caller. These are internal implementation details rather than part of the public API, so they are not demonstrated here.

## 15) Session Info

    sessionInfo()

    ## R version 4.4.0 (2024-04-24)
    ## Platform: aarch64-apple-darwin20
    ## Running under: macOS 26.6.2
    ##
    ## Matrix products: default
    ## BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib
    ## LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
    ##
    ## locale:
    ## [1] C/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
    ##
    ## time zone: America/Los_Angeles
    ## tzcode source: internal
    ##
    ## attached base packages:
    ## [1] stats     graphics  grDevices utils     datasets  methods   base
    ##
    ## other attached packages:
    ## [1] plotly_4.11.0 ggplot2_4.0.2 LUCIDus_3.2.0
    ##
    ## loaded via a namespace (and not attached):
    ##  [1] tidyr_1.3.1        sass_0.4.10        generics_0.1.4     shape_1.4.6.1
    ##  [5] stringi_1.8.7      lattice_0.22-7     hms_1.1.4          digest_0.6.39
    ##  [9] magrittr_2.0.4     evaluate_1.0.5     grid_4.4.0         RColorBrewer_1.1-3
    ## [13] iterators_1.0.14   fastmap_1.2.0      foreach_1.5.2      jsonlite_2.0.0
    ## [17] glmnet_4.1-10      Matrix_1.7-4       progress_1.2.3     nnet_7.3-20
    ## [21] survival_3.8-3     mclust_6.1.2       httr_1.4.7         purrr_1.2.0
    ## [25] crosstalk_1.2.2    viridisLite_0.4.2  scales_1.4.0       lazyeval_0.2.2
    ## [29] codetools_0.2-20   networkD3_0.4.1    jquerylib_0.1.4    cli_3.6.5
    ## [33] rlang_1.1.6        crayon_1.5.3       splines_4.4.0      withr_3.0.2
    ## [37] cachem_1.1.0       yaml_2.3.11        tools_4.4.0        dplyr_1.1.4
    ## [41] boot_1.3-32        vctrs_0.6.5        R6_2.6.1           lifecycle_1.0.4
    ## [45] htmlwidgets_1.6.4  pkgconfig_2.0.3    glasso_1.11        bslib_0.9.0
    ## [49] pillar_1.11.1      gtable_0.3.6       data.table_1.17.8  glue_1.8.0
    ## [53] Rcpp_1.1.0         tidyselect_1.2.1   xfun_0.54          tibble_3.3.0
    ## [57] data.tree_1.2.0    knitr_1.50         dichromat_2.0-0.1  farver_2.1.2
    ## [61] htmltools_0.5.9    igraph_2.2.1       labeling_0.4.3     rmarkdown_2.30
    ## [65] compiler_4.4.0     prettyunits_1.2.0  S7_0.2.1