---
title: "Generalized NDR: associations, communities, and latent scores"
author: "Zsolt T. Kosztyan, Marcell T. Kurbucz, Attila I. Katona, and Zahid Khan"
bibliography: references.bib
link-citations: true
output:
  rmarkdown::html_vignette:
    toc: true
    number_sections: true
    css: vignette.css
    self_contained: false
    lib_dir: nda-vignette-libs
vignette: >
  %\VignetteIndexEntry{Generalized NDR: associations, communities, and latent scores}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7.2,
  fig.height = 4.8,
  dpi = 96,
  fig.retina = 1,
  out.width = "100%",
  warning = FALSE,
  message = FALSE
)
library(nda)
set.seed(42)
```

<script>
document.addEventListener("DOMContentLoaded", function () {
  var toc = document.getElementById("TOC");
  if (!toc) return;
  toc.setAttribute("aria-label", "Table of contents");
  if (!toc.querySelector(".toc-title")) {
    var title = document.createElement("div");
    title.className = "toc-title";
    title.textContent = "On this page";
    toc.insertBefore(title, toc.firstChild);
  }
});
</script>

# What `ndr()` estimates

Generalized network-based dimensionality reduction has three linked stages:

1. estimate or receive an association matrix;
2. detect communities in the squared-association network;
3. construct one standardized, centrality-weighted score per retained
   community.

Squaring lets community detection use relationship strength. The unsquared
association is retained for interpretation: positive edges are green,
negative edges are red, and absolute magnitude controls width. Community
membership is encoded by node colour only; indicators use one common shape by
default. This visual grammar remains feasible when a model contains many
communities.

<div class="visual-pipeline">
<div class="pipeline-step"><strong>Association</strong>Variables or observations become nodes.</div>
<div class="pipeline-step"><strong>Community</strong>Strongly connected nodes form modules.</div>
<div class="pipeline-step"><strong>Score</strong>Each retained module defines one latent score.</div>
</div>

# Full, partial, and semipartial networks

The association method and conditioning type are independent choices.
`cor_method` supports Pearson, Spearman, Kendall, distance, Gaussian-rank, and
biweight associations. `cor_type` supports full, partial, and semipartial
forms.

```{r association-comparison}
fit_full <- ndr(
  swiss, cor_method = "pearson", cor_type = "full",
  centrality = "pagerank", seed = 42
)
fit_partial <- ndr(
  swiss, cor_method = "pearson", cor_type = "partial",
  centrality = "pagerank", seed = 42
)
fit_semipartial <- ndr(
  swiss, cor_method = "pearson", cor_type = "semipartial",
  centrality = "authority", seed = 42
)

comparison <- data.frame(
  network = c("full", "partial", "semipartial"),
  factors = c(fit_full$factors, fit_partial$factors,
              fit_semipartial$factors),
  retained = c(sum(fit_full$membership > 0),
               sum(fit_partial$membership > 0),
               sum(fit_semipartial$membership > 0)),
  directed = c(fit_full$stats$directed,
               fit_partial$stats$directed,
               fit_semipartial$stats$directed)
)
comparison
```

Partial association asks whether two indicators remain related after the
others are controlled on both sides. Semipartial association controls one side
and may be directed; hub and authority centralities can therefore answer
different questions.

# Inspecting communities and latent scores

The full Pearson fit provides a compact example of the fitted object.

```{r inspect-fit}
fit_full
summary(fit_full)
fit_full$membership
round(fit_full$loadings, 3)
round(fit_full$communality, 3)
fit_full$stats$community_sizes
```

`membership` identifies the score to which an indicator contributes.
`loadings` connect indicators to the resulting scores; `communality` describes
how well each indicator is represented. A zero membership means exclusion.

For a confirmatory illustration, a fixed two-community structure makes the
network and biplot stable across community-detection backends.

```{r fixed-membership}
visual_fit <- ndr(
  swiss,
  cor_method = "pearson",
  membership = c(1, 1, 2, 2, 2, 0),
  centrality = "eigenvector",
  seed = 42
)
data.frame(
  indicator = colnames(visual_fit$R),
  community = unname(visual_fit$membership)
)
```

# Predicting new observations

```{r prediction}
new_scores <- predict(visual_fit, swiss[1:5, , drop = FALSE])
round(new_scores, 3)
identical(predict(visual_fit), visual_fit$scores)
```

Prediction reuses the training means, scales, indicator weights, centrality
weights, latent centring, latent scaling, and rotation. Consequently, scores
for new observations remain on the fitted model's coordinate system.

# Supplying an arbitrary matrix and clustering observations

With `covar = TRUE`, `ndr()` accepts a square matrix directly. This makes the
method usable beyond correlation matrices. The example below follows the
clustering use described in @Kosztyan2023: Euclidean distances between the
standardized Swiss observations are converted to similarities.

```{r observation-clustering}
scaled_swiss <- scale(swiss)
distance_matrix <- as.matrix(dist(scaled_swiss))
similarity_matrix <- 1 - normalize(distance_matrix)
diag(similarity_matrix) <- 1

cluster_fit <- ndr(
  similarity_matrix,
  covar = TRUE,
  min_comm = 2,
  centrality = "eigenvector",
  seed = 42
)
cluster_fit$factors
table(cluster_fit$membership)
head(cluster_fit$membership, 10)
is.null(cluster_fit$scores)
```

Here the nodes are observations rather than indicators. The returned
membership is therefore an observation clustering. Since an arbitrary matrix
contains no original row-level transformation, latent scores for new rows are
not available in this mode.

```{r clustering-network, fig.cap="Observation clusters obtained from a normalized Euclidean similarity matrix."}
cluster_network <- plot(
  cluster_fit,
  cuts = 0.55,
  interactive = FALSE,
  show_weights = FALSE,
  shape = "automatic",
  vertex.label.size = 7,
  layout = "fr"
)
cluster_network
```

# Static and interactive networks

## Static association network

```{r ndr-static-network, fig.cap="NDR association network: colour denotes community, edge colour denotes sign, and width denotes absolute association."}
static_network <- plot(
  visual_fit,
  cuts = 0.3,
  interactive = FALSE,
  show_weights = TRUE,
  shape = "automatic"
)
static_network
```

All observed indicators use the same shape. The two communities differ only
by colours from the default light qualitative palette. Solid green and red
edges distinguish positive and negative
associations; their thickness is scaled between the requested minimum and
maximum widths. Each edge label has an opaque background and is displaced from
the edge so that neither the line nor another label overwrites the number.

## Interactive association network

The interactive representation uses exactly the same colour, shape, sign, and
width grammar. Nodes can be moved, while zooming, panning, selection, and
nearest-neighbour highlighting support closer inspection.

```{r ndr-interactive-network}
interactive_network <- plot(
  visual_fit,
  cuts = 0.3,
  interactive = TRUE,
  show_weights = TRUE,
  shape = "automatic"
)
interactive_network$widget
```

# Biplot

```{r ndr-biplot, fig.cap="Two-factor NDR biplot for the fixed Swiss-data communities."}
score_biplot <- biplot(
  visual_fit,
  view_3D = FALSE,
  interactive = FALSE
)
score_biplot
```

Nearby observation points have similar score profiles. Loading arrows indicate
the direction of increasing indicator values. The graph explains association
structure; the biplot explains observations and indicators in latent-score
space. Neither should be read as a causal model.

# Inference, filtering, and weighting

The optional style controls can be demonstrated on the same fitted network.
All observed indicators remain rectangles, and community uses colour only.
Shapes are supplied in association-matrix column order.

```{r custom-network, fig.cap="Explicit common node shape, custom signed colours, and bounded edge widths."}
custom_network <- plot(
  visual_fit, interactive = FALSE,
  shape = rep("rectangle", ncol(visual_fit$R)),
  positive.color = "#007A53", negative.color = "#B71C1C",
  positive.lty = 1, negative.lty = 1,
  edge.width.range = c(1, 5), show_weights = TRUE
)
custom_network
```

For a four-node graph, an individual shape vector could instead be
`c("circle", "circle", "square", "square")`; shapes are not assigned by
community. Optional `positive.lty` and `negative.lty` control line types.
Both default to 1 (solid). Widths are scaled within the displayed graph;
separate graphs require a common scale for direct width comparisons.

Association tests and bootstrap intervals can be executed on a small example
without dominating build time. The low replicate count below illustrates the
returned structure; final inference should use a larger `n_boot`.

```{r inference}
inference_fit <- ndr(
  swiss,
  cor_method = "pearson",
  test = TRUE,
  p_adjust = "BH",
  alpha = 0.10,
  bootstrap = TRUE,
  n_boot = 30,
  conf = 0.90,
  seed = 42
)
round(inference_fit$correlation_p, 3)
head(inference_fit$stats$bootstrap$lower)
head(inference_fit$stats$bootstrap$upper)
```

The principal filtering controls are `min_R`, `min_evalue`,
`min_communality`, `com_communalities`, and `min_comm`. The `weight` argument
supplies one finite score weight per indicator. These filters act at different
stages: network formation, centrality screening, representation, loading
separation, and minimum module size.

# Sparse storage and reproducible settings

```{r sparse-fit}
sparse_fit <- ndr(
  swiss,
  min_R = 0.15,
  sparsecalc = TRUE,
  seed = 42
)
inherits(sparse_fit$R, "sparseMatrix")
sparse_fit$factors
sparse_fit$stats$community_sizes
```

Sparse storage is helpful after thresholding a large network. Parallel workers
mainly benefit pairwise distance association and bootstrap calculations; small
Pearson examples usually run faster sequentially. A fixed `seed` makes the
stochastic parts reproducible.

# References

The cited literature covers the original NDA method, the generalized
association-matrix formulation, the clustering interpretation, the software
interface, and the supervised extension [@Kosztyan2022; @Kosztyan2023;
@Kosztyan2024; @Kosztyan2026Packages; @Kosztyan2026NBSEM].

::: {#refs}
:::


