This document describes what the current R code does in the child
algorithm. It is intended as a debugging aid and a long-term technical
reference. References to code use function names and step numbers rather
than line numbers, because line numbers become stale with each code
edit.
How to Read This Document
This reference documents the child algorithm cleaning steps (Child
Steps 5, 6, 7, 9, 11, 13, 15/16, 17, 19, 21, 22) plus child-specific
concepts, the cleanchild() working dataframe, and
child-only output material.
Wrapper-level material lives in
vignette("wrapper"): input format, unit
conversion, the full z-score calculation pipeline (CSD method, WHO/CDC
blending, recentering), batching, dispatch, output reassembly, and the
Child Step 2b GA correction (which runs in cleangrowth()
preprocessing).
Each algorithm step section here has:
- A summary table (scope, prior/next step, exclusion codes, code
location)
- Overview
- Key terms and variable names
- Logic and implementation
- Rationale for selected decisions
- Configurable arguments in scope
- Variables created and dropped
- Checklist findings
Sections that do not apply to a given step are marked “N/A” rather
than omitted.
This document covers, in order:
- Child-specific Key Concepts
- Z-Score Summary (brief; full mechanics in wrapper reference)
cleanchild() main algorithm structure
- Working Dataframe and Output (child-specific mechanics)
- Variable Glossary (child-specific variables)
- Output Format (child exclusion codes + CF rescue codes)
- Configurable Arguments (child-specific)
- Complete List of Steps
- Step-by-step walkthroughs (Early Child Step 13 through Child Step
22)
Key Concepts
This section covers concepts specific to the child algorithm. For
wrapper-level concepts (data retention, CSD z-scores, prematurity
correction, recentering, three growth parameters,
internal_id tiebreaking), see
vignette("wrapper") → Key Concepts (wrapper-level).
Three growth parameters (child scope): The
child algorithm cleans WEIGHTKG, HEIGHTCM (including LENGTHCM relabeled
to HEIGHTCM; see length.adjust parameter), and HEADCM.
HEADCM is cleaned through age 5 years — HC measurements with
agedays >= 1827 are marked
Exclude-Not-Cleaned (reported as
Exclude-HC-Out-of-Range) because the WHO HC z-score
reference ends at 5 years, so recentered z-scores cannot be formed
beyond that. The recentering file provides HC rcmedian.sd
through 5 years. For HC ages 24 months–5 years, the velocity check
(Child Step 17) uses only mindiff = -1.5 with no upper
bound since the WHO HC velocity reference ends at 24 months; EWMA steps
provide the primary outlier detection for this range.
Designated Other Parameter (DOP): Many
steps compare a measurement against a different growth parameter for the
same subject. The designated other parameter (DOP) for weight is height;
the DOP for height is weight; the DOP for HC is height. DOP comparisons
are used in temporary SDE resolution (Child Step 5), the moderate EWMA
step (Child Step 15 last-ext rules), final SDE resolution
(Child Step 13), and pairs/singles evaluation (Child Step 19). In Child
Step 13, DOP medians are computed for one-day SDE resolution
(cross-parameter comparison) and DOP is used as a secondary sort key for
selecting which SDE to keep.
Ordered steps: Like the adult algorithm,
the child algorithm processes steps in a specific order, removing more
extreme problems first and getting more refined later. Same-day
extraneous values (described below) are handled in repeated steps
because of bidirectional impact of same-day selection and other
exclusions. Some errors cause over- or under-detection of other errors,
so they must be excluded first. The order of steps is central to the
algorithm.
Iterations within steps: Several steps
(Evil Twins, EWMA1, EWMA2) are iterative. If more than one value is a
candidate for exclusion, they select the most extreme candidate and then
re-evaluate for further exclusions. This continues until no candidates
remain or too few values are left to evaluate. This is a central part of
the algorithm and reduces the likelihood of erroneous values leading to
exclusion of true values.
Temporary vs. permanent exclusions:
Same-day extraneous (SDE) values are multiple values for the same
parameter on the same day. growthcleanr selects the most plausible SDE
for inclusion in the final dataset. SDEs are first flagged temporarily
(Child Step 5) and later resolved permanently (Child Step 13).
Carried-forward values may be “rescued” and returned to Include status
based on z-score similarity criteria. These temporary mechanisms allow
the algorithm to refine decisions as the dataset gets cleaner.
No separate parameter pipelines: Unlike the
adult algorithm, which processes height and weight in completely
separate working dataframes, the child algorithm keeps all parameters in
a single data.table (data.df) with parameter tracked in a
param column. Child Steps that operate on a single
parameter use by = .(subjid, param) groupings or explicit
filtering. This design supports the cross-parameter comparisons (DOP)
that are central to the child algorithm.
CF rescue: The child algorithm has a
carried-forward (CF) rescue system (Child Step 6) that attempts to
distinguish genuinely repeated measurements from data-entry artifacts.
In the default cf_rescue = "standard" mode, each string of
identical values is judged on how far the carried value has drifted from
the real value it copies and how the trajectory continues past it;
cf_rescue = "none" and cf_rescue = "all"
provide all-or-nothing behavior. Rescued CFs are returned to
Include. See vignette("cf-rescue") for the
full rule.
EWMA (Exponentially Weighted Moving
Average): The central mechanism for detecting implausible
values. For each measurement, the EWMA computes a weighted average of
the subject’s other measurements for the same parameter, giving
exponentially more weight to temporally closer values. The deviation
from EWMA (dewma) indicates how far a value is from what we would
predict. Default exponent is -1.5 (child) vs. -5 (adult). With -1.5, the
weighting is less dominated by the nearest neighbor, spreading influence
across a wider time range — appropriate for pediatric data where growth
trajectories change more rapidly.
The .child_valid() function: A
critical gatekeeper that determines which rows participate in each step.
See Support Functions → .child_valid()
below for flag semantics and behavior.
Age-dependent internal_id tiebreaking: When
resolving same-day duplicates, the child algorithm uses an age-dependent
rule based on internal_id (the sequential integer assigned
in id-sorted order at session start): at birth (agedays == 0), keep the
lowest internal_id (earliest measurement, before postnatal
fluid shifts and interventions); at all other ages, keep the highest
internal_id (later measurement, which may represent a more
careful re-measurement). This differs from the adult algorithm, which
always keeps the highest internal_id. The user’s original
id column is never used as a tiebreaker; it is preserved
untouched into output.
Support Functions
Child-algorithm-internal helpers defined in
R/child_clean.R. Each entry lists purpose, callers,
input/return contract, key invariants the function assumes at entry, and
the enclosing function name as a code anchor. For wrapper-level helpers
that both algorithms share, see vignette("wrapper") →
Shared Helpers.
.child_exc(param_val, suffix)
Purpose. Generates a child exclusion code from a
reason suffix. Returns paste0("Exclude-C-", suffix).
Callers. Essentially every place in
cleanchild() that writes an Exclude-C-*
code.
Inputs and return contract.
param_val — accepted but ignored. Retained only to
avoid churning the call sites that pass it. Can safely be any value,
including an unused column reference inside a data.table j
expression.
suffix — reason code (e.g. "PIV",
"CF", "Traj", "Evil-Twins",
"Temp-Same-Day", "Unevaluable-Trajectory").
Must correspond to a level in exclude.levels or the factor
assignment silently produces NA.
- Returns a character scalar or vector (vectorised via
paste0).
Key invariants. The full canonical list of allowed
suffix values is in
vignette("exclusion-codes"). Adding a new suffix requires
also adding the corresponding code to exclude.levels in
cleangrowth().
Code location. R/child_clean.R,
function .child_exc().
get_dop(param_name)
Purpose. Scalar lookup of the designated other
parameter (DOP) for a given growth parameter.
Callers. Two sites inside cleanchild()
— in the Child Step 15/16 block (the last-ext DOP lookup)
and in the Child Step 19 pairs/singles block. Both callers pass
df$param[1] from a single-param working subset.
Inputs and return contract.
param_name — scalar character: "WEIGHTKG",
"HEIGHTCM", or "HEADCM".
- Returns a scalar character: the DOP.
WEIGHTKG -> HEIGHTCM,
HEIGHTCM -> WEIGHTKG, HEADCM -> HEIGHTCM
(no reverse mapping; HC’s DOP is height, not the other way around).
Key invariants. Scalar-only — the internal
if / else if / else chain requires a scalar logical test.
Vector input produces an R warning / error. No validation: anything
other than "WEIGHTKG" or "HEIGHTCM" falls
through to the HEADCM branch.
Code location. R/child_clean.R,
function get_dop().
identify_temp_sde(df, exclude_from_dop_ids = NULL)
Purpose. Selects the “keeper” on each
(subjid, param, agedays) group that has more than one valid
measurement, marking the rest for a temporary same-day-extraneous flag.
The keeper is the row closest to the subject-parameter
tbc.sd median (primary) and the cross-parameter DOP
tbc.sd median (secondary), with an age-dependent
internal_id tiebreaker.
Callers. Seven sites inside
cleanchild(). Only the Step 13 caller passes a non-NULL
exclude_from_dop_ids:
| Step 5 (initial) |
NULL |
First temp-SDE pass |
| Step 6 (recalc) |
NULL |
After CF identification |
| Step 7d (recalc) |
NULL |
After PIV |
| Step 9d (recalc) |
NULL |
After Evil Twins |
| Step 11 mid-loop (subset recalc) |
NULL |
After each EWMA1 iteration |
| Step 11c end-of-step |
NULL |
Full-dataset refresh |
| Step 13 |
temp_sde_ids_step13 |
Final SDE, with DOP anchor restricted |
Inputs and return contract.
df — data.table column-subset containing
id, internal_id, subjid,
param, agedays, tbc.sd,
exclude, and orig_row. Not mutated.
exclude_from_dop_ids — optional vector of
id values to remove from the DOP median calculation only
(they still contribute to the SP median).
- Returns a logical vector aligned to the caller’s row order. TRUE
marks rows the caller should flag
"Exclude-C-Temp-Same-Day"; FALSE means leave
exclude unchanged.
Key invariant. Identical same-day values are removed
earlier (Early Child Step 13 SDE-Identicals), so by the time
identify_temp_sde() runs, all same-day values in any given
(subjid, param, agedays) group are dissimilar.
Robust SP median. A single clearly-extreme value
that has not yet been excluded by a later trajectory step can drag the
(subjid, param) tbc.sd median toward it and
make the wrong same-day value win. Before computing the SP median, the
function compares each Include value to the group’s
provisional median; if exactly one Include value is
> 3 z from it, that value is dropped from the median
computation (both the primary median.spz and, in the Step
13 path, the DOP-restricted median). The dropped value still receives a
median and is still SDE-evaluated — it is only barred from
anchoring the median. far is evaluated for Include
rows only, and the “exactly one” guard means two mutually-distant values
(which cannot be disambiguated) are both kept.
Code location. R/child_clean.R,
function identify_temp_sde().
calc_otl_evil_twins(df)
Purpose. Marks each row with otl = TRUE
if it is the inner member (B or C) of an anchored over-the-limit
pair. “OTL” stands for “over the limit.” See the Evil Twins
(Child Step 9) → OTL calculation section below for the full
anchored-pair walkthrough; this entry is the support-function
summary.
Callers. One site: Child Step 9 (Evil Twins) inside
cleanchild().
Inputs and return contract.
df — data.table with subjid,
param, tbc.sd, ctbc.sd at
minimum, sorted by
(subjid, param, agedays, internal_id).
- Returns
df with an added otl boolean
column. On nrow(df) < 4L inputs (too few rows to form an
A,B,C,D group) it sets otl = FALSE for all rows and returns
early.
Logic. Within four consecutive eligible rows A, B,
C, D (in sort order), the inner pair (B, C) is over the limit
when its jump exceeds 5 on both tbc.sd and
ctbc.sd; it is anchored when A holds B to within 3
on either scale and D holds C to within 3 on
either scale. When both hold, B and C are flagged OTL;
the outer anchors A and D are not. A missing flank (no A before B, or no
D after C) disqualifies the pair, so single end-spikes and singles
embedded between normals are ceded to the EWMA steps. Implemented with
shift(..., by = .(subjid, param)) (one- and two-away
lag/lead of tbc.sd/ctbc.sd); the per-SP
shift() returns NA at subject-parameter
boundaries, so comparisons never cross SPs.
Key invariants. Adjacent-pair comparison assumes the
caller’s sort is (subjid, param, agedays, internal_id) —
neighbors in row order are neighbors in age within the same
subject-parameter. The jump threshold is strict > 5 on
both tbc.sd and ctbc.sd; the
anchor threshold is <= 3 on either
scale (lenient). Requiring both scales to jump prevents a false positive
when GA correction moves only one of the two z-scores for a potcorr
subject.
Code location. R/child_clean.R,
function calc_otl_evil_twins().
ewma(agedays, z, ewma.exp, ewma.adjacent = TRUE, window = 15, cache_env = NULL)
Purpose. Core exponentially-weighted moving-average
helper. For each observation i, computes a weighted average of all other
observations in the group, with weights
(5 + |agedays_i - agedays_j|) ^ ewma.exp_i and
w_ii = 0. Returns three variants (ewma.all,
ewma.before, ewma.after) that exclude
different subsets of the group. It operates on a single
z-score vector per call; the corrected score is obtained by a
second ewma() call on ctbc.sd
(the two calls share a delta matrix via cache_env).
Callers. Three direct call sites in
cleanchild():
- Child Step 11 (EWMA1 Extreme) — iterative; calls
ewma()
directly inside a per-group closure, once on tbc.sd and
once on ctbc.sd, passing a shared cache_env so
the two calls reuse one age-gap delta matrix. (This is
not the incremental ewma_cache_*
API.)
- Child Step 13 — final SDE resolution (
ewma_df).
- Child Step 17 — height/HC velocity tiebreaker for 3+-measurement
groups (recomputed from scratch each iteration, not cached).
Child Step 15/16 (EWMA2 Moderate) does not call
ewma() directly — it uses the incremental
ewma_cache_init() / ewma_cache_update()
machinery instead (see next entry).
Inputs and return contract.
agedays — integer vector.
z — numeric z-score vector, aligned with
agedays (callers pass tbc.sd on the first call
and ctbc.sd on the second).
ewma.exp — scalar or per-observation vector. All
in-algorithm callers pass a per-observation vector that varies by widest
neighbor age gap.
ewma.adjacent — if FALSE, only ewma.all is
returned (one-element list).
window — max Include observations on each side to
include; Inf disables windowing.
cache_env — optional environment; when supplied, the
windowed age-gap delta matrix is computed once and reused across paired
calls on the same agedays + exponents (e.g. the
tbc.sd then ctbc.sd calls in Step 11).
- Returns a named list of numeric vectors (not a
data.frame). Single-element when
ewma.adjacent = FALSE.
Key invariants. Self-weight is zero, so observation
i is excluded from its own EWMA. Groups with n <= 2
follow first/last-row conventions documented inline. The +5
constant prevents infinite weight for zero-gap pairs; the negative
ewma.exp makes weight decay with gap.
Code location. R/child_clean.R,
function ewma().
ewma_cache_init(...) /
ewma_cache_update(...)
Purpose. Incremental EWMA machinery for the
iterative EWMA2 loops (Child Steps 15/16).
ewma_cache_init() builds a per-group weight matrix and
initial cached weighted sums; ewma_cache_update() removes
an excluded observation from the cache in O(n) arithmetic without
rebuilding the matrix. (EWMA1 — Child Step 11 — does
not use this API; it calls ewma() directly
with a shared cache_env delta matrix, see previous
entry.)
Callers. Used by the EWMA2 (Child Step 15 and Child
Step 16) iteration loops in cleanchild(); not called by any
other step.
Inputs and return contract.
ewma_cache_init() — takes the same arguments as
ewma() plus initial exclude state. Returns an
environment with the weight matrix, cached row sums, and cached weighted
sums per variant.
ewma_cache_update() — takes the environment plus the
position of the newly-excluded observation; mutates the environment in
place to reflect the removal, checking up to 2 positions on each side
for max-gap shifts.
- The child
ewma() function reads from the environment
when supplied with cache_env.
Code location. R/child_clean.R,
functions ewma_cache_init() and
ewma_cache_update().
as_matrix_delta(agedays)
Purpose. Internal helper that builds an
n × n matrix of pairwise absolute age differences from an
agedays vector. Used by ewma() and
ewma_cache_init() for building the age-gap weight
matrix.
Callers. Two sites, both in the EWMA machinery
above.
Inputs and return contract. Takes a numeric vector;
returns a square numeric matrix. No edge-case guards — a zero-length
input returns a 0 × 0 matrix.
Code location. R/child_clean.R,
function as_matrix_delta().
calc_and_recenter_z_scores(df, cn, ref.data.path, measurement.to.z = NULL, measurement.to.z_who = NULL)
Purpose. Recalculates recentered z-scores for a
perturbed measurement column (p_plus /
p_minus). Applies the same WHO/CDC blending window and
NA-fallback logic as the main cleangrowth() z-score
calculation, then subtracts the precomputed rcmedian.sd.
The perturbed z-scores (tbc.p_plus,
tbc.p_minus) are computed once for all rows in
cleangrowth() preprocessing (the recentering
block) and ride in on data.df, so the trajectory steps that
consume them (the Child Step 11 and Child Step 15/16 plus/minus checks)
read them directly without rebuilding per batch.
Callers. Two sites in cleangrowth()
preprocessing — in the recentering block, right after
tbc.sd/ctbc.sd are formed — once with
cn = "p_plus" and once with cn = "p_minus".
Both callers pass pre-built measurement.to.z /
measurement.to.z_who closures to avoid repeat disk reads.
(The NULL-closure fallback in the function body exists only for direct /
debugging use.)
Inputs and return contract.
df — data.table with param,
agedays, sex, rcmedian.sd, and
the measurement column named by cn. Modified in place:
intermediate cn.orig_cdc, cn.orig_who,
cn.orig, and a new tbc.<cn> column are
added.
cn — character scalar column name.
ref.data.path — only consulted when the closures are
NULL.
- Returns
df with the new tbc.<cn>
column. Intermediate cn.orig* columns are left on
df but are not consumed downstream.
Key invariants. rcmedian.sd must
already be populated by the main recentering step; the helper does not
re-merge the recentering file.
Code location. R/child_clean.R,
function calc_and_recenter_z_scores(). For the underlying
z-score mechanics (CSD formula, blending boundaries, NA-fallback logic),
see vignette("wrapper") → Z-Score Infrastructure.
.compute_corrected_z(dt, meas, measurement.to.z, measurement.to.z_who)
Purpose. The Child Step 2b gestational-age
corrected-z engine. Computes the corrected,
not-yet-recentered CSD z-score for an arbitrary measurement
vector, holding the GA-correction parameters (cagedays,
reference M/S_upper/S_lower)
fixed at the values derived from the subject’s actual first weight, so
the same correction is applied to the real measurement
and to the ±5%-wt / ±1 cm plus/minus perturbations. Three components
combined by age: (1) the uncorrected WHO/CDC blend at chronological age
(mirrors the main z blend; computed by the shared helper
.gc_uncorrected_z(), which cleangrowth() also
uses to rescore the GA-window rows), (2) the gestational-age reference
CSD z ((v_fenton − M)/(S_side · M), with WT scaled ×1000 to
grams) at gestational-corrected age, and (3) the corrected WHO/CDC z at
cagedays. The standing/recumbent offset
(.gc_len_offset, 0.7 cm for both WHO and CDC) is added to
HT for the corrected WHO/CDC z when
agedays >= 730 & cagedays < 730
(.gc_ht_start_day, the day the references switch to
height). HC uses WHO at all ages; HT/WT blend WHO→CDC over ages 2–5. The
gestational-age reference z-score is preferred over corrected-WHO for
ages ≤2 (when available). The final corrected z: ≤2y uses
sd.c; 2–4y blends sd.c with the uncorrected
sd.orig_m; else uncorrected. Returns the corrected z
vector; subtract rcmedian.sd downstream to get
ctbc.
Inputs. dt must carry
param, agedays, sex,
ageyears_2b, cagedays, M,
S_upper, S_lower, potcorr;
meas is the measurement vector to z-score (same
length/order as dt); the two closures are the CDC-only and
WHO-blended read_anthro() closures.
Callers. cleangrowth() Step 2b
preprocessing — once each for the actual value
(v + lin_off), p_plus, and
p_minus, and again for the GA-window rows after their
default adjustment (vignette("wrapper"), Phase 8, fact 6).
The standing/recumbent offset it applies in the GA window is
.gc_len_offset (0.7 cm), shared with
length.adjust.
Code location. R/child_clean.R,
function .compute_corrected_z(). For the GA-correction
narrative (potcorr selection, the corroboration gate), see “Gestational
Age Correction (Child Step 2b)” below and
vignette("wrapper").
.gc_uncorrected_z(param, agedays, sex, meas, measurement.to.z, measurement.to.z_who)
Purpose. The uncorrected CSD z-score at
chronological age with the WHO/CDC age blend (WHO under 2 years and for
HEADCM, CDC over 5 years, linear blend from 2 to 5 years for weight and
height, falling back to whichever reference is available). Returns
sd.orig_who, sd.orig_cdc, and the blended
sd.orig.
Callers. .compute_corrected_z() (the
uncorrected side of its 2–4 year blend) and the Child Step 2b GA-window
adjustment, which rescores window rows after subtracting the offset.
Code location. R/child_clean.R,
function .gc_uncorrected_z(). The main z-score calculation
in cleangrowth() computes the same blend inline.
.child_catchup_protect(agedays, tbc.sd, ctbc.sd)
Purpose. Given one (subjid, param)
group’s vectors sorted by agedays, returns a logical vector marking
values to PROTECT from trajectory exclusion because they are members of
an early preterm catch-up increase. A consecutive Include pair (earlier
→ later) qualifies when the earlier member is younger than 3 months, the
z rises from the earlier to the later member, and the rise stays within
allowed = min(4, 2 + 0.5 × ceiling(max(0, interval_months − 1)))
on at least one of the tbc / ctbc frames (2 z
for the first month, +0.5 z per full-or-partial month after, capped at
4). Both members of a qualifying pair are marked; a value is then kept
only if its own tbc.sd <= -1.5 (genuinely low), since
above that gate the corrected frame is ignored.
Callers. Child Steps 11, 15, and 16, applied only to
GA-corrected subjects (gated on ctbc.sd != tbc.sd somewhere
in the group), after candidate marking and before worst-value selection.
There is no separate corroboration gate here: GA correction itself is
now corroboration-gated (see Child Step 2b), so only
genuinely-corroborated preterms reach this protection. See “Preterm
catch-up protection” under EWMA2 (Child Steps 15/16).
Contract. Pure function of the three input vectors;
returns a logical vector aligned to the input rows. It can only remove
values from candidacy, never add an exclusion.
Cross-references to wrapper helpers
The child algorithm also uses several wrapper-level helpers; their
detailed treatment lives in vignette("wrapper") → Shared
Helpers:
read_anthro(path, cdc.only) — builds CDC / WHO closures
from inst/extdata/ reference tables. Called once per
closure at the top of cleangrowth() (and as a fallback
inside calc_and_recenter_z_scores() when closures are not
supplied).
gc_preload_refs(path) — pre-loads both closures as a
list for repeated cleangrowth() calls (~0.9 s saved per
call).
Variable Glossary
For input/identification variables (line,
id, internal_id, subjid,
param, agedays, sex,
v, v.orig), z-score variables
(sd.orig*, sd.corr, rcmedian.sd,
tbc.sd, ctbc.sd), and prematurity correction
variables (potcorr, potcorr_wt,
uncorr, fengadays, pmagedays,
cagedays), see vignette("wrapper") → Variable
Glossary (wrapper-level).
This section covers child-specific variables only.
Exclusion and status variables
exclude |
factor |
Preprocessing |
Exclusion code or “Include” |
cf_rescued |
character |
Child Step 6 |
CF rescue reason; empty string for non-rescued rows |
nnte |
logical |
Preprocessing |
Always FALSE in current code. Retained as a placeholder and has no
effect on results. |
Carried-forward variables (Child Step 6)
All of these are internal Child Step 6 working columns; all are
dropped before Child Step 6 exits (via cols_to_drop_6 or
earlier cleanup).
v.orig |
numeric |
Copy of the raw measurement (created in Child Step 5, used in Child
Step 6 detection). |
cf_binary |
logical |
TRUE if the row is currently Exclude-C-CF. Temp,
cleaned mid-step. |
originator |
logical |
TRUE for an Include whose next row (subject/param/ageday sort order)
is a CF. Temp, cleaned mid-step. |
originator_seq |
integer |
cumsum(originator) per subject-param. Temp, cleaned
mid-step. |
cf_string_num / cs |
integer |
Sequential string number per subject-param. cs is the
alias retained through the rescue phase. |
originator_z |
numeric |
The originator’s sd.orig z-score, propagated forward to
its CFs. Temp, cleaned mid-step. |
seq_win |
integer |
Position within a CF string: 0 for originator,
1, 2, 3, ... for subsequent CFs. |
absdiff |
numeric |
|sd.orig − originator_z| for each CF. |
ageday_has_include |
logical |
TRUE if any row on the same (subjid, param, agedays) has
exclude == "Include". Restricts CF rescue eligibility (not
originator assignment). |
orig_ageday |
integer |
The originator’s agedays, propagated forward to its CFs. Used to
compute the cf_dage_orig design-aid column under
cf_detail. |
EWMA variables (internal, not in output)
ewma.all |
numeric |
EWMA excluding only the current observation |
ewma.before |
numeric |
EWMA excluding current + prior observation |
ewma.after |
numeric |
EWMA excluding current + next observation |
dewma.all |
numeric |
tbc.sd - ewma.all |
dewma.before |
numeric |
tbc.sd - ewma.before |
dewma.after |
numeric |
tbc.sd - ewma.after |
Gestational Age Correction (Child Step 2b, preprocessing)
Child Step 2b runs in cleangrowth() preprocessing,
before dispatch to cleanchild(). Full walkthrough — potcorr
identification, reference merge, correction failure handling, and the
uncorr revert check — lives in vignette("wrapper") →
Gestational Age Correction (Child Step 2b, preprocessing).
In brief: subjects whose first weight is consistent with prematurity
or small-for-gestational age (SGA) are flagged potcorr and
receive an assigned gestational age for cleaning purposes. Their
z-scores are recomputed against gestational-age reference values at
corrected age, with corrected WHO/CDC fallback above 2 years and a
smoothed corrected→original blend over 2–4 years. The result is
sd.corr, which downstream steps read through
ctbc.sd. By default (length.adjust = FALSE),
Step 2b also presumes recumbent length for height rows of a corrected
subject whose chronological age is 2 years or older but whose corrected
age is under 2 years, so it also rescores those rows’ uncorrected
z-scores (sd.orig); see vignette("wrapper") →
“Phase 4b: GA-window recumbent presumption”. The gestational-age
reference values and their source are described in
vignette("wrapper") → “Gestational-age reference
values”.
Corroboration gate. A potcorr flag from
a single low first weight is only trusted (correction kept) if true
prematurity is corroborated by EITHER form 1 — a
consistent early weight trajectory (the uncorr revert check: ≥2 weights
< 2y whose corrected early spread is no worse than uncorrected), OR
form 2 — a same-day height or head
circumference whose own uncorrected z is <= -1.5
and no more than 3.5 z above the first
weight’s uncorrected z (short, or small-headed, as well as light; the
cap stops a merely-somewhat-low HT/HC from corroborating a far-lower
weight). A subject corroborated by neither is reverted to uncorrected
(uncorr = 1, sd.corr = sd.orig). This
withdraws the false correction from a misplaced-decimal birth weight
(low weight but a normal same-day length and no consistent early
follow-up) so the implausible value is no longer hidden — it is then
caught by the trajectory steps. Form 2 also keeps the correction for
real short/small-headed preterms that the form-1 check alone
over-reverted. Because form 1, height, and HC are ORed, HC only decides
the outcome when there is no early-weight corroboration and no
confirming height.
SDE Resolution (Child Step 13, including Early 13)
| Scope |
All parameters |
| Exclusion codes |
Exclude-C-Identical,
Exclude-C-Extraneous |
| Code location |
Early Child Step 13 is inline at the top of
cleanchild() in child_clean.R. Main Child Step
13 is later in cleanchild() and reuses
identify_temp_sde() (defined in
child_clean.R). |
Overview
Same-day extraneous (SDE) resolution runs in two phases:
- Early Child Step 13 (before Child Steps 5/6):
Removes same-day identical values so they don’t inflate CF string
detection
- Main Child Step 13 (after Child Step 11):
Permanently resolves all remaining same-day duplicates using EWMA-based
selection
SDE-Identicals (Early Child Step 13)
| Prior step |
Preprocessing |
| Next step |
Child Step 5 (Temporary SDEs) |
Runs on data.df before any other cleaning step.
Logic:
- Sort by
subjid, param, agedays, internal_id
- For each
(subjid, param, agedays, v) group — same
subject, parameter, day, AND measurement value — count Include rows
- If count > 1, select one to keep via age-dependent
internal_id rule (birth: lowest
internal_id; other ages: highest
internal_id)
- Others →
Exclude-C-Identical
Handles partial identicals: [10.5, 10.5, 11.2] excludes one 10.5 but
leaves 11.2 untouched.
Main Child Step 13
| Prior step |
Child Step 11 (EWMA1) |
| Next step |
Child Step 15 (EWMA2) |
After Child Steps 5–11 have removed PIVs, Evil Twins, extreme EWMA
outliers, and carried forwards, Child Step 13 permanently resolves all
remaining same-day duplicates. It works on a subset
(data.sde) containing only subjects with same-day
measurements, then merges results back.
Phase A: Setup
- Capture temp SDE ids for DOP median exclusion
(
temp_sde_ids_step13)
- Safety checks: warn if Include duplicates exist (these would
indicate a bug in prior steps)
- Re-run
identify_temp_sde() with
exclude_from_dop_ids — in Child Step 13, temp SDEs are
excluded from DOP median calculation (unlike Child Step 5 where all
values contribute)
- Pre-filter to subjects with same-day measurements
Before Phase B proper, a defensive assertion checks that every
(subjid, param, agedays) group in data.sde already has
distinct values — Early Child Step 13 (before Child Steps 5/6) resolves
every same-day same-value pair among Include rows, v is
never reassigned afterwards in the child path, and no row re-enters this
valid set except Exclude-C-Temp-Same-Day → Include
(already-Include at the early pass). Exclude-C-Identical is
therefore assigned in exactly one place (Early Child Step 13); the
assertion stop()s if the invariant is ever violated, rather
than silently re-deriving the same answer.
Phase B1: One-Day SDE + SDE-All-Extreme
For subject-params with data on only ONE unique day
(one_day_sde_flag):
SDE-All-Extreme: If the closest value to the
day’s median is still > 2 SD away, ALL values on that
day are marked Exclude-C-Extraneous. Threshold: strict
> 2.
DOP medians: For one-day SDEs, cross-parameter
medians provide a secondary sort key. DOP mapping: WT↔︎HT, HC→HT. Uses
only fully included values (!was_temp_sde).
One-Day selection: Sort by:
absdiff_rel_to_median ascending
absdiff_dop_for_sort ascending (Inf if NA)
- Age-dependent
internal_id tiebreaker. Keep first;
others → Exclude-C-Extraneous
Phase B2: SDE-EWMA resolution
For subject-params with data on multiple days:
- Compute EWMA from fully included non-temp-SDE values only.
Robust drop: a lone far value
(
exclude == "Include", > 3 z from the SP
provisional median, with n_far == 1 for the subject-param)
is also excluded from this EWMA, mirroring the robust median drop in
identify_temp_sde. This stops a lingering end-spike extreme
— one the Evil Twins anchor rule cedes to Traj, so it is still Include
at Step 13 — from inflating spa_ewma at an adjacent SDE day
and falsely tripping SDE-All-Extreme into excluding
both members of a plausible same-day pair.
- The EWMA is computed on both the uncorrected z (
tbc.sd
→ ewma.all) and the GA-corrected z (ctbc.sd →
c.ewma.all), using the same rows and exponents. For
subjects without GA correction, ctbc.sd == tbc.sd, so
c.ewma.all == ewma.all.
- Assign EWMA values to temp SDEs by copying the
ewma.all
(and c.ewma.all) from their non-SDE same-day
counterpart
spa_ewma = max(ewma.all) and
c.spa_ewma = max(c.ewma.all) per
(subjid, param, agedays) — used as the reference
points
absdewma = |tbc.sd - spa_ewma| and
c.absdewma = |ctbc.sd - c.spa_ewma| for each row
- Corrected date (
corr_date): a
(subjid, param, agedays) group where any eligible value has
ctbc.sd != tbc.sd, i.e. the same-day values are themselves
GA-corrected. Only corrected dates use the corrected frame in the next
two steps; every other date (children without GA correction, and
GA-corrected children once their values are no longer corrected) uses
the uncorrected frame only.
- SDE-All-Extreme: If
min_absdewma > 1 for a group with 2+ eligible values —
and, on a corrected date, also c.min_absdewma > 1 (or
the corrected value is missing) — all →
Exclude-C-Extraneous. Threshold: strict
> 1.
- SDE-EWMA selection: On a corrected date, sort
eligible values by
c.absdewma ascending, then
absdewma ascending, then the age-dependent
internal_id tiebreaker. On any other date, sort by
absdewma ascending, then the tiebreaker. Keep first →
Include; others → Exclude-C-Extraneous
Phase B3: Merge back
SDE results are merged back to data.df by
internal_id
(data.df <- merge(data.df, sde_results, by = "internal_id", all.x = TRUE)).
Only rows that were Include or Temp-SDE are overwritten — permanent
exclusions from prior steps (PIV, Evil Twins, etc.) are preserved. Extra
columns from SDE processing are dropped.
Rationale
- EWMA-based tiebreaking for multi-day subject-params
uses the trajectory context built up by the other cleaning steps. By the
time Main Child Step 13 runs, extreme outliers (Child Steps 7, 9, 11)
have been excluded, so the remaining EWMA is a reasonable proxy for the
subject’s true trajectory.
- One-day-only subject-params get a cross-parameter check
(DOP) because without multiple days there is no trajectory to
anchor selection; using the other parameter’s value (e.g., height for
weight) adds a sanity check that the kept SDE is consistent with the
child’s overall size.
- SDE-All-Extreme thresholds (
> 2 and
> 1) are strict to avoid excluding entire SPAs
when a single plausible value is within reach of the trajectory; the
asymmetry between the one-day (> 2) and EWMA-based
(> 1) thresholds reflects the weaker trajectory
information available for one-day SPAs.
- Both frames for GA-corrected subjects. For a
GA-corrected preterm infant, uncorrected z rises steeply during
catch-up, and the uncorrected EWMA lags behind because early NICU values
are still in its window. A correct same-day value can then sit more than
1 above the uncorrected EWMA. Requiring both frames to be extreme before
SDE-All-Extreme fires matches the both-frames requirement in Child Steps
11 and 15/16. The keeper is chosen on the corrected frame because the
lagging uncorrected EWMA would favor a wrong value that happens to sit
just below it. The corrected frame is used only on dates whose values
are themselves corrected: once a child’s values are no longer corrected,
the corrected EWMA can still be shifted by earlier corrected values in
its window, so those dates keep the uncorrected-frame rule.
- Two-phase SDE resolution (Early 13 → Child Step 5 → Main
13) separates the cheap, unambiguous case (same-day identicals)
from the harder case (same-day non-identicals) so that CF string
detection does not confuse identical SDEs with carried-forward
strings.
Configurable arguments in scope for Child Step 13
None. The SDE-All-Extreme thresholds (> 2 one-day and
> 1 EWMA-based) are hardcoded.
Variables created and dropped
All Main Child Step 13 working variables live on the
data.sde subset (or on ewma_df, which is
merged back into data.sde). They are not carried back to
data.df at merge time: the Phase B3 block (the
setdiff(names(data.df), keep_cols_sde) drop) drops every
column not in keep_cols_sde (captured as
keep_cols_sde <- names(data.df) before Phase B
construction). The Phase A defensive assertion (dup_check)
is dropped inline, before Phase B.
- Phase B1 (one-day):
was_temp_sde,
n_days_with_data, has_sde_day,
one_day_sde_flag, median_tbc,
absdiff_rel_to_median,
min_absdiff_rel_to_median, HT_dop_med,
WT_dop_med, HC_dop_med,
absdiff_dop_med, absdiff_dop_for_sort,
tiebreaker_oneday, keep_id_oneday.
- Phase B2 (EWMA):
ewma_df scratch
columns (diff_before, diff_after,
maxdiff, ageyears, exp_val,
ewma.all, ewma.before,
ewma.after, c.ewma.all), then on
data.sde after the merge: ewma.all,
ewma.before, ewma.after,
c.ewma.all, spa_ewma, c.spa_ewma,
absdewma, c.absdewma,
n_available, min_absdewma,
c.min_absdewma, corr_date,
tiebreaker_ewma, keep_id_ewma.
(ewma_fill and c_ewma_fill are created and
dropped inline.)
Inside the separately-called identify_temp_sde()
(invoked from Phase A), the scratch columns median.spz,
median.dopz, absdmedian.spz,
absdmedian.dopz, extraneous.this.day,
extraneous, and orig_row live on a copy of the
input and are discarded when the helper returns — they never reach
data.sde or data.df.
No persistent columns are added to data.df by Step
13.
Checklist findings
- DOP mapping consistent: All three Phase B1 DOP
median branches (
HT_dop_med, WT_dop_med,
HC_dop_med) follow the canonical mapping — WT↔︎HT is
symmetric, HC→HT is asymmetric (HC uses HT as its anchor; HT does not
use HC).
- Boundaries: SDE-All-Extreme
> 2
(one-day) and > 1 (EWMA-based) — both strict.
Age-dependent id tiebreaker uses internal_id (not the
user’s id) in Child Step 13.
.child_valid() calls: Temp SDEs
included when building the data.sde subset. EWMA computed
on Include-only rows (excludes temp SDEs). DOP medians exclude temp SDEs
via !was_temp_sde.
- Merge safety: Only Include and Temp-SDE rows are
overwritten on merge-back — permanent exclusions from earlier steps are
preserved.
- Parameter scope: All three params are handled. DOP
has a param-specific mapping (see
get_dop()).
- Factor levels: All SDE exclusion codes
(
Exclude-C-Identical, Exclude-C-Extraneous)
exist in exclude.levels.
Temporary SDE Resolution (Child Step 5)
| Scope |
All parameters |
| Operates on |
Valid rows (via .child_valid(), includes temp
SDEs) |
| Prior step |
Early Child Step 13 (SDE-Identicals) |
| Next step |
Child Step 6 (Carried Forwards) |
| Exclusion code |
Exclude-C-Temp-Same-Day |
| Code location |
Inline in cleanchild() in child_clean.R;
dispatches to identify_temp_sde() (defined in
child_clean.R) without the Step-13
exclude_from_dop_ids argument. |
Overview
After identical same-day values have been removed, remaining same-day
duplicates (SDEs with different values) need temporary
resolution. Child Step 5 selects the most plausible value for each
same-day group and temporarily excludes the others. These temporary
exclusions are revisited in Child Step 13 (final SDE resolution) after
the dataset has been further cleaned.
The function identify_temp_sde() is also reused in Child
Step 13 with different parameters (see Child Step 13).
Key terms
| SP median |
Subject-parameter median: median tbc.sd for a given
subject and parameter, across all valid values |
| DOP median |
Designated-other-parameter median: the SP median of the other
parameter for the same subject (WT↔︎HT, HC→HT) |
absdmedian.spz |
Absolute distance from tbc.sd to SP median |
absdmedian.dopz |
Absolute distance from tbc.sd to DOP median |
Logic
Input: A data.table with columns
id, internal_id, subjid, param, agedays, tbc.sd, exclude.
Child Step 5a — Identify SDE days: For each
(subjid, param, agedays) group, if more than one valid row
exists, all rows in the group are flagged as
extraneous.this.day = TRUE.
Child Step 5b — Compute SP medians: For each
(subjid, param), compute median.spz = median
of tbc.sd across ALL valid values (not just non-SDE days).
If exactly one Include value in the group sits
> 3 z from the provisional median, it is dropped from
this computation (robust SP median — see
identify_temp_sde() in Support Functions); the same
carve-out applies to the DOP median in the Step 13 path.
Child Step 5c — Compute DOP medians: For each
parameter, look up the SP median of its designated other parameter for
the same subject:
- WEIGHTKG → uses HEIGHTCM median
- HEIGHTCM → uses WEIGHTKG median
- HEADCM → uses HEIGHTCM median
In Child Step 5, all valid values contribute to DOP medians. In Child
Step 13 (when this function is reused), temp SDEs are excluded from DOP
median calculation via the exclude_from_dop_ids
parameter.
Child Step 5d — Compute distances: For each SDE-day
row:
absdmedian.spz = |tbc.sd - median.spz|
absdmedian.dopz = |tbc.sd - median.dopz|
Fallback logic when medians are unavailable:
- If SP median is NA but DOP median exists: use DOP median as the
primary distance
(
absdmedian.spz = |tbc.sd - median.dopz|)
- If both medians are NA: use
|tbc.sd| (treat 0 as
median)
- If DOP median is NA: set
absdmedian.dopz = Inf (sorts
last)
Child Step 5e — Select value to keep: Within each
(subjid, param, agedays) group, sort by:
absdmedian.spz ascending (closest to SP median
first)
absdmedian.dopz ascending (closest to DOP median as
tiebreaker)
- Age-dependent
internal_id tiebreaker:
- At birth (
agedays == 0): internal_id
ascending (keep lowest)
- At other ages:
internal_id descending (keep
highest)
The first value after sorting is kept; all others are marked
extraneous = TRUE.
Child Step 5f — Return result: The function returns
a logical vector in the original row order (not the
keyby-sorted order), indicating which rows should be
excluded. The caller sets these rows to
Exclude-C-Temp-Same-Day.
Rationale
- SP median as primary: The value closest to the
subject’s own trajectory (as captured by their median z-score) is most
likely correct.
- DOP median as tiebreaker: When two values are
equally close to the SP median, the one whose z-score is also closer to
the other parameter’s median is preferred. This cross-parameter check
adds robustness.
- Temporary resolution: This is intentionally
conservative — the dataset still contains errors that could distort
medians. Child Step 13 re-resolves SDEs after PIV, Evil Twins, and EWMA1
have removed more extreme values.
Safety check
After Child Step 5, the code verifies that no
(subjid, param, agedays) group has more than one
Include row. If duplicates are found, a warning is issued
(indicates a bug in identify_temp_sde()).
Configurable arguments in scope for Child Step 5
None. Selection uses SP median → DOP median → age-dependent
internal_id tiebreaker — all hardcoded.
Variables created and dropped
v.orig (created): A copy of the measurement values
before any transformations. Created just before Child Step 5 runs, used
in Child Step 6 CF detection (exact value comparison) and the
imperial design-aid flag. Dropped after Child Step 6.
subj.dup (created): List of subject IDs that have at
least one temporary SDE. Used for efficiency in later steps that only
need to process SDE subjects.
Checklist findings
- DOP mapping consistent: WT↔︎HT, HC→HT, same mapping
used in Child Step 13.
- Boundaries: Ties broken by age-dependent
internal_id (lowest at birth, highest otherwise).
.child_valid() call: Includes temp
SDEs (the candidates being resolved).
- Parameter scope: All three params handled.
- Factor level:
Exclude-C-Temp-Same-Day
exists in exclude.levels.
- Safety check: Post-step verifier has never fired in
testing.
PIV: Population Implausible Values (Child Step 7)
| Scope |
All parameters |
| Operates on |
Valid rows including temp SDEs |
| Prior step |
Child Step 6 (Carried Forwards) |
| Next step |
Child Step 9 (Evil Twins) |
| Exclusion codes |
Exclude-C-PIV |
| Code location |
Inline in cleanchild() in
R/child_clean.R |
Overview
PIV identifies measurements that are biologically impossible or
extremely unlikely, using three blocks applied in sequence:
- Absolute PIV — fixed limits on raw measurements
(
v).
- Standardized PIV — cutoffs on unrecentered CSD
z-scores (
sd.orig).
- BMI limits — limits on the BMI of same-day
weight-height pairs, using only values that passed blocks 1 and 2.
After all three blocks, the temp-SDE identification is rerun: all
rows currently flagged Exclude-C-Temp-Same-Day are reset to
Include, then identify_temp_sde() is called
against the post-PIV state.
The working valid_set is computed once at the top of the
step via
.child_valid(data.df, include.temporary.extraneous = TRUE)
and reused across all three PIV blocks. It contains Include
rows and rows flagged Exclude-C-Temp-Same-Day; any other
non-Include row is not in valid_set and so
cannot be assigned Exclude-C-PIV here.
Absolute PIV thresholds
All conditions use strict < or > (not
<= / >=). Rows that match get
exclude := "Exclude-C-PIV".
Weight (WEIGHTKG):
First year (agedays <= 365) |
v < 0.2 kg |
Published minimum viable birth weight; allows preterm |
After first year (agedays > 365) |
v < 1 kg |
Minimum plausible outpatient weight |
Birth (agedays == 0) |
v > 10.5 kg |
Maximum plausible birth weight |
ageyears < 2 |
v > 40 kg |
Highest plausible weight under 2y |
| Any age |
v > 600 kg |
Published maximum human weight |
Height (HEIGHTCM):
| Any age |
v < 18 cm |
— |
| Any age |
v > 244 cm |
Published maximum |
Birth (agedays == 0) |
v > 65 cm |
— |
Head circumference (HEADCM):
| Any age |
v < 13 cm |
— |
| Any age |
v > 75 cm |
Published maximum |
Birth (agedays == 0) |
v > 50 cm |
Birth maximum |
Standardized PIV thresholds
Uses unrecentered z-scores (sd.orig).
Each cutoff is a user-settable cleangrowth() parameter; the
default values (shown in the tables below) preserve the prior hardcoded
behavior. The standardized block adds a
!grepl(piv_pattern, exclude) guard where
piv_pattern <- "^Exclude-C-PIV$". The guard skips rows
that the absolute block just assigned Exclude-C-PIV,
because valid_set was computed before the absolute block
ran and therefore still treats those rows as valid. Rows originally
flagged Exclude-C-Temp-Same-Day that were not hit by
absolute PIV can be overwritten to Exclude-C-PIV by the
standardized block; the 7d temp-SDE rerun afterward re-flags any SPA
that still has a duplicate Include.
Weight:
| Any age |
sd.orig > piv.z.wt.high |
piv.z.wt.high (22) |
Weight has no standardized low cutoff. With a
positive weight the CSD z below the median asymptotes to
-M/csd_neg as the weight → 0, peaking near -10 in infancy
and shallower elsewhere, so a low-weight z can never reach a useful
cutoff — the absolute floors (v < 0.2 kg under 1y,
v < 1 kg after) catch low weights instead. (HT/HC keep
their low cutoffs because those z’s are reachable just above
their absolute floors.)
Height:
ageyears < 1 |
sd.orig < piv.z.ht.low.young |
piv.z.ht.low.young (-25) |
ageyears >= 1 |
sd.orig < piv.z.ht.low.old |
piv.z.ht.low.old (-15) |
| Any age |
sd.orig > piv.z.ht.high |
piv.z.ht.high (8) |
The default upper HT cutoff (piv.z.ht.high = 8) is
tighter than the default upper WT cutoff
(piv.z.wt.high = 22). Analysis of CHOP data showed the
±15/±25 range was too loose for heights.
Head circumference:
| Any age |
sd.orig < piv.z.hc.low |
piv.z.hc.low (-15) |
| Any age |
sd.orig > piv.z.hc.high |
piv.z.hc.high (15) |
BMI limits
Weight and height are each checked on their own above, so a weight
and a height that are each plausible can still give an impossible BMI
(for example, a low-normal weight with a high-normal height). The BMI
block catches these. It runs after the absolute and standardized blocks,
so a value already excluded as PIV never feeds a BMI.
Pairs. Every same-day combination of a weight and a
height among the remaining valid_set rows (temp SDEs
included, so a day with two weights and one height gives two pairs). BMI
= v (kg) / (v (cm) / 100)^2.
Out-of-range pair (all strict, like the absolute
limits):
| Any age |
BMI < 3 |
ageyears < 2 |
BMI > 50 |
ageyears >= 2 |
BMI > 150 |
Which value is excluded. A value is excluded only if
none of its same-day pairings is in range. For each out-of-range
pair:
- If only one of the two values has no in-range pairing that day, that
value is excluded. (Example: two weights and one height on a day; one
weight gives an out-of-range BMI and the other does not. The height has
an in-range pairing, so the first weight is excluded.)
- If neither value has an in-range pairing (always the case when the
day has one weight and one height), the value with the larger
|sd.orig| is excluded. A missing sd.orig
counts as 0; an exact tie excludes the weight.
- If both values have in-range pairings, nothing is excluded.
Excluded values get the same code as the other PIVs,
Exclude-C-PIV. The BMI limits are not
user-configurable.
Temp SDE re-evaluation (7d)
After all three PIV blocks, all rows currently flagged
Exclude-C-Temp-Same-Day are reset to Include,
then identify_temp_sde() is rerun on the full dataset. This
mirrors the Step-5 temp-SDE logic against the post-PIV state: if the
prior temp-SDE keeper on an SPA has just been excluded as PIV, another
value in that SPA becomes the flagged duplicate.
Rationale
- Two PIV blocks (absolute then standardized) catch
complementary error modes. Absolute PIV catches data that is impossible
regardless of reference standard (e.g., a weight of 800 kg).
Standardized PIV catches values that are plausible in isolation but
implausible for a human (e.g., a height z-score of -50 — the raw value
might be 30 cm, which is within the absolute range but impossibly
short).
- Different age bands for the lower HT standardized
cutoff (
< 1 year
vs. >= 1 year): Very young children, especially preterm
infants, have much wider normal z-score ranges on the low side; CDC/WHO
references don’t extend to very low birth weights, so an uncorrected
z-score of -20 can be normal for a 24-week preemie whose
gestational-age-corrected z-score is within range. (Weight has no lower
standardized cutoff — see the Weight note above.)
- Tighter upper HT cutoff (
piv.z.ht.high = 8)
than upper WT (piv.z.wt.high = 22): Heights are
bounded biologically; a z-score of 22 for height is impossible, while
extremely high weight z-scores are observed in real severe-obesity
data.
- Uncorrected z-scores (
sd.orig):
Standardized PIV runs against the reference z-score (pre-GA-correction)
because the PIV cutoffs are meant to catch data-entry errors — applying
GA correction first would narrow the range on a preterm weight and make
the PIV boundary effectively parameter-specific.
- BMI limits after the individual limits: Weight and
height limits can each be met by a pair whose combination is impossible,
which is common in cross-sectional data where each child has one weight
and one height. Running the BMI check last means it only judges pairs
whose values are individually plausible. The limits are wide (BMI under
3, or over 50 under 2y and over 150 from 2y) so they catch only
impossible combinations, not very thin or very heavy children.
- Temp SDE re-evaluation after PIV: PIV can exclude
the row that was the temp-SDE keeper on an SPA, at which point a
different value in the SPA needs to be promoted to Include and the other
values re-flagged.
Configurable arguments in scope for Child Step 7
Six user-settable standardized-PIV cutoffs:
piv.z.wt.high (22), piv.z.ht.low.young (-25),
piv.z.ht.low.old (-15), piv.z.ht.high (8),
piv.z.hc.low (-15), piv.z.hc.high (15). Weight
has no standardized low cutoff. Absolute-PIV thresholds
(v < 0.2 kg, v > 244 cm, etc.) and the
BMI limits are not user-configurable.
Variables created and dropped
ageyears: created at the top of Child Step 7 for age
threshold checks, dropped at the end of the step.
piv_pattern: local character scalar
("^Exclude-C-PIV$") used as the standardized-block guard
and to drop PIV rows from the BMI pairs; not stored on
data.df.
bmi_rows, bmi_pairs, bmi_out,
bmi_excl: local tables/vectors for the BMI block; not
stored on data.df.
Checklist findings
- Operates-on set.
.child_valid(data.df, include.temporary.extraneous = TRUE)
is the correct selector for PIV: Include plus temp-SDE rows.
- Boundaries — all strict inequalities.
< 0.2, < 1, > 10.5,
> 244, < -25, > 22, etc.
A weight of exactly 0.2 kg is not excluded; a standardized z of exactly
-25 is not excluded.
- Age boundary at 1 year. Standardized PIV uses
ageyears < 1 vs. ageyears >= 1. A child
whose ageyears equals 1 exactly falls in the
>= 1 band.
- Valid-set refresh.
valid_set is
computed once and not refreshed between absolute and standardized PIV.
The standardized block relies on the
!grepl(piv_pattern, exclude) guard to skip rows just
assigned Exclude-C-PIV by the absolute block. Non-
Include rows other than
Exclude-C-Temp-Same-Day are not in valid_set
and so cannot be overwritten by Child Step 7.
- Parameter scope. All three params (WEIGHTKG,
HEIGHTCM, HEADCM) have their own absolute and standardized thresholds.
The BMI block uses WEIGHTKG and HEIGHTCM only; pairs are formed by
(subjid, agedays).
- Factor levels.
Exclude-C-PIV is
present in exclude.levels; absolute and standardized PIV
share the single code.
- Standardized-PIV cutoffs are parameterized. Six
piv.z.* cutoffs flow from cleangrowth()
through cleanchild() into Child Step 7. See the
“Configurable arguments in scope for Child Step 7” subsection
above.
EWMA1: Extreme EWMA (Child Step 11)
| Scope |
All parameters |
| Operates on |
Include values only (no temp SDEs), requiring 3+ per
subject-param |
| Prior step |
Child Step 9 (Evil Twins) |
| Next step |
Child Step 13 (Final SDE Resolution) |
| Exclusion code |
Exclude-C-Traj-Extreme |
| Code location |
Inline in cleanchild() in child_clean.R;
support functions ewma() and
identify_temp_sde() also defined in
child_clean.R |
Overview
EWMA1 flags extreme outliers using Exponentially Weighted Moving
Averages. For each measurement, the EWMA predicts what the value
“should” be based on the subject’s other measurements for the same
parameter. Values that deviate extremely from their EWMA prediction are
flagged Exclude-C-Traj-Extreme.
The step is iterative at two levels:
- Per subject-param: flag at most ONE worst value per pass.
- Globally: repeat across all subject-params until no new exclusions
occur.
This prevents one extreme value from distorting the EWMA of nearby
values and causing a cascade of false exclusions.
The step has three blocks:
- 11a. Pre-filter / setup — count and extreme filters
narrow the set of subject-params, and the subjects that currently carry
temp SDEs are captured for later targeted recalculation.
- 11b. Iteration loop — global while loop; each pass
processes one subject-param at a time via a per-group closure, flags at
most one worst value, and runs a targeted temp-SDE recalc.
- 11c. End-of-step temp SDE refresh — global reset +
rerun of
identify_temp_sde() before Child Step 13.
11a. Pre-filter / setup
Two pre-filters narrow the work:
- Count filter: Subject-params with
<= 2 Include values are skipped (EWMA needs 3+
points).
- Extreme filter: Subject-params where no Include
value has
|tbc.sd| > 3.5 are skipped — if the most
extreme z-score does not exceed 3.5, no value can meet the
tbc.sd > 3.5 criterion.
In addition, the subjects that currently carry
Exclude-C-Temp-Same-Day are captured up front as
subj_with_sde; the iteration loop only recalculates temp
SDEs for subjects that are in this set AND produce a new EWMA1 exclusion
in a given pass.
11b. Iteration loop
Each pass processes one subject-param at a time via a per-group
closure operating on a copy of the group’s rows
(copy(.SD)). The closure performs four computations and at
most one exclusion.
EWMA computation
For each subject-param group with > 2 Include
values:
- Exponent calculation — age-gap-dependent exponent
using linear interpolation:
- Gap <= 1 year: exponent = -1.5
- Gap >= 3 years: exponent = -3.5
- Between: linear interpolation
(
-1.5 - (ageyears - 1))
ageyears is based on the maximum of the gap before and
after each measurement.
- EWMA computation — calls
ewma() with
windowed matrix operations. Three variants are computed:
ewma.all: excluding only the current observation
ewma.before: excluding current + prior
ewma.after: excluding current + next
- The same three are computed for
ctbc.sd (corrected),
prefixed with c.. If ctbc.sd == tbc.sd for all
rows (vast majority — non-potcorr subjects), the ctbc columns are copied
rather than recomputed.
- Deviation (dewma):
dewma.all = tbc.sd - ewma.all
dewma.before = tbc.sd - ewma.before
dewma.after = tbc.sd - ewma.after
c.dewma.all = ctbc.sd - c.ewma.all
Exclusion criteria
A value is a potential exclusion (pot_excl = TRUE) if
ALL of:
Positive outlier:
dewma.all > 3.5
dewma.before > 3
dewma.after > 3
tbc.sd > 3.5
- plus/minus check (uncorrected):
tbc.p_minus - ewma.all > 3.5 AND
tbc.p_minus > 3.5
- corrected check:
ctbc.sd is NA, OR
(c.dewma.all > 3.5 AND the corrected plus/minus check
ctbc.p_minus - c.ewma.all > 3.5 AND
ctbc.p_minus > 3.5)
OR negative outlier:
dewma.all < -3.5
dewma.before < -3
dewma.after < -3
tbc.sd < -3.5
- plus/minus check (uncorrected):
tbc.p_plus - ewma.all < -3.5 AND
tbc.p_plus < -3.5
- corrected check:
ctbc.sd is NA, OR
(c.dewma.all < -3.5 AND
ctbc.p_plus - c.ewma.all < -3.5 AND
ctbc.p_plus < -3.5)
The plus/minus check protects a value from exclusion when nudging the
measurement in the less-extreme direction (a ±5% weight or ±1 cm
height/HC change, recentered to tbc.p_* /
ctbc.p_*) would bring it within threshold — covering
expected short-term variation (bladder fullness, clothing) and
measurement precision. It runs on the uncorrected z (against
ewma.all) and, for GA-corrected values, on the corrected z
(against c.ewma.all). For non-potcorr subjects
ctbc.sd equals tbc.sd, so the corrected checks
repeat the uncorrected ones.
Additional constraints:
- The last Include value is never excluded here (its
EWMA is one-sided). The first (birth) value is not
auto-protected, so an extreme birth error is catchable in Step 11; a
legitimate birth is instead protected only by the catch-up rule
below.
- Preterm catch-up protection: for a GA-corrected
subject,
.child_catchup_protect() protects both members of
an early (< 3 mo) rising pair whose z-rise stays within the cap (see
that helper) on at least one frame and whose own
tbc.sd <= -1.5. This protects legitimate fast early
catch-up. The same helper is used by Child Steps 15/16. There is no
separate corroboration gate on the protection — GA correction itself is
corroboration-gated upstream (Child Step 2b), so only corroborated
preterms are corrected and thus reach this protection.
- At most ONE value is flagged per subject-param per iteration.
Worst-value selection
When more than one value meets criteria, the worst is selected by
order(pot_excl, abs(tbc.sd + dewma.all), -internal_id, decreasing = TRUE)[1].
The sort key abs(tbc.sd + dewma.all) combines the z-score
magnitude with its deviation — values that are both extreme and far from
EWMA prediction score highest. Lowest internal_id breaks
ties.
Iteration bookkeeping
After each per-group pass:
- Subject-params with a NEW exclusion this pass are identified
(comparing
had_ewma1_before flags against the post-pass
state).
- Temp SDEs are recalculated ONLY for subjects that are both in
subj_with_sde (had a temp SDE going into Child Step 11) AND
produced a new exclusion this pass — the post-pass recalc block resets
those subjects’ temp SDEs to Include and reruns
identify_temp_sde() on that subset. A temp SDE can never be
flagged during a pass (only Include rows are exclusion candidates), but
this recalc can promote a set-aside temp SDE to
Include; since the next iteration reprocesses exactly the subject-params
that just excluded, a promoted value can then be excluded as
Exclude-C-Traj-Extreme in a later iteration.
- The next iteration’s
sp_to_process is the set of
subject-params with new exclusions from this pass.
- The loop terminates when no subject-param produces a new
exclusion.
11c. End-of-step temp SDE refresh
After the loop, all Exclude-C-Temp-Same-Day rows are
reset to Include and identify_temp_sde() is run globally
across the full dataset. This final pass catches any residual drift left
by the per-iteration targeted recalc, which only handles subjects that
started with temp SDEs. It does NOT use
exclude_from_dop_ids — that DOP-median biasing is specific
to Child Step 13.
Rationale
- “Extreme” EWMA (Child Step 11) vs. “moderate” EWMA (Child
Steps 15/16): Splitting EWMA into two passes lets the extreme
pass run before SDE resolution (Child Step 13) and velocity checks
(Child Step 17). Extreme outliers distort the median, the SP/DOP checks
used by SDE, and the neighbor-to-neighbor differences used by velocity —
removing them first makes the downstream steps more reliable.
- High thresholds (3.5 / 3 / 3.5 / 3) for both
dewma and tbc.sd: All four conditions
together make this an “extreme” filter. Values that meet these
thresholds are essentially never legitimate measurements.
- Corrected-z check
(
c.dewma.all > 3.5): For non-potcorr subjects,
ctbc.sd equals tbc.sd, so the corrected check
repeats the uncorrected one. For potcorr subjects, an extreme value must
be extreme in both the uncorrected and corrected z-scores to be
excluded.
- Age-gap-dependent EWMA exponent: Shorter gaps
(within 1 year) use a less-negative exponent (-1.5), giving nearby
neighbors more weight. Longer gaps (3+ years) use -3.5, appropriate for
widely-spaced data where only the closest values carry meaningful
signal.
- Last value never excluded: EWMA is unreliable at
the endpoints (only one neighbor informs the prediction), so the last
value is never excluded here. The first/birth value is not
auto-protected — it is protected only as part of a legitimate preterm
catch-up rise — so an extreme birth error is catchable in this step.
Endpoints not excluded here fall to Child Steps 15/16 (moderate EWMA)
and Child Step 19 (pairs/singles).
- At most one exclusion per subject-param per pass:
Prevents the EWMA of one extreme value from pulling other values into
the excluded region in a single pass. After recomputing, the “next
worst” may no longer look extreme.
- Pre-filter by
|tbc.sd| > 3.5: If
the most extreme z-score in a subject-param is below the
tbc.sd > 3.5 criterion, no value can meet it — cheap way
to skip a large fraction of subject-params in typical data.
Configurable arguments in scope for Child Step 11
ewma_window |
15 |
Maximum number of Include observations on each side that contribute
to the EWMA weighting. Passed into ewma(). Same parameter
used in Child Steps 11, 13, 15/16, and 17. |
Child Step 11 thresholds (> 3.5, > 3,
< -3.5, < -3) are not
user-configurable.
Variables created and dropped
All EWMA working columns (ewma.all,
ewma.before, ewma.after,
c.ewma.all, c.ewma.before,
c.ewma.after, dewma.all,
dewma.before, dewma.after,
c.dewma.all) are computed inside the per-group closure on a
local copy (copy(.SD)) and are not persisted to
data.df. Bookkeeping columns
(had_ewma1_before, subj_with_sde,
sp_to_process) are local to the iteration loop and cleaned
up at the end.
Checklist findings
- Boundaries — all strict:
> 3.5,
> 3, < -3.5, < -3 for
dewma; > 3.5 / < -3.5 for
tbc.sd and c.dewma.all.
.child_valid() call: Correctly
excludes temp SDEs via
include.temporary.extraneous = FALSE. Only Include values
participate in EWMA calculation and as exclusion candidates.
- ctbc optimization: Skips ctbc EWMA computation when
ctbc.sd == tbc.sd (copies ewma.* to
c.ewma.* instead). Correct for non-potcorr subjects, which
are the vast majority.
- Parameter scope: All 3 params (WEIGHTKG, HEIGHTCM,
HEADCM) handled uniformly.
- Factor levels:
Exclude-C-Traj-Extreme
exists in exclude.levels.
- Efficiency: Good — two levels of pre-filtering
(count + extreme), targeted SDE recalc, and subject-params drop out of
the loop as they stop producing exclusions.
ewma_window parameter: Passed through
to ewma() — controls max Include observations on each side
(default 15).
- Boundaries — tiebreaker:
internal_id
is the final tiebreaker in the worst-value order() call
(lowest internal_id wins).
EWMA2: Moderate EWMA (Child Steps 15/16)
| Scope |
Child Step 15: all params (birth HT/HC excluded — except an HT/HC SP
with exactly 3 valid values one of which is a birth, where the birth is
kept; see Overview); Child Step 16: birth HT/HC only |
| Operates on |
Include values only (no temp SDEs), 3+ per subject-param |
| Prior step |
Child Step 13 (Final SDE Resolution) |
| Next step |
Child Step 17 (Height/HC Velocity) |
| Exclusion codes |
Exclude-C-Traj (all moderate EWMA sub-rules collapsed
into one code) |
| Code location |
Inline in cleanchild() in child_clean.R.
Reads the tbc.p_plus/tbc.p_minus (and
ctbc.*) z-scores built in cleangrowth()
preprocessing; uses ewma_cache_init() /
ewma_cache_update() for the incremental EWMA inside the
iteration loop, and get_dop() for the Step 15
last-ext DOP lookup. |
Overview
EWMA2 uses the same EWMA mechanism as EWMA1 but with lower
thresholds and position-specific rules. Where
EWMA1 catches extreme outliers (thresholds ~3.5), EWMA2 catches moderate
outliers (thresholds ~1–4 depending on position and context).
Child Step 15 handles all measurements except birth HT/HC. Step 16
handles birth HT/HC separately because birth values have different
clinical expectations and need higher thresholds.
Birth HT/HC in a 3-value SP. There is one exception
to “birth HT/HC excluded from Step 15.” If an HT/HC subject-param has
exactly 3 valid values, one of which is a birth
(agedays == 0), the birth is kept as a normal Step 15
participant — it enters the EWMA and is exclusion-eligible
(scored by the birth-value rules, which fire on
agedays == 0 for any param). For SPs with ≥ 4 valid values
the birth is excluded from Step 15 and evaluated only in Step 16.
Rationale: removing the birth from a 3-value SP would leave only 2
non-birth values, below the 3-value threshold Step 15 needs, so the
whole SP would drop out of Step 15 and an extreme non-birth value could
reach Include (e.g. a very high HC a few days after a normal birth HC —
Step 11 protects it as the last value, and HC velocity does not bound
short intervals, so only Step 15 can catch it). Keeping the birth lets
Step 15 evaluate the SP. Because Step 16 only processes HT/HC SPs that
still have ≥ 3 valid values, if Step 15 excludes any value from such a
3-value SP it falls below Step 16’s gate and the birth is not
re-evaluated there, so the two steps do not double-handle the birth. The
set of qualifying SPs (keep_birth_sp15) is computed once
from the initial valid counts and held fixed so the incremental EWMA
cache only ever sees removals.
Additional criteria (addcrit)
All EWMA2 rules require addcrit — a supplementary check
that the measurement differs from BOTH neighbors by more than 1 SD even
after a ±5%/±1cm perturbation toward each neighbor. This protects values
that are close to a neighbor: even if EWMA says the value is outlying,
if it sits within 1 SD of a neighbor (even after the nudge), it is
likely real. The check runs on the uncorrected z and, for GA-corrected
values, on the corrected z as well.
Neighbor difference variables (computed within each
subject-param group, in agedays order; each is the difference to the
next/prior Include neighbor):
- Uncorrected:
tbc_diff_next/prior (from
tbc.sd), tbc_diff_plus_next/prior (from
tbc.p_plus), tbc_diff_minus_next/prior (from
tbc.p_minus)
- Corrected:
ctbc_diff_next/prior (from
ctbc.sd), ctbc_diff_plus_next/prior (from
ctbc.p_plus), ctbc_diff_minus_next/prior (from
ctbc.p_minus)
tbc.p_plus/tbc.p_minus and
ctbc.p_plus/ctbc.p_minus are the recentered
z-scores of the perturbed measurements, built in preprocessing with the
same WHO/CDC blending — and, for the corrected versions, the same
gestational-age-correction pathway — as the original z-scores. For a
non-corrected value ctbc == tbc and
ctbc.p == tbc.p, so the corrected clauses equal the
uncorrected ones and add nothing.
addcrithigh — the value must stay more than 1 SD
above each neighbor on the uncorrected z and the corrected z, including
under both perturbations:
dewma.before > 1 & dewma.after > 1 &
((tbc_diff_next > 1 & tbc_diff_plus_next > 1 & tbc_diff_minus_next > 1) | is.na(tbc_diff_next)) &
((tbc_diff_prior > 1 & tbc_diff_plus_prior > 1 & tbc_diff_minus_prior > 1) | is.na(tbc_diff_prior)) &
((ctbc_diff_next > 1 & ctbc_diff_plus_next > 1 & ctbc_diff_minus_next > 1) | is.na(ctbc_diff_next)) &
((ctbc_diff_prior > 1 & ctbc_diff_plus_prior > 1 & ctbc_diff_minus_prior > 1) | is.na(ctbc_diff_prior))
addcritlow: Mirror with < -1 for all
comparisons.
NA handling: endpoints have NA for their missing neighbor (first row
has NA prior, last row has NA next), so addcrit passes on
that side automatically.
Child Step 15 exclusion rules
All rules require:
- The appropriate
addcrit (addcrithigh or
addcritlow)
dewma.all threshold (uncorrected)
c.dewma.all threshold (corrected) — when
c.dewma.all is available, it must also exceed the threshold
(for non-potcorr subjects ctbc.sd equals
tbc.sd, so this repeats the uncorrected check):
(c.dewma.all > threshold | is.na(c.dewma.all))
| middle |
Not first/last |
— |
> 1 / < -1 |
> 1 / < -1 |
— |
| birth-WT |
agedays == 0 |
next < 365.25 |
> 3 / < -3 |
> 3 / < -3 |
— |
| birth-WT-ext |
agedays == 0 |
next >= 365.25 |
> 4 / < -4 |
> 4 / < -4 |
— |
| first |
first_meas |
next < 365.25 |
> 2 / < -2 |
> 2 / < -2 |
— |
| first-ext |
first_meas |
next >= 365.25 |
> 3 / < -3 |
> 3 / < -3 |
— |
| last |
Last row |
prev < 2*365.25 |
> 2 / < -2 |
> 2 / < -2 |
abs(tbc_prev) < 2 |
| last-high |
Last row |
prev < 2*365.25 |
> abs(tbc_prev) /
< -abs(tbc_prev) |
> min(abs(tbc_prev), 3) /
< -min(abs(tbc_prev), 3) |
abs(tbc_prev) >= 2 |
| last-ext |
Last row |
prev >= 2*365.25 |
> 3 / < -3 |
> 3 / < -3 |
abs(tbc_prev) < 2, DOP diff |
| last-ext-high |
Last row |
prev >= 2*365.25 |
> 1+abs(tbc_prev) /
< -1-abs(tbc_prev) |
> 3 / < -3 |
abs(tbc_prev) >= 2, DOP diff |
Note: For last-high and last-ext-high the
uncorrected dewma.all bar scales with
abs(tbc_prev), and the corrected c.dewma.all
gate is held at or below that bar: min(abs(tbc_prev), 3)
for last-high (so it never exceeds the uncorrected bar), and a fixed 3
for last-ext-high (whose bar 1 + abs(tbc_prev) is always
>= 3). So the corrected gate never sits above a value’s
own uncorrected bar.
Gap definitions:
- birth-WT, first: gap =
agedays[next] - agedays (to next
measurement)
- last: gap =
agedays[last] - agedays[last-1] (from prior
measurement)
first_meas definition:
- WEIGHTKG: first Include row per subject-param where
agedays > 0 (birth WT is handled by birth-WT rules)
- HEIGHTCM/HEADCM: first Include row among non-birth rows per
subject-param (birth HT/HC excluded from Child Step 15 entirely —
handled by Child Step 16)
DOP diff (last-ext and last-ext-high):
Cross-parameter check. For positive outliers:
tbc.sd - tbc_dop > 4 (or tbc_dop is NA).
For negative: tbc.sd - tbc_dop < -4 (or
tbc_dop is NA). Where tbc_dop is the same-day
DOP value if available, otherwise the median tbc.sd of all
DOP values. If no DOP data exists, tbc_dop is NA and the
check passes.
Key boundaries:
- Gap thresholds:
< 365.25 vs
>= 365.25 (strict <);
< 2*365.25 vs >= 2*365.25 for last
- All dewma thresholds are strict
>/<
abs(tbc_prev) < 2 vs >= 2
distinguishes “normal prior” from “extreme prior” for last-value
rules
Birth HT/HC (Child Step 16)
Separate step for birth height and HC values, using the same EWMA
infrastructure but with birth-specific rules only. Uses the same addcrit
definitions and c.dewma.all pattern as Child Step 15.
| birth-HT-HC |
< 365.25 |
> 2 / < -3 |
> 2 / < -3 |
| birth-HT-HC-ext |
>= 365.25 |
> 3 / < -4 |
> 3 / < -4 |
The high-side thresholds (> 2 / > 3)
match Step 15’s non-birth first-value bar; the low side stays looser
(< -3 / < -4). So Step 16 is looser only
for smaller (low) births: a high birth value that drops to
normal (e.g. a birth length/HC z just over 4 falling to ~2) is judged at
the same bar as any other first value and excluded, while a low birth
value rising in legitimate preterm catch-up keeps the wider
tolerance.
Only processes HT/HC subject-params that themselves have a birth
measurement (agedays == 0) with 3+ Include values. A subject with birth
HT but no birth HC (or vice versa) has only the birth-carrying param
processed here. Structurally identical to Child Step 15 (iterative,
cached EWMA, one exclusion per iteration).
Preterm catch-up protection
After candidates are marked and before worst-value selection, both
Child Step 15 and Child Step 16 protect the members of an early catch-up
increase from exclusion (helper .child_catchup_protect()).
Preterm infants legitimately rise quickly in z during early infancy, so
a fast early increase should not be treated as an outlier.
The protection applies only to GA-corrected subjects (those with
ctbc.sd != tbc.sd for some value). For a consecutive
Include pair (earlier → later, in agedays order) it protects
both members when:
- the earlier member is younger than 3 months,
- the z-score rises from the earlier to the later member (upward only
— an early drop keeps the normal stricter thresholds, since a
drop is more likely a real error), and
- the rise does not exceed
allowed = min(4, 2 + 0.5 × ceiling(max(0, interval_months − 1)))
on at least one of the uncorrected (tbc) or corrected
(ctbc) frames — 2 z for the first month, +0.5 z per
full-or-partial month after, capped at 4 z. See the
.child_catchup_protect() Support Function entry for the
canonical statement.
A value is then protected only if its own uncorrected z is itself low
(tbc.sd <= -1.5). Above that gate the corrected frame is
ignored, so a value that is not low but whose corrected z was raised by
GA correction is left as a candidate. Both members of a qualifying pair
are protected — protecting only the later (catch-up) value would be
pointless if the earlier baseline value were then dropped. The
protection only removes values from candidacy; it never creates an
exclusion, and it leaves non-corrected subjects unchanged.
Isolated low uncorrected birth-WT guard
A second, weight-only protection runs in Child Step 15 (after the
birth-WT rules mark candidates, before worst-value selection) for the
uncorrected case the catch-up protection above does not cover.
A birth weight (row 1, agedays == 0) is un-flagged when
all of:
- it is uncorrected
(
ctbc.sd == tbc.sd),
- it is low (
tbc.sd < -2),
- it is isolated — the next Include weight is
>= 2*365.25 days away (the same window as form-1
GA-correction corroboration), and
abs(tbc.sd) + (tbc.sd[next] - tbc.sd) <= 7.
The intent: a fairly-plausible low birth weight should not be dragged
out merely because a distant string of high weights pulls the EWMA up.
Only when the birth is both substantially low and the
jump to the next weight is large (sum > 7) is it
excluded. This is gated to WEIGHTKG and to uncorrected births because
corrected preterm catch-up is already handled by
.child_catchup_protect() above. Like the catch-up
protection, it only removes the birth from candidacy and never creates
an exclusion.
Worst-value selection
Both steps use abs(tbc.sd + dewma.all) as the sort key
(same as EWMA1). Ties on the sort key are broken by lowest
internal_id — the row with the lowest
internal_id among the tied candidates becomes the
exclusion. Ties are rare because abs(tbc.sd + dewma.all) is
continuous, but internal_id makes the selection
deterministic.
Global iteration
Same pattern as EWMA1: subject-params drop out when they stop
producing new exclusions. EWMA caches (ewma2_caches,
ewma2b_caches) persist across iterations for incremental
updates. Step 15’s DOP snapshot (used by the last-ext and
last-ext-high rules) is refreshed each iteration; Step 16
has no DOP lookup.
Rationale
- “Moderate” thresholds (1, 2, 3, 4) by position:
Middle measurements have neighbors on both sides, so a smaller deviation
(|dewma| > 1) plus the
addcrit check is enough. First
and last measurements have only one-sided context, so higher thresholds
(2 or 3) avoid false positives; the exact threshold depends on how far
the gap is.
- Gap-dependent thresholds (
< 365.25
vs. >= 365.25, < 2*365.25
vs. >= 2*365.25): A value an unusual distance
from a close-in-time neighbor is more suspicious than the same distance
over a multi-year gap, during which the child’s trajectory may
legitimately have shifted.
addcrit (perturbation check): Even
when EWMA says a value is off-trajectory, if the measurement is within 1
SD of an adjacent value — even after ±5% or ±1 cm perturbation — the two
look like they came from the same trajectory on a neighbor-to-neighbor
basis. Excluding it would create a spurious gap.
- Multiplicative perturbation for weight (±5%), additive for
height/HC (±1 cm): Weight changes are naturally proportional;
±5% reflects expected measurement precision at any weight. Height/HC are
typically measured to the nearest centimeter, so ±1 cm is the rounding
tolerance regardless of absolute length.
- Birth HT/HC as a separate step (Child Step 16):
Birth heights and HCs have different clinical expectations and much
wider plausible ranges than later measurements (growth is rapid; GA is
uncertain). The low-side thresholds stay wider (−3 /
−4) to reflect this (a low birth value rising in legitimate catch-up),
while the high side was tightened to Step 15’s
first-value bar (2 / 3) so an implausibly high birth value that drops to
normal is still caught. Separating the logic keeps Child Step 15’s rules
cleaner.
last-high and last-ext-high cap
the corrected check at the uncorrected bar: The
dewma threshold scales with abs(tbc_prev) to
allow higher tolerance for already-extreme trajectories. The corrected-z
gate is min(abs(tbc_prev), 3) for last-high and a fixed 3
for last-ext-high — at or below the value’s own uncorrected bar, so a
genuinely implausible value is not excused just because the prior value
was unusual, while the corrected gate never rises above the uncorrected
bar (which for abs(tbc_prev) in [2,3) a fixed
3 would have done, silently over-tightening).
- DOP diff for
last-ext rules: When the
time gap is large, the DOP comparison acts as a sanity check against
trajectory drift — a weight that jumps up alone without the
corresponding height change is more suspicious than a coordinated
shift.
Configurable arguments in scope for Child Steps 15/16
ewma_window |
15 |
Maximum number of Include observations on each side that contribute
to the EWMA weighting. Same parameter used in Child Steps 11, 13, 15/16,
and 17. |
Moderate EWMA thresholds (1, 2, 3, 4) and the ±5% / ±1 cm
perturbation magnitudes are not user-configurable.
Variables created and dropped
first_meas (created): created before Child Step 15,
dropped after Child Step 16. (p_plus, p_minus
and their z-scores come from cleangrowth()
preprocessing.)
sp_key (created): created for Child Steps 15/16,
dropped after Child Step 16
- EWMA working columns and neighbor-difference columns are created
inside per-group closures and not persisted to
data.df.
ewma2_caches and ewma2b_caches
(incremental EWMA caches): held in local environments during each
iteration loop; freed (rm()) after their respective step
completes.
Checklist findings
- Boundaries: All dewma thresholds are strict
> / <. Gap thresholds use strict
< vs >= consistently.
.child_valid() calls: Child Step 15
excludes temp SDEs AND birth HT/HC. Child Step 16 excludes temp SDEs,
includes only birth HT/HC subjects.
- p_plus/p_minus values: WT uses multiplicative
(±5%), HT/HC uses additive (±1 cm). Consistent with clinical
interpretation (5% of weight is proportional; 1 cm is the measurement
precision for length).
- first_meas: Recalculated each iteration (exclusions
change which row is “first”).
- DOP lookup: Uses pre-computed keyed snapshot
(
dop_snap) refreshed each iteration. O(log n) lookup via
get_dop().
- Parameter scope: Child Step 15 handles all 3 params
(birth HT/HC excluded). Child Step 16 handles HT/HC birth only (WT birth
is in Child Step 15).
- Factor levels: The single
Exclude-C-Traj code used by EWMA2 exists in
exclude.levels.
- Efficiency: Pre-filter by
tbc.sd range
> 1 (analogous to EWMA1’s > 3.5 filter but with a lower bound).
Incremental EWMA caching via ewma_cache_init() /
ewma_cache_update().
- No temp SDE rerun after Child Step 15/16. Unlike
Steps 7/9/11, there is no temp SDE recalculation after EWMA2. Temp SDEs
were permanently resolved in Child Step 13.
Height/HC Velocity (Child Step 17)
| Scope |
HEIGHTCM and HEADCM only (not WEIGHTKG) |
| Operates on |
Valid rows excluding temp SDEs, 2+ per subject-param |
| Prior step |
Child Steps 15/16 (EWMA2) |
| Next step |
Child Step 19 (Pairs and Singles) |
| Exclusion codes |
Exclude-C-Abs-Diff |
| Code location |
Inline in cleanchild() in child_clean.R.
Uses Tanner height velocity (loaded once in cleangrowth()
and passed in) and WHO HT / HC velocity reference tables (loaded inside
cleanchild() from inst/extdata/); calls
ewma() directly (not the cache API) for the 3+ measurement
tiebreaker, .child_valid() for row eligibility, and
.child_exc() for the exclusion code. |
Overview
Child Step 17 checks whether the raw measurement change between
consecutive observations is physiologically plausible. Unlike EWMA steps
that work on z-scores, this step works on raw cm values. Height/HC
should not decrease more than allowed (mindiff) or increase more than
allowed (maxdiff). The allowed change depends on:
- Age and gap (Tanner velocity for HT; WHO velocity for HT under 24
months of age with gaps under 9 months, and for HC under 24 months)
- Time gap between measurements
- Parameter (HC has tighter tolerances than HT)
Velocity references. Height velocity limits use the
Tanner and Davies (1985) standards and the WHO growth velocity standards
(2009); head circumference velocity limits use the WHO standards only.
How the step chooses between the two height references is described
below.
Tanner JM, Davies PS. Clinical longitudinal standards for height and
height velocity for North American children. J Pediatr. 1985
Sep;107(3):317-329. doi: 10.1016/s0022-3476(85)80501-1.
World Health Organization. WHO Child Growth Standards: Growth
velocity based on weight, length and head circumference: Methods and
development. Geneva: World Health Organization; 2009. ISBN
9789241547635. Link
Weight is excluded because weight can legitimately decrease.
Velocity references and threshold computation
Height uses two reference systems applied in
priority order. Each pair of adjacent measurements gets a
mindiff (minimum allowed change, typically negative =
allowed decrease) and maxdiff (maximum allowed
increase):
Tanner (midpoint age ≥ 18 months):
tanner.months = 6 + 12 * floor(midpoint_age_years)
where midpoint is the mean of the two agedays values. Each Tanner
reference row is centered on a half-year (30 months = the 2–3 year
interval, 42 = 3–4 years, …), so this picks the row whose one-year
interval contains the midpoint. Midpoints from 18 months up to 2 years
also use the 30-month row (the lowest entry). Below 18 months,
tanner.months is NA, so the NA filter matches the merge
key.
- Merge with Tanner reference table →
min.ht.vel and
max.ht.vel (annual velocity values by sex and age)
- max.ht.vel floors (gap-dependent):
- Always: floor at 2.54 cm (1 inch)
- If gap > 2 months: floor at 5.08 cm (2 inches)
- If gap > 6 months: floor at 10.16 cm (4 inches)
- If gap > 1 year: floor at 20.32 cm (8 inches)
- mindiff (gap < 1 year):
0.5 * min.ht.vel * (gap_years)^2 - 3
- mindiff (gap ≥ 1 year):
0.5 * min.ht.vel - 3
- maxdiff (gap < 1 year):
2 * max.ht.vel * (gap_years)^0.33 + 5.5
- maxdiff (gap ≥ 1 year):
2 * max.ht.vel * (gap_years)^1.5 + 5.5
- The two pairs of formulas are continuous at gap = 1 year, so the
boundary (d_agedays = 365.25) produces the same mindiff/maxdiff from
either branch.
WHO (ages 0–24 months, height):
whoagegrp.ht = round(agedays / 30.4375) when
agedays/30.4375 ≤ 24; otherwise NA. Never exceeds 24. Also
NA-ed out when the next measurement (this row’s ageyears + gap) exceeds
24 months, so a pair crossing the WHO/CDC boundary does not extrapolate
past the WHO reference. In the pre-filter this is applied directly to
whoagegrp.ht; inside the per-group while loop the same
condition zeroes out who_mindiff_ht /
who_maxdiff_ht each iteration because
d_agedays changes as rows are excluded. This 24-month range
(through 730.5 days) is intentionally different from the day-730
length/height boundary used elsewhere (vignette("wrapper"),
Phase 8, fact 2): a pair of heights ending at day 730 is still checked
against the WHO increments.
- WHO increment age groups based on gap (
d_agedays)
between adjacent measurements:
- gap < 46 days → 1 month
- 46 ≤ gap < 76 → 2 months
- 76 ≤ gap < 107 → 3 months
- 107 ≤ gap < 153 → 4 months
- gap ≥ 153 → 6 months
- Merge with WHO velocity reference →
who_mindiff_ht,
who_maxdiff_ht at the selected increment interval
- Gap scaling: If actual gap < reference interval
(e.g., 45 days against a 2-month reference), scale
who_mindiff_ht by gap / reference. If actual gap >
reference interval, scale who_maxdiff_ht by gap /
reference. The mindiff-side is scaled only when shorter than reference;
the maxdiff-side is scaled only when longer. Intuition: a shorter
interval allows less loss; a longer interval allows more gain.
- For gaps < 9 months (WHO preferred):
- Transform:
mindiff = who_mindiff * 0.5 - 3
- Transform:
maxdiff = who_maxdiff * 2 + 3
- For gaps ≥ 9 months (Tanner preferred, WHO
fallback):
- Only use WHO if Tanner data unavailable (no
min.ht.vel)
- WHO fallback transform:
mindiff = who_mindiff * 0.5 - 3,
maxdiff = who_maxdiff * 2 + 3
- Default when no reference data available:
mindiff = -3 (allow up to 3 cm decrease)
- Birth adjustment: none — an implausibly high
birth-length value that drops to normal is caught at the same loss floor
as any other pair. The gain side is unaffected in practice: at birth
maxdiff = who_maxdiff*2 + 3 is already ~26.8 cm, so no
realistic gain is ever flagged.
HC uses WHO only (no Tanner):
whoagegrp.hc = round(agedays / 30.4375) when
agedays/30.4375 ≤ 24; otherwise NA. Same cap as HT.
- WHO increment age groups: HC has no 1-month reference — the smallest
interval is 2-month, so gaps shorter than 46 days (or longer than ~199
days) fall through to the fallback default below.
- 46 ≤ gap < 76 → 2 months
- 76 ≤ gap < 107 → 3 months
- 107 ≤ gap < 153 → 4 months
- 153 ≤ gap < 200 → 6 months
- Gap scaling: Same proportional scaling as HT
(mindiff shrinks when actual gap < reference; maxdiff grows when
actual > reference).
- Tolerance transform:
mindiff = who_mindiff * 0.5 - 1.5 and
maxdiff = who_maxdiff * 2 + 1.5 (tighter than HT: ±1.5 cm
vs ±3 cm).
- Default when HC is outside the WHO-reference range
(agedays beyond ~24 months, or gap outside the 46–199-day band):
mindiff = -1.5, maxdiff left at NA (no upward
bound). HC measurement pairs outside the reference band therefore gate
only the decrease side and effectively skip the increase-side check.
This is intentional: (1) for height, the upper velocity bound catches
very few values that the lower bound does not already catch; (2) an
appropriate HC upper limit for ages beyond 24 months would need to be
interval- and starting-age-specific and lacks an established reference;
(3) large HC increases outside the reference band are already likely to
be identified by EWMA steps (Child Steps 9, 11, 15, or 16) or by the
subsequent measurement showing an implausibly large decrease.
- Birth adjustment: none — as with HT, birth HC drops
are caught at the standard loss floor.
Violation detection and resolution
Child Step 17 uses a pre-filter that computes all thresholds in one
vectorized pass, then only enters the per-group while loop for
subject-params with at least one violation.
Violation condition: A pair violates if:
diff_prev < mindiff_prior (decreased too much from
prior) OR diff_next < mindiff (next decreased too much
from this) OR diff_prev > maxdiff_prior (increased too
much from prior) OR diff_next > maxdiff (next increased
too much from this)
Where diff_prev = v - v[prior],
diff_next = v[next] - v (raw cm differences, not
z-scores).
Resolution within each group:
For 3+ measurements, EWMA-based tiebreaker:
- Compute EWMA on raw
tbc.sd values (via
ewma(), not the cache API — EWMA is recomputed from scratch
each iteration).
- For each violating pair, flag which member has a worse EWMA
relationship:
bef.g.aftm1: TRUE if |dewma.before|
exceeds the prior row’s |dewma.after|
aft.g.aftm1: TRUE if |dewma.after| exceeds
the next row’s |dewma.before|
- Exclude the flagged member. Among multiple candidates, select by
highest
|dewma.before| (codes 1,3) or
|dewma.after| (codes 2,4), with internal_id as
the tiebreaker (lowest internal_id wins).
- Remove excluded row, recalculate, repeat until no violations
remain.
For exactly 2 measurements:
- EWMA is still computed per the shared code path above but is not
consumed — the tiebreaker uses
abs(tbc.sd) directly.
- The abs-comparison uses
>= (not strict
>) so that tied abs(tbc.sd) pairs with a
violating diff flag both rows; the
order(-absval, internal_id) candidate selection then picks
one (lowest internal_id among tied candidates).
- Only one row is excluded per violating 2-row group.
Rationale
- Raw-cm units (not z-scores): Height and HC velocity
are bounded by physiology (children do not shrink by more than a few cm
per year; growth velocity is well-characterized by age and sex). Working
in raw cm uses published velocity references directly.
- Weight is excluded: Weight can legitimately
decrease (illness, puberty, deliberate loss), so a velocity-based
plausibility check would produce too many false positives.
- Two references (Tanner + WHO) for height: Tanner
has age/sex-specific peak-velocity information used for pubertal growth;
WHO has finer-grained month-by-month velocities for infancy. The
priority rule — Tanner for gaps ≥ 9 months (when the smoothed reference
is appropriate) and WHO for shorter gaps at young ages — uses each where
it is most accurate.
- Gap-scaled tolerances (
* 0.5 - 3 /
* 2 + 5.5): Published velocity values are central
tendencies; the tolerance formulas expand them into plausibility bands
that widen with longer gaps (more cumulative variation possible).
- Gap-dependent
max.ht.vel floors: Short
intervals should allow at least the measurement precision (1 inch);
longer intervals should allow catch-up growth or other legitimate high
velocity that brief-interval formulas would over-restrict.
- Tighter HC tolerances (±1.5 cm vs ±3 cm for
height): HC grows more slowly and is measured more precisely,
so any large change is less likely to be plausible than the same
absolute change in height.
- EWMA-based tiebreaker within violating pairs (3+
measurements): When a pair violates, one of the two must be
wrong but either could be. Comparing each candidate’s EWMA relationship
to its other neighbor identifies which end of the pair drifted from the
trajectory.
- For exactly 2 measurements: No EWMA context exists,
so the more-extreme-z value is excluded as a coarse heuristic.
Configurable arguments in scope for Child Step 17
ewma_window |
15 |
Max Include observations on each side for the 3+-measurement EWMA
tiebreaker (ewma() call inside the while loop) |
All velocity thresholds and tolerance formulas are hardcoded;
velocity references come from inst/extdata/ reference
tables (Tanner height velocity, WHO HT velocity, WHO HC velocity).
Variables created and dropped
Pre-filter columns live on pf (a local subset of
data.df) and are discarded when rm(pf) runs at
the end of the pre-filter block. Per-group closure columns
(tanner.months, whoagegrp.ht,
whoagegrp.hc, whoinc.age.ht,
whoinc.age.hc, min.ht.vel,
max.ht.vel, who_mindiff_ht,
who_maxdiff_ht, who_mindiff_hc,
who_maxdiff_hc, mindiff, maxdiff,
mindiff_prior, maxdiff_prior,
diff_prev, diff_next,
bef.g.aftm1, aft.g.aftm1,
val_excl, val_excl_code, absval,
d_agedays, ewma.all, ewma.before,
ewma.after, dewma.all,
dewma.before, dewma.after,
exp_vals) live on the closure-local df and are
discarded when the closure returns — only exclude is
written back to data.df. The top-level sp_key
column is dropped explicitly after the step.
Checklist findings
- Boundaries:
diff < mindiff and
diff > maxdiff — strict inequalities. WHO interval
boundaries use >= / < consistently. The
Tanner and formula-side boundaries (< 365.25 /
>= 365.25) are continuous at gap = 1 year so the
boundary assignment matches either limit.
.child_valid() call: Excludes temp
SDEs, excludes weight.
- Sort order: Explicit
order(agedays, internal_id) inside each while-loop
iteration; internal_id is the deterministic tiebreaker
(same-day rows don’t reach Step 17 because SDEs are resolved
upstream).
- Parameter scope: HT and HC only. HC has separate
velocity tables, tighter tolerances, and no Tanner.
- Factor levels:
Exclude-C-Abs-Diff
exists in exclude.levels.