Carried-forward value rescue

What a carried-forward value is

A carried-forward (CF) value is a measurement that is pulled forward from a prior measurement rather than being measured on the day of the encounter. Left in the data, such values distort trajectories, affecting both cleaning and analysis.

growthcleanr attempts to detect and exclude these by identifying values exactly equal to the measurement on the immediately prior day for the same subject and parameter, referred to below as “consecutive identical values” or “potential CFs”. Truly independent consecutive identical measurements occur, especially when measurements are rounded, estimated, or taken at short intervals, and in older children or in children with poor growth.

We first identified CFs as a problem when we saw children with strings of identical values over years followed by a quick rise. Even when measurements are at short intervals, if measurements are taken with recommended precision (nearest 0.1 cm or 0.1 kg, with more precision in young infants), inherent variation in measurement means that most consecutive measurements will not be identical. A substantial number of CFs can cause significant systematic bias. Therefore, in earlier versions of growthcleanr, the algorithm was aggressive and removed all potential CFs. However, this results in overexclusion of true consecutive identical measurements, which may be common in datasets with features listed above (many rounded or estimated measurements, short intervals, older children, or children with poor growth). Overexclusion can also lead to bias.

Distinguishing between independent consecutive identical measurements and carried-forward values is more difficult in some circumstances than identifying implausible values. We believe this current method for likely CF identification is an improvement on the prior aggressive rule, but we anticipate that it may evolve with further work.

Throughout this vignette, “CF” is used as shorthand for a potential CF: a consecutive identical value that detection has provisionally excluded (Exclude-C-CF) pending the rescue decision. This terminology is used primarily for historical reasons, as that is how these values have been referred to throughout development of the algorithm. A value that is not rescued stays excluded and is a confirmed carried-forward value; a rescued value is treated as an independent measurement.

To distinguish identical independent measurements from carried-forward measurements, Child Step 6 does three things:

1. Detects consecutive identical values and organizes them into strings — a string begins with a non-CF originator (a real measurement) and continues through the consecutive potential CFs that copy it forward. These values are provisionally excluded as CFs.
2. Decides whether each string is more likely to be an independently measured consecutive identical value or a value carried forward from prior.
3. Rescues plausible strings back to Include, leaving implausible ones excluded as Exclude-C-CF.

Determining criteria for rescuing potential CFs is among the most difficult tasks in automated or manual data cleaning in large part because they are dependent on practice patterns. In some datasets, carried-forward values are extremely common and can significantly impact analysis and cleaning. Rounding increases the likelihood of consecutive identical independent measurements. In other datasets, the vast majority of consecutive identical values are truly independent.

Scope	All child parameters (WEIGHTKG, HEIGHTCM, HEADCM)
Operates on	Valid rows (including temporary SDEs), restricted to subject-parameter combinations that have at least one duplicate raw value
Prior step	Child Step 5 (Temporary SDEs)
Next step	Child Step 7 (PIV)
Exclusion code	Exclude-CF (Summary, in exclude) / Exclude-C-CF (Detailed, in exclude_detailed)
Rescue markers	Rescued, Rescued-All (in the internal cf_rescued column)
Controlled by	cf_rescue ("standard" default / "none" / "all") and cf_detail (diagnostic columns)
Code location	CF logic is inline in cleanchild() in R/child_clean.R

---

Rescue modes

The cf_rescue argument selects one of three modes:

• "standard" (default) — decide each CF string on its last CF using the deltaz/nextz rule (below) and apply that decision to the whole string.
• "none" — no rescue. Every detected CF stays Exclude-C-CF.
• "all" — every detected potential CF is rescued, including one that shares a day with another non-CF Include. This preserves the caller’s “treat consecutive identical values as plausible” intent; any resulting multiple-Include same-day groups are resolved later by Child Step 13 (final SDE resolution). Some consecutive identical values may be excluded later by other steps.

Caution with "all". Keeping every consecutive identical value can lead to real measurements being excluded. For example, a boy has a real height of 100.5 cm at 3.5 years; that value is copied forward at 4, 4.5, 5, and 5.8 years, and a real height of 119.4 cm is recorded at 6 years (z = recentered z-score, tbc.sd):

Age (y)	Height (cm)	z	"standard"	"all"
3.5	100.5	0.19	Include	Include
4.0	100.5	−0.64	Exclude-CF	Include (CF-R)
4.5	100.5	−1.41	Exclude-CF	Include (CF-R)
5.0	100.5	−2.07	Exclude-CF	Include (CF-R)
5.8	100.5	−3.03	Exclude-CF	Include (CF-R)
6.0	119.4	0.49	Include	Exclude-Pattern

Under "standard", the four copies are excluded and both real heights are kept. Under "all", the copies are kept, so the child appears to stop growing for about 2.3 years. The copied heights are close enough together that the consistently decreasing z-scores seem plausible, and the real 6-year height is then excluded because it no longer fits the decreasing z-score trajectory.

Carried-forward values are very challenging to address, and no mode is right for every patient or dataset.

---

Detection

The standard rescue rule

Overall, rescue is designed to identify children for whom age, measurement interval, or growth trajectory make it plausible that a consecutive identical value is an independent measurement that happens to be identical, rather than a carried-forward value. We explored incorporating rounding into this rule but did not identify a satisfactory method to include it.

Standard mode evaluates only eligible CFs (seq_win > 0 and not sharing a day with an Include). Two quantities drive the decision, both computed on the un-recentered score sd.orig:

• cf_deltaz = |CF z − originator z| (≥ 0) — how far the carried value has drifted from the real value it copies.
• cf_nextz = signed (next − this) change from the CF to the next Included value within the subject-parameter — negative when the trajectory continues downward past the CF, positive when it turns back up.

Why this rule

• Detection is exact equality of raw measurements. 121.92 is not considered identical to 121.9200001.

• Trajectory shape, not distance alone. A consecutive identical value is plausible when it lies on a genuinely flat or declining trajectory (stagnant growth) and implausible when it is a flat notch of significant length in an otherwise rising trajectory (a copied value the child has since grown past). The sum cf_deltaz + cf_nextz captures this: it equals twice how far the carried value’s z-score sits below the midpoint of its originator’s and its next real value’s z-scores (which coincides with the straight line joining them when the CF falls at the midpoint in age). The sum is small or negative for a smooth decline or true stagnation (keep) and large and positive for a deep notch that then recovers (a true CF).

• The cf_deltaz <= 1.5 cap prevents a single large carried drop from being rescued on the strength of a steeply rising next value alone. Intentionally, a very large drift in z-score from the originator cannot be rescued no matter how far the trajectory later climbs.

• The age split at 13 years (end of series). With no next value to define the trajectory’s direction, the rule falls back to how far the CF has drifted, with a looser allowance for adolescents (whose z-scores drift more) than for younger children.

• Originator z-score is taken before gestational-age correction because it sets the expected scale of random variation, and GA correction is applied uniformly to all of a preterm subject’s values in a string, so it does not change the difference between them.

• "all" mode ignores the same-day-Include collision so that the caller’s intent (“treat CFs as plausible”) overrides the usual rule against multiple Includes per same-day group; that group is then resolved by Child Step 13 on its merits.

---

Interaction with same-day extraneous values

Child Step 6 shares data with the same-day extraneous (SDE) machinery in two ways:

• Temporary SDE re-evaluation. After CF detection, any rows still flagged Exclude-C-Temp-Same-Day may have a stale Step-5 selection now that some same-day Includes have become CFs. Step 6 resets those rows to Include and re-runs the temporary-SDE selection (identify_temp_sde(), the same routine used in Child Step 5) so the choice accounts for the CF removals.

• SDE-Identical values. Rows flagged Exclude-C-Identical in Early Child Step 13 would break the positional string logic (which relies on adjacent-row comparisons). They are set aside during string construction and restored afterward, then re-sorted by (subjid, param, agedays, internal_id).

---

Diagnostic output (cf_detail = TRUE)

Setting cf_detail = TRUE adds six columns to the output. The first three describe CF status and the rescue decision; the last three are development aids for tuning CF logic.

Column	Values / meaning
cf_status	"Orig" (string originator), "CF-R" (carried forward and rescued (not excluded as CF); a later step may still exclude it), "CF-NR" (carried-forward, not rescued), "CF-E" (same-day-extraneous potential CF — shares a day with an Include, never rescue-eligible; in cf_rescue = "all", same-day CFs are not excluded at this step and are labelled CF-R, with cf_deltaz NA; Child Step 13 then chooses one value for the day, so such a row can end as Exclude-Extraneous), "Not-CF" (no CF relationship), "Adult" (adult row)
cf_deltaz	|CF z − originator z| for positionally-detected CFs; NA for Orig, Not-CF, CF-E, and Adult rows. The whole string is decided on its last carried-forward value; values shown on earlier rows are informational.
cf_nextz	signed (next − this) z-change from a CF to the next Included value within the subject-parameter; populated for CF-R and CF-NR; NA for Orig, Not-CF, CF-E, Adult, and end-of-series rows. The whole string is decided on its last carried-forward value; values shown on earlier rows are informational.
imperial	(design aid) whole/half imperial-unit flag — whole pounds for WEIGHTKG, whole/half inches for HEIGHTCM/HEADCM (tolerance 0.01 imperial unit); computed for every row; NA on missing-measurement and Adult rows
cf_dage_orig	(design aid) day gap from a CF’s originator to the CF (agedays_CF − agedays_orig; always positive); populated for CF-R/CF-NR (not same-day CFs in cf_rescue = "all", which are labelled CF-R but have NA); NA otherwise
cf_dage_next	(design aid) day gap from a CF to the next eligible Included value (the agedays analogue of cf_nextz); NA for Orig, Not-CF, CF-E, Adult, and end-of-series rows

To count rescued values, run with cf_detail = TRUE and count rows with cf_status == "CF-R". In cf_rescue = "all", same-day CFs are also labelled CF-R, so in that mode the count includes them.

The imperial flag and the originator interval (cf_dage_orig) are computed only when cf_detail = TRUE, alongside the other diagnostic columns, and surface only through them; they do not affect the rescue decision.

The internal cf_rescued column (values "", "Rescued", "Rescued-All") tracks the rescue outcome during processing but is not returned in the default output.

---

Arguments

Argument	Default	Role
cf_rescue	"standard"	Rescue mode: "standard" (deltaz/nextz rule, decided per string’s last CF), "none" (exclude all detected CFs), "all" (rescue all detected CFs)
cf_detail	FALSE	If TRUE, add the six diagnostic columns above to the output

The standard-mode thresholds (1, 1.5, 0.5, and the 13-year split) are set inline in the Child Step 6 rescue block in R/child_clean.R. For the full list of columns cleangrowth() can return, see vignette("output"); for the exclusion codes, see vignette("exclusion-codes").

---

Determinism notes

• All CF sorts include internal_id as the final tiebreaker (setkey(data.df, subjid, param, agedays, internal_id) at the top of cleanchild(), and the same order inside the CF block and after the SDE-Identical restoration), so string construction and the “next Included value” lookup are deterministic.
• Detection compares raw values (v.orig) by exact equality; the rescue quantities use sd.orig (the pre-GA-correction CSD score).
• Child Step 6 does not use the designated other parameter (DOP) and does not perform age-dependent birth tiebreaking (that lives in Early Child Step 13 and Child Step 5).