Changes that may simplify your workflow

These are new options. Code that does not use them does not need to change.

New input formats are supported. growthcleanr previously only worked for long data (one row per observation, with weight and height on separate rows). It can now handle semi-wide data (one row per visit, with weight, height, and head circumference on the same row) and wide data (one row per subject). Set input_format = "semi_wide" or input_format = "wide"; results come back in the same layout unless you set output_format. For ready-to-run scripts, see “Semi-wide data (one row per visit)” and “Wide data (one row per subject)” in vignette("example-scripts"). For full detail, see “Input format: long, semi-wide, and wide” in vignette("wrapper"). The Shiny app also handles these formats.

Additional input recognized. For example, date of birth (dob) and visit date (visit_date) may now be entered instead of agedays. sex now accepts the following by default: 0=male, 1=female, M/F, Male/Female, B/G, Boy/Girl, any capitalization, as well as custom codes. For ready-to-run scripts, see “Use dates instead of age in days” and “Set your sex coding (including 1 = male, 2 = female)” in vignette("example-scripts"). For full detail on accepted date formats, see “Date parsing” in vignette("wrapper"). The Shiny app also handles dates and custom sex codes.

Install the updated version

Note If you already have an older growthcleanr installed, a) detach it if it is loaded detach("package:growthcleanr", unload = TRUE), b) uninstall it remove.packages("growthcleanr"), c) restart R (important!), d) and then install using the instructions below.

growthcleanr version 3 is available from CRAN:

install.packages("growthcleanr")

Basic command

New option: pass a data frame directly (instead of vectors).

If your data is in standard growthcleanr format (described below) use the code below. For ready-to-run scripts for other formats (semi-wide, wide), column names, coding, etc. see vignette("example-scripts"); for parallel processing and checkpointing on larger datasets, see vignette("large-datasets").

gc_runtrial(my_data)

result <- cleangrowth(my_data)   # growthcleanr now takes an optional id column (unique per row); without one, ids are generated and a warning is shown

result <- cleangrowth(my_data,
                      checkpoint_dir = "gc_checkpoints",
                      parallel = TRUE)

Default columns

By default, cleangrowth() returns all of your entered columns and the additional columns below. If you did not enter an id column, a generated id is also returned.

Extra input columns are carried through by default. With carry_through = TRUE (the default), growthcleanr returns the six input columns (renamed to your names) and the result columns alongside any other columns from your input, so the results attach directly to your full dataset. Set carry_through = FALSE to return only the six input columns plus the results; in that case, keep your id column and merge on it to reattach the results. An input column with the same name as a growthcleanr output column is returned as userinput_<name>.

Command and output

• A Shiny app (growthcleanr_app()) for cleaning data without writing R code. See the Shiny app how-to guide.
• growthcleanr now accepts data in semi-wide and wide formats. See the R interface how-to guide, Changes that may simplify your workflow, “Semi-wide data (one row per visit)” and “Wide data (one row per subject)” in vignette("example-scripts"), and “Input format: long, semi-wide, and wide” in vignette("wrapper").
• Users may input date of birth and visit date rather than agedays. See “Use dates instead of age in days” in vignette("example-scripts") and “Date parsing” in vignette("wrapper").
• The command-line driver, which cleans a CSV file from a terminal or batch job without writing R code, was updated for the new options and gives the same results as the equivalent R call. See vignette("command-line"). If you have shell scripts that call the 2.2.3 driver, see item 6 in Updating code written for growthcleanr 2.2.3.
• cleangrowth() returns a data.table, not a vector.
• More user-friendly exclusion codes, reported at three levels — a binary recommendation (bin_exclude), a short Summary set (default), and a more granular Detailed set (opt-in). Details
• Addition of an optional id variable that ensures deterministic, replicable handling of borderline values across multiple runs of growthcleanr on the same dataset.
• growthcleanr accepts M/F, male/female, B/G, and boy/girl in the sex column without any specification (in addition to 0=male, 1=female). It is not case-sensitive. M/F and B/G can also be used in the package cdcanthro that calculates z-scores, allowing use of growthcleanr and cdcanthro on the same dataset without conversion. Other codings can be set with sex_coding.
• A trial run of a small sample of your data, gc_runtrial(my_data), reports how your columns, sex coding, and parameters were read. See Check your setup first.

Efficiency

• A new feature, checkpointing, reduces memory use; with it, run-time for larger datasets increases almost linearly. Checkpointing is on by default: for datasets with more than 5,000 subjects, growthcleanr cleans in checkpoints saved to a temporary folder and deletes them when the run finishes. Set checkpoint_dir to keep the checkpoint files so an interrupted run can resume, or checkpoint = FALSE to write no intermediate files to disk. See vignette("large-datasets").
• Parallel processing is simpler to use: set the parallel argument to TRUE. Parallel processing and checkpointing can be used together.
• Multiple internal efficiency improvements invisible to the user.

Changes to child algorithm

• Improved handling of infants: The prior pediatric algorithm was optimized for children 2 years of age and older. 2.2.3 contained a preliminary version of an algorithm with new features for infants, including infants whose growth pattern is consistent with prematurity or small-for-gestational age. There is now a single combined child version incorporating modifications for infants.
• The infant handling includes “correction” for an assigned gestational age when early measurements are very low. The correction is designed to improve handling for babies who are premature or small for gestational age.
• Addition of head circumference. Head circumference was only evaluated by the preliminary infants algorithm in 2.2.3, which was not recommended for research use; for practical purposes HC cleaning is new in version 3.
• Improved handling of carried forwards. Many consecutive identical values that would previously have been excluded as CFs are now “rescued” if it seems reasonably likely that they were independently measured rather than carried forward. Setting thresholds for rescue is extremely challenging because the performance of different thresholds varies widely based on site-specific practices, so these methods may continue to evolve.
• Improved recentering. Rather than deriving recentering medians from your data (or from NHANES data for smaller datasets), this version of growthcleanr uses built-in smoothed recentering medians from multi-center data which provide protection from areas of low visit density affecting recentering and therefore results. You can supply your own recentering medians with sd.recenter =, or turn recentering off with sd.recenter = "none". See vignette("recentering") for how the defaults were built and how to customize. In the past, growthcleanr could produce different results for an individual child based on the growth patterns of other children in the dataset. Results are now reproducible regardless of the size and makeup of the dataset.

Changes to adult algorithm

• Refined “wtallow” formulas that determine allowed degree of weight deviation.
• Loosening of height variation limits. (The default height window is looser.)
• A redesigned adult algorithm with four permissiveness levels, set via adult_permissiveness: "loosest", "looser" (the default), "tighter", and "tightest". These set all adult thresholds at once; the tightest level is intended for evaluations of relatively healthy populations. permissiveness_presets() returns the settings for each level. Of the four, "looser" (the default) is the most similar to the prior adult algorithm, though none of them reproduce it exactly. For the intended use of each level, see Section 1D of vignette("configuration"). For a ready-to-run script, see “Data that includes adults” in vignette("example-scripts"). To change individual adult thresholds without switching level, use adult_overrides (see ?cleangrowth and “User overrides” in vignette("adult-algorithm")).
• When a “weight cap” (now referred to as “scale max”) was set, 2.2.3 excluded all of a subject’s weights when one was recorded at the scale maximum; weights at the cap that fit the subject’s trajectory are now kept.
• Removal of steps that identified specific types of errors (e.g., hundreds). These contained bugs, were inefficient, and did not improve accuracy.

Changes to both algorithms

• New “evil twins” step that excludes consecutive erroneous measures that were missed previously.
• Redesigned handling of more than one measurement on the same day. The prior adult algorithm often excluded all values when there was more than one measurement per parameter on the same day. This has been fixed. When comparing results with 2.2.3, compare by subject, parameter, and day, not by row.
• More customization available, especially for adults. See vignette("configuration") for more details.
• Removal of unit error/swap correction for children and similar “special cases” code in adult algorithm. They were lower-yield and error-prone. Detection of errors from these causes is now folded into the rest of the algorithm.
• Addition of the unit_error_range output column to provide an indicator of whether a value is in a common range for unit errors. This column is calculated independently of exclusions and is not a definitive designation of a unit error. Patients with consistent atypical growth cannot always be distinguished from patients with unit errors. In other words, if a patient’s data is entirely in the wrong units, longitudinal cleaning can mistake them for children with consistent atypical growth. If a large proportion of values have unit_error_range = TRUE, examine your data to look for unit errors. The unit error range check is a useful indicator but can be wrong. Do not assume that these values should be corrected to other units. Some values are plausible in either unit, so they are never flagged. Other true values may be flagged, such as in data with measurements from premature infants. Importantly, many types of errors can result in a value that appears to be a unit error; “correcting” these for unit errors will result in erroneous values.
• Adjusted the limits beyond which all values were excluded. These were previously called BIVs and are now called hard-limits.
• The longwide() command was present in 2.2.3 but not widely known. It facilitates conversion of long data to semi-wide format (height and weight on one row) for calculation of BMI. For children, this can be used with the ext_bmiz() command to calculate extended BMI z-scores in growthcleanr. growthcleanr now also accepts and returns semi-wide and wide layouts directly, via input_format and output_format.
• Other refinements that improve performance but would mostly be invisible to users

Optional output columns

Additional output columns can be requested with arguments such as exclude_detail, cf_detail, and full_detail. See vignette("configuration") for the full list and vignette("output") for what each column contains.

Inspecting or troubleshooting a result

If you want to understand why a particular value was excluded, learn more about your dataset, or customize your use of growthcleanr output, these options add diagnostic columns (all off by default):

• exclude_detail = TRUE – the finer Detailed exclusion code, which names the reason more specifically than the Summary code.
• cf_detail = TRUE – for carried-forward decisions, the deviation values behind a rescue-or-exclude call.
• corr_detail = TRUE – gestational-age correction status.
• recenter_detail = TRUE – the recentered z-scores the child algorithm used (specialized output for a related tool in development).
• debug = TRUE – the full set of intermediate z-scores, mostly used for debugging and algorithm refinement.
• display_gc_id = TRUE – the internal row id growthcleanr used, useful when tracing same-day tiebreaks.
• full_detail = TRUE – turns on all of the above at once, plus tri_exclude and internal_detail.

cf_detail, corr_detail, recenter_detail, and the debug z-scores apply to child rows only; for adult rows, use exclude_detail = TRUE.

For a ready-to-run script, see “See more detail about why values were excluded” in vignette("example-scripts").

Why the codes changed

The prior codes indicated which step a value was excluded in but were overly detailed for the vast majority of users and used unnecessarily jargon-y descriptions. We aimed for exclusion codes with clearer names in fewer categories that are more relevant to researchers.

To support different needs, the updated version reports exclusions at three levels:

• Binary - growthcleanr’s binary recommendation: Include or Exclude
• Summary (the default exclude column) — a small set of clear, consolidated reasons that are more aligned with what researchers would want to report.
• Detailed (opt in with exclude_detail = TRUE) — finer gradations than the Summary codes, but still organized around user-relevant reasons rather than internal step mechanics. Most Detailed codes keep a -C- (child) or -A- (adult) marker showing which algorithm assigned them (e.g. Exclude-C-Pair). Every Detailed code matches to one Summary code.

The Summary exclusion codes

These are the values that can appear in the default exclude column. Exclusion codes no longer designate whether they are for Adults or not (this is available in the detailed codes). More detailed mapping of how prior codes match to these codes is available below.

“Hard limits” instead of “biologically implausible”

In prior versions, values far from population norms were labeled biologically implausible (BIV). The updated version calls these hard limits. The name change is driven primarily by the new adult permissiveness levels. When using any but the loosest set of limits, the hard limits will intentionally exclude biologically plausible values.

The Exclude-Single category uses different limits and methods than the hard-limit step: its limits consider the subject’s other parameter when available (child: the other parameter’s z-score; adult: a same-day BMI).

Mapping from prior exclusion codes

If your code refers to specific exclusion-code names, use the mapping below. The Summary column is what now appears in exclude by default.

Child (pediatric) measurements:

This lists the codes from the default algorithm (the preliminary infants algorithm, an opt-in that was not recommended for research, has been removed).

Exclude-SD-Cutoff excluded values whose recentered or unrecentered z-score was more extreme than ±25 (the sd.extreme and z.extreme defaults). It maps to Exclude-Hard-Limit; the updated hard-limit step uses different limits (see Child Step 7 in vignette("child-algorithm") and Section 1B of vignette("configuration")). Exclude-Too-Many-Errors-Other-Parameter has no successor: the updated algorithm no longer propagates a high error load from one parameter to the other.

Adult (Exclude-Adult-*) measurements:

Exclude-Scale-Max (adult scale maximum) was previously Exclude-Adult-Weight-Cap and Exclude-Adult-Weight-Cap-Identical.

Adult codes with no direct successor. The updated algorithm removed the standalone digit-error, transposition, and swap detection steps, so these prior codes no longer occur: Exclude-Adult-Hundreds (+ -RV), Exclude-Adult-Transpositions (+ -RV), Exclude-Adult-Swapped-Measurements, and Exclude-Adult-Possibly-Impacted-By-Weight-Cap. A genuinely extreme value that one of these once flagged should still be excluded by the Evil Twins (Exclude-Extreme), hard-limit (Exclude-Hard-Limit), or trajectory (Exclude-Pattern) steps.

Child codes with no direct successor. Unit-error and swap detection were removed from the algorithm, so these prior codes no longer occur: Unit-Error-High, Unit-Error-Low, Unit-Error-Possible, Swapped-Measurements. (An informational unit_error_range output column flags values consistent with a unit error, but the algorithm no longer corrects or excludes on that basis.)

New codes

These have no prior-version equivalent: Exclude-Not-GC-Param, Exclude-HC-Out-of-Range (head circumference after 5 years), Exclude-Identical (same-day identical values are now split out from other extraneous values for children; adults already had Exclude-Adult-Identical-Same-Day), and the extreme-pair step that feeds Exclude-Extreme.