Package {canpumf}

Repair command-file labels from a validated PDF dictionary

Description

Compares every variable and value label in the merged command-file metadata against the PDF dictionary, fills in labels the command file left blank, replaces labels the PDF demonstrably extends (the upstream-truncation fingerprint), and records every divergence – acted on or not – in a ledger.

Usage

.pumf_apply_pdf_repairs(metadata, pdf, validation)

Arguments

Details

A repair is withheld only where the frequency check actively contradicts the PDF parse (status '"mismatch"'); a variable the check could not reach is not evidence against it. Each ledger row carries the variable's validation status so corroborated repairs are distinguishable from merely uncontradicted ones.

Value

List with elements 'metadata' (possibly modified) and 'repairs' (the ledger tibble).

Pick the guide that matches this module's layout

Description

Detection shortlists candidates on filename and block count, which cannot separate a release that ships both its original and its revised user guide: PALS 2006's two English guides document the same 746 variables, but the original's fields all sit eight columns to the left of the revised file's. Scoring each candidate's 'Position:'/'Length:' headers against the command file's layout picks the right one – and does so per language, so the English and French guides of the same edition are chosen together.

Usage

.pumf_pdf_choose_candidates(pdf_paths, layout)

Arguments

Value

'pdf_paths' with 'eng'/'fra' possibly replaced.

Compare the guide's field positions against the command file's layout

Description

The structural corroboration channel, independent of the frequencies: each guide block prints the variable's 'Position:' (and usually 'Length:'), and the command file's layout gives the same two numbers. Agreement across the whole document is what tells "the right guide whose counts were tabulated on a different population" apart from "the wrong guide".

Usage

.pumf_pdf_position_agreement(pdf, layout)

Arguments

Value

List with 'n' (blocks that could be compared), 'agree', 'rate' ('NA' when nothing was comparable) and 'ok', a logical vector aligned to 'pdf$variables'.

Resolve remaining duplicate blocks by which one reconciles with the data

Description

After [.pumf_pdf_select_blocks()] a name can still map to several blocks – typically a respondent key that sits at the same offset in every module of a multi-module survey. The frequencies settle it: only the block describing the file actually being parsed reconciles against it. Where none does, the first block is kept and its (mismatching) status stands, so nothing is repaired from a block we could not confirm.

Usage

.pumf_pdf_resolve_duplicate_blocks(pdf, validation)

Arguments

Value

List with the filtered 'pdf' and 'validation' (one row per name).

Filter guide blocks by the layout's field position

Description

A user guide covering several linked files documents each of them in turn, so a shared respondent key ('RECID', 'PERSONID') gets one block per module – each with that module's own frequencies. Keyed by name alone the first block wins, which then contradicts every other module's data file.

Usage

.pumf_pdf_select_blocks(pdf, layout)

Arguments

Details

This is the cheap first pass: the block header prints 'Position:' and ‘Length:', and the command file’s layout gives the same two numbers for this module, so a name documented at different offsets in different modules is resolved here. It cannot separate blocks that share an offset (every module's file starts 'RECID' at position 1) – those go to [.pumf_pdf_resolve_duplicate_blocks()], which decides on the frequencies.

Value

'pdf', with 'variables'/'codes'/'freqs'/'ranges' filtered to the surviving blocks.

Find the registered sibling whose config best fits an unregistered year

Description

Considers only plain four-digit-year keys ('series/2023') so multi-part versions (Census '1971/individuals_prov') never inherit across types. Prefers the newest sibling not later than 'version'; if the requested year predates every entry, falls back to the oldest registered sibling.

Usage

.pumf_registry_newest_sibling(series, version)

Value

the chosen sibling version string, or 'NULL' if no year-keyed sibling

Validate a PDF frequency parse against the microdata

Description

For every variable the PDF dictionary describes, compares the per-code frequencies printed in the guide against a tabulation of the actual data file. This is what turns the PDF from an unverifiable transcription into a checkable source: a variable whose counts do not reconcile is never used to repair a label.

Usage

.pumf_validate_pdf_freqs(pdf, layout, data_path, data_encoding = "CP1252")

Arguments

Value

Tibble with columns 'block', 'name', 'status', 'n_codes', 'n_matched', 'note' – one row per guide block, so a name documented once per module of a multi-module guide gets one row per module.

Generate bootstrap weights for a PUMF dataset

Description

For a **DuckDB-backed lazy table** (the typical case), bootstrap replicate weights are written directly into the DuckDB file as a separate table and exposed through a persistent VIEW that joins the main survey table with the BSW columns. The returned 'tbl' references this view, so all downstream dplyr operations have access to every replicate.

Usage

add_bootstrap_weights(
  tbl,
  weight_col,
  id_col = NULL,
  strata_cols = NULL,
  n_replicates = 500L,
  prefix = "CPBSW",
  bsw_table = NULL,
  seed = NULL,
  overwrite = FALSE
)

Arguments

Details

For an **in-memory 'data.frame' or 'tibble'**, bootstrap weights are generated entirely in memory and the augmented data frame is returned.

Bootstrap weights are generated by the rescaled bootstrap: for each replicate a sample of n rows is drawn with replacement; the bootstrap weight for row i in replicate b is 'original_weight[i] * count[i,b]', where 'count[i,b]' is the number of times row i appeared in draw b.

**Incremental re-runs (DuckDB path):** when a BSW table already exists the call only does the work needed to satisfy the request: * **More replicates** than stored (and no new rows): the additional replicate columns are appended; existing columns are kept. * **New rows** in the main table (some rows have no weights yet): because a bootstrap replicate resamples the full population, added rows invalidate the existing weights of their resampling universe, so those weights are deleted and regenerated. Unstratified, this regenerates every row; when 'strata_cols' are in effect, only the strata that gained rows are regenerated and complete strata keep their existing weights. * **Neither:** the stored weights are reused without recomputation. Pass 'overwrite = TRUE' to force a full fresh regeneration regardless.

**Multiple weight columns (hierarchical data):** by default 'bsw_table' is named after 'weight_col' (e.g. '"pumf_bsw_wstpwgt"'), so calling the function twice with different weight columns (e.g. household weight and person weight) produces two independent BSW tables and two separate views without any conflict.

**Connection note (DuckDB path):** calling this function fully shuts down the DuckDB in-process instance held by 'tbl' (because a write connection requires exclusive access). The input 'tbl' and any other lazy tables backed by the same DuckDB file become invalid after the call. Use the returned tbl instead.

**Filtered input tbls (DuckDB path):** bootstrap weights always cover the complete physical survey table. If 'tbl' has dplyr 'filter()' operations applied, they are captured and automatically re-applied to the returned VIEW tbl so the visible rows match the original subset. Other operations ('select()', 'mutate()', etc.) are not replayed – they would interfere with the BSW columns – so apply them manually to the returned tbl if needed.

**ID column (DuckDB path):** a stable row identifier is needed to link the main table to the BSW table. If 'id_col' is 'NULL' (the default): * The survey registry 'bsw_join_key' is used when available (e.g. '"PEFAMID"' for SFS 2016-2023) – no table modification needed. * Otherwise a 'pumf_row_id' column (DuckDB 'rowid') is added to the main survey table. The 'ALTER TABLE ADD COLUMN' is O(1); the 'UPDATE' that fills the values is O(n).

Value

* **DuckDB path:** a lazy 'dplyr::tbl()' backed by a persistent DuckDB VIEW that contains all original survey columns plus the 'n_replicates' bootstrap weight columns, with any input 'filter()' operations re-applied. * **In-memory path:** the input 'data.frame' / 'tibble' with bootstrap weight columns appended so that 'n_replicates' replicates are present. If the input already carries replicate columns for 'prefix', only the additional ones are generated (existing columns are preserved); when it already has at least 'n_replicates', the data frame is returned unchanged.

See Also

[bsw_info()], [remove_bootstrap_weights()], [get_pumf()]

Examples

sfs <- get_pumf("SFS", "2019")
if (!is.null(sfs)) {
  sfs_bsw <- add_bootstrap_weights(sfs, weight_col = "PWEIGHT",
                                   n_replicates = 200L, seed = 42L)
  bsw_info(sfs_bsw)
  close_pumf(sfs_bsw)
}

Add a harmonised gender/sex column to an LFS table

Description

LFS introduced 'GENDER' (with values '"Men+"' / '"Women+"' / '"Non-binary persons"') to replace the binary 'SEX' variable ('"Male"' / '"Female"') starting in 2020. In any given row exactly one of the two columns is non-'NA'. 'add_lfs_GENDER_SEX()' coalesces them into a single harmonised column, recoding 'SEX' values to the 'GENDER' scale so the result is consistent across all LFS vintages.

Usage

add_lfs_GENDER_SEX(tbl)

Arguments

Details

Works on both unlabelled tables (columns 'SEX' / 'GENDER', output column named 'GENDER_SEX') and labelled tables produced by [label_pumf_columns()] (columns '"Sex of respondent"' / '"Gender of respondent"', output column named '"Gender/sex of respondent"').

The mapping applied to 'SEX' / '"Sex of respondent"' when the gender column is 'NA':

• '"Male"' \rightarrow '"Men+"'

• '"Female"' \rightarrow '"Women+"'

The output column is inserted after 'GENDER' / '"Gender of respondent"' when present, or after 'SEX' / '"Sex of respondent"' otherwise.

Value

The same lazy table with a new harmonised gender/sex column.

See Also

[get_pumf()], [label_pumf_columns()], [add_lfs_SURVDATE()]

Examples

lfs <- get_pumf("LFS")   # NULL if StatCan is unreachable
if (!is.null(lfs)) {
  # Unlabelled
  lfs |> add_lfs_GENDER_SEX() |>
    dplyr::count(SEX, GENDER, GENDER_SEX) |> dplyr::collect()

  # Labelled
  lfs |> label_pumf_columns() |> add_lfs_GENDER_SEX() |>
    dplyr::count(`Sex of respondent`, `Gender of respondent`,
                 `Gender/sex of respondent`) |> dplyr::collect()

  close_pumf(lfs)
}

Add a date column to an LFS table

Description

Creates a date column set to the first day of the survey month and inserts it immediately after the survey month column. Works on both unlabelled tables (columns 'SURVYEAR' / 'SURVMNTH', date column named 'SURVDATE') and labelled tables produced by [label_pumf_columns()] (columns '"Survey year"' / '"Survey month"', date column named '"Survey date"').

Usage

add_lfs_SURVDATE(tbl)

Arguments

Value

The same lazy table with a new date column positioned after the survey month column.

See Also

[get_pumf()], [label_pumf_columns()], [add_lfs_GENDER_SEX()]

Examples

lfs <- get_pumf("LFS", "2023")   # NULL if StatCan is unreachable
if (!is.null(lfs)) {
  # Unlabelled
  lfs |> add_lfs_SURVDATE() |> dplyr::select(SURVYEAR, SURVMNTH, SURVDATE) |>
    dplyr::distinct() |> dplyr::collect()

  # Labelled
  lfs |> label_pumf_columns() |> add_lfs_SURVDATE() |>
    dplyr::select(`Survey year`, `Survey month`, `Survey date`) |>
    dplyr::distinct() |> dplyr::collect()

  close_pumf(lfs)
}

Summarise bootstrap weight tables present in a PUMF DuckDB database

Description

Queries the DuckDB file backing a PUMF lazy table for bootstrap weight tables created by [add_bootstrap_weights()] and returns a one-row-per-table summary tibble. Returns an empty tibble (invisibly) when no BSW tables are found.

Usage

bsw_info(tbl)

Arguments

Value

A tibble with columns:

'weight_col'

The weight column the BSW table was built from (matched back to the case used in the main survey table).

'bsw_table'

Name of the DuckDB table storing the weights.

'view_name'

Name of the DuckDB VIEW joining survey + BSW.

'view_exists'

Whether the companion VIEW is present.

'n_replicates'

Number of bootstrap replicate columns.

'size_mb'

Estimated table size in megabytes (from DuckDB metadata; 'NA' when unavailable).

See Also

[add_bootstrap_weights()], [remove_bootstrap_weights()]

Examples

sfs <- get_pumf("SFS", "2019")
if (!is.null(sfs)) {
  sfs_bsw <- add_bootstrap_weights(sfs, weight_col = "PWEIGHT", seed = 1L)
  bsw_info(sfs_bsw)
  close_pumf(sfs_bsw)
}

Warn when French label coverage is partial

Description

Emits a warning when French files were clearly present (some label_fr are non-NA) but more than threshold of variable labels are missing. Surveys with no French files at all (all label_fr are NA) do not trigger the warning.

Usage

check_bilingual_coverage(metadata, threshold = 0.2)

Arguments

Value

NULL invisibly.

Close the DuckDB connection backing a PUMF lazy table

Description

Disconnects the DuckDB connection associated with 'x'. 'x' may be either a lazy 'dplyr::tbl()' returned by [get_pumf()] (the connection embedded in the tbl is closed) or a DuckDB connection object returned by [get_pumf_connection()] (closed directly). After calling this function the table or connection can no longer be queried.

Usage

close_pumf(x)

Arguments

Details

All lazy tables and sibling modules opened from one [get_pumf()] call share a single connection, so a single 'close_pumf()' on any of them releases it.

Closing is only necessary when you need to release the file lock – for example, before calling 'get_pumf(..., refresh = TRUE)' on the same survey, or before writing to the DuckDB from another process. Read-only connections (the default) do not block other readers.

Value

Invisibly 'NULL'.

See Also

[get_pumf()], [get_pumf_connection()]

Examples

sfs <- get_pumf("SFS", "2019")
if (!is.null(sfs)) {
  # ... analysis ...
  close_pumf(sfs)
}

# Also accepts a raw connection from get_pumf_connection()
con <- get_pumf_connection("SHS", "2017")
if (!is.null(con)) {
  DBI::dbListTables(con)
  close_pumf(con)
}

Detect parseable metadata formats in a PUMF directory tree

Description

Walks the directory tree under pumf_dir and returns every recognised command-file format. Multiple formats may be present in one release; all are returned so that the caller can run all applicable parsers and merge results.

Usage

detect_formats(pumf_dir, sps_mask = NULL)

Arguments

Value

A named list (possibly empty) whose elements are a subset of "lfs_csv", "cpss_csv", "sas_cards", "spss_split", and "spss_mono". Each element is either a single path string or (for spss_mono) a list(eng = ..., fra = ...) where fra is NULL when no French file is found.

Harmonised Labour Force Survey timeline, 1976 onward

Description

Stacks the historical monthly LFS files ('"LFS_HIST"', 1976 to 2005) and the current LFS files ('"LFS"', 2006 onward) into one lazy table with a curated common set of variables, so that long time series can be pulled with a single query.

Usage

get_lfs_timeline(
  lang = c("eng", "fra"),
  sources = .lfs_timeline_series,
  refresh = FALSE,
  cache_path = getOption("canpumf.cache_path", tempdir())
)

Arguments

Details

The two series keep their own DuckDB files. This function attaches both **read-only** to an in-memory DuckDB and returns a view over them, so it never blocks (and is never blocked by) other readers. By default it reads only what is already loaded. Load data first with, for example, 'get_pumf("LFS_HIST", "1995")' or 'get_pumf("LFS", "2015")', or pass 'refresh = "auto"' to bring both series up to date before opening the timeline.

The harmonised table has these columns: * 'SOURCE' ('"LFS_HIST"' or '"LFS"'), 'SURVYEAR' and 'SURVMNTH'. * Numeric variables in plain units. The hours and wage variables of the current LFS files carry implied decimals (tenths of hours, cents), which are removed here. 'FINALWT' is 'FWEIGHT' in LFS_HIST. * Categorical variables whose codes are the same in both series (e.g. 'PROV', 'AGE_12', 'COWMAIN', 'EFAMTYPE'). These carry the current LFS labels. * Recoded variables: - 'LFSSTAT': the three unemployed categories of LFS_HIST are collapsed. - 'GENDER_SEX': LFS_HIST 'SEX' and the current 'SEX'/'GENDER', on the 'GENDER' scale, as in [add_lfs_GENDER_SEX()]. - 'MARSTAT': four categories (married or common-law, single, widowed, separated or divorced). The files before November 1999 only have these four. - 'CMA': Montreal, Toronto, Vancouver or other. It is 'NA' before 1987, when LFS_HIST does not identify CMAs. - 'SCHOOLN': non-student, full-time or part-time student. - 'AGYOWNK': the four age groups of the youngest child in the current files. - 'NAICS_18': industry in the 18 groups of LFS_HIST. The 21 current groups are merged. - 'EDUC': the 1990 onward classification (LFS_HIST 'EDUC90'). It is 'NA' before 1990, whose categories do not map onto it. - 'WHYPT': reasons for part-time work from 1997 ('WHYPTNEW').

Occupation, immigration and the LFS_HIST-only family and spouse variables are not part of the harmonised table. Use [get_pumf()] on each series for those.

Rebasing: LFS_HIST weights are rebased to different Censuses by period (1987-1995 to 2001, 1996-2000 to 2006, 2001-2005 to 2011; 1976-1986 are not rebased). Levels can therefore jump at the seams between periods and at 2006.

Value

A lazy 'dplyr::tbl()' over the view 'lfs_timeline'. Categorical columns are factors. [label_pumf_columns()] and [pumf_var_labels()] work on it. Release it with [close_pumf()].

See Also

[get_pumf()], [add_lfs_SURVDATE()]

Examples

tl <- get_lfs_timeline()
if (!is.null(tl)) {
  tl |>
    dplyr::filter(SURVMNTH == 6L) |>
    dplyr::group_by(SURVYEAR, LFSSTAT) |>
    dplyr::summarise(persons = sum(FINALWT, na.rm = TRUE), .groups = "drop") |>
    dplyr::collect()
  close_pumf(tl)
}

Get a Statistics Canada PUMF dataset as a lazy DuckDB table

Description

Main entry point for the canpumf package. Downloads (if needed), parses metadata, applies bilingual labels, and returns a lazy 'dplyr::tbl()' backed by a DuckDB file in the cache directory. Subsequent calls reuse the cached DuckDB without re-downloading.

Usage

get_pumf(
  series = NULL,
  version = NULL,
  lang = "eng",
  cache_path = getOption("canpumf.cache_path", tempdir()),
  refresh = FALSE,
  redownload = FALSE,
  read_only = TRUE,
  registry = NULL,
  module = NULL,
  borealis = NULL,
  register_connection = getOption("canpumf.register_connection", TRUE),
  ...
)

Arguments

Details

The Labour Force Survey is treated specially: all versions share a single database, so the full history can be queried as one table. Pass 'version = "YYYY"' (annual) or '"YYYY-MM"' (monthly). '"LFS"' covers 2006 onwards, from Statistics Canada. '"LFS_HIST"' covers January 1976 to December 2005 in the legacy (pre-2017) layout, from the Borealis Dataverse. It is released monthly only, so a ‘"YYYY"' version loads that year’s twelve months, and its labels are harmonised across months. 'refresh = "auto"' loads every available version that is not yet in the database; this is only valid for '"LFS"' and '"LFS_HIST"'.

Value

A lazy 'dplyr::tbl()' backed by a DuckDB connection. Data values are pre-labeled as factors. Call 'dplyr::collect()' to materialise a local tibble, [label_pumf_columns()] to rename columns to their human-readable labels, or [close_pumf()] to release the connection. Returns 'invisible(NULL)' with an informative message if the data must be downloaded but Statistics Canada is unreachable.

See Also

[label_pumf_columns()], [pumf_var_labels()], [pumf_metadata()], [close_pumf()], [list_canpumf_collection()]

Examples

# Download and open the SFS 2019 as a lazy DuckDB table.
# get_pumf() returns NULL if Statistics Canada is unreachable.
sfs <- get_pumf("SFS", "2019")
if (!is.null(sfs)) {
  dplyr::glimpse(sfs)

  # Collect a local tibble (here, the first 100 records)
  sfs_local <- sfs |>
    head(100) |>
    dplyr::collect()

  # Release the connection when done
  close_pumf(sfs)
}

# French labels (opened and released on its own connection)
sfs_fr <- get_pumf("SFS", "2019", lang = "fra")
if (!is.null(sfs_fr)) close_pumf(sfs_fr)

Get a read-write DuckDB connection to a PUMF database

Description

Runs the full pipeline and returns a raw read-write [DBI::DBIConnection-class]. Use this when you need direct SQL access — to persist custom views, join derived tables, or inspect DuckDB internals. For everyday analysis use [get_pumf()], which returns a safer read-only lazy 'dplyr::tbl()'.

Usage

get_pumf_connection(
  series = NULL,
  version = NULL,
  lang = "eng",
  cache_path = getOption("canpumf.cache_path", tempdir()),
  refresh = FALSE,
  redownload = FALSE,
  ...
)

Arguments

Value

A [DBI::DBIConnection-class] in read-write mode. Disconnect with 'DBI::dbDisconnect(con, shutdown = TRUE)' when done. For a safer read-only lazy table use [get_pumf()] instead. Returns 'invisible(NULL)' with an informative message if the data must be downloaded but Statistics Canada is unreachable.

See Also

[get_pumf()]

Examples

con <- get_pumf_connection("SFS", "2019")  # NULL if StatCan is unreachable
if (!is.null(con)) {
  tables <- DBI::dbListTables(con)
  DBI::dbGetQuery(con, sprintf('SELECT COUNT(*) AS n FROM "%s"', tables[1]))
  DBI::dbDisconnect(con, shutdown = TRUE)
}

Rename PUMF table columns to human-readable variable labels

Description

Takes a lazy 'dplyr::tbl()' returned by [get_pumf()] and returns the same lazy table with column names replaced by the variable labels from the survey metadata (e.g. 'PHHSIZE' becomes '"Household size"'). Duplicate labels are disambiguated by appending ' (VAR_NAME)'.

Usage

label_pumf_columns(tbl)

Arguments

Details

The 'tbl' must have been produced by [get_pumf()]; the function reads survey provenance (series, version, cache path, language) from the underlying DuckDB connection. Use [pumf_var_labels()] to inspect the name-to-label mapping without renaming.

Value

A lazy 'dplyr::tbl()' with column names replaced by human-readable variable labels. Columns with no metadata label are left unchanged.

See Also

[pumf_var_labels()], [get_pumf()]

Examples

sfs <- get_pumf("SFS", "2019")
if (!is.null(sfs)) {
  sfs_labeled <- label_pumf_columns(sfs)
  colnames(sfs_labeled)
  close_pumf(sfs_labeled)
}

Get Labour Force Survey PUMF data from a shared longitudinal DuckDB

Description

Manages a single 'LFS.duckdb' file that accumulates all downloaded LFS versions. Each call either retrieves already-loaded data or downloads, parses, labels, and appends a new version.

Usage

lfs_get_pumf(
  version = NULL,
  lang = "eng",
  cache_path = getOption("canpumf.cache_path", tempdir()),
  refresh = FALSE,
  redownload = FALSE,
  read_only = TRUE
)

Arguments

Details

**Version types**: - '"YYYY"' (e.g. '"2023"') — annual file released by StatCan after year-end. - '"YYYY-MM"' (e.g. '"2024-06"') — monthly file for the current year.

When an annual file for year Y is loaded and monthly files for that year are already in the database, the monthly rows are replaced (supersession). Conversely, if an annual for year Y is already loaded, requesting a monthly for that year returns the annual data filtered to that month without re-downloading.

**Connection note**: the returned 'tbl' holds an open DuckDB connection. Loading a second version (i.e. calling 'lfs_get_pumf' again while holding the first result) requires the first tbl's connection to be closed first. Use [close_pumf()] or 'dplyr::collect()' the result before the next call.

Value

A lazy 'dplyr::tbl()', or 'invisible(NULL)' when 'version = NULL' and no data has been loaded.

List available LFS PUMF versions

Description

Scrapes the Statistics Canada LFS PUMF publication page and returns a tibble of all available annual and monthly versions with their download URLs. Requires an internet connection. For the broader collection of all supported surveys see [list_canpumf_collection()].

Usage

list_available_lfs_pumf_versions()

Value

A tibble with columns 'Date' (human-readable label from the StatCan page), 'version' (a string of the form '"YYYY"' for annual versions or '"YYYY-MM"' for monthly versions), and 'url' (direct download link). If the StatCan website is unreachable the function returns an empty tibble (with those columns) and a warning rather than erroring.

See Also

[get_pumf()], [list_canpumf_collection()]

Examples

lfs_versions <- list_available_lfs_pumf_versions()
tail(lfs_versions)

Browse the Statistics Canada PUMF collection on Borealis

Description

Lists the Statistics Canada Public Use Microdata File datasets held in the [Borealis](https://borealisdata.ca) Dataverse (the ODESI PUMF collection and the Census PUMFs). Borealis carries vintages that Statistics Canada no longer posts, such as the 1971–1986 Census PUMFs. Any dataset listed here can be loaded with 'get_pumf(series, version, borealis = <doi or row>)'; see [list_borealis_pumf_files()] to inspect a dataset's files first.

Usage

list_borealis_pumf_catalogue(
  refresh = FALSE,
  verbose = TRUE,
  cache_path = getOption("canpumf.cache_path")
)

Arguments

Details

Where Statistics Canada also posts a dataset for direct download, the ‘statcan' column is 'TRUE'. Prefer StatCan’s copy in that case (via 'get_pumf(series, version)' without 'borealis ='): the Borealis files are re-deposits and can carry transcription errors. 'get_pumf()' warns when an explicitly requested Borealis dataset is flagged this way. The flag is a heuristic match on catalogue number, series title, years and cycle number against [list_statcan_pumf_catalogue()], so check 'statcan_title' before relying on it.

The catalogue is fetched from the public Dataverse search API. There are several thousand datasets and Borealis renders them slowly, so pages are requested concurrently ('getOption("canpumf.borealis_parallel", 8)'), and a full fetch takes about a minute. The result is cached for the session and, when 'canpumf.cache_path' is set, persisted to '<cache_path>/borealis_catalogue.rds'. A persisted copy older than 'getOption("canpumf.catalogue_max_age_days", 30)' days triggers a warning. If Borealis is unreachable the last persisted copy is returned with a warning.

Value

A tibble with one row per dataset: 'title', 'year' (the first year in the title), 'language' ('"eng"'/'"fra"', guessed from the title), 'statcan' (logical, the dataset is also available from Statistics Canada), 'statcan_series' and 'statcan_title' (the matching StatCan catalogue entry, 'NA' when none), 'series' (the Borealis series name), 'doi', 'dataverse', 'file_count', 'published_at' and 'url'. English and French versions of a PUMF are separate datasets.

See Also

[list_borealis_pumf_files()], [get_pumf()]

Examples

# needs internet access; fails gracefully when Borealis is unreachable
cat <- tryCatch(list_borealis_pumf_catalogue(), error = function(e) NULL)
if (!is.null(cat)) dplyr::filter(cat, grepl("1971 Census", title))

List the files of a Borealis PUMF dataset

Description

Shows every file in a Borealis dataset together with the role canpumf assigns it ('data', 'metadata', 'doc' or 'skip') and whether 'get_pumf(..., borealis =)' would download it. Use it to check which data file and command files a DOI provides before loading it.

Usage

list_borealis_pumf_files(doi)

Arguments

Value

A tibble with one row per file: 'file_id', 'filename', 'directory', 'size' (bytes), 'md5', 'content_type', 'original' (the uploaded file behind a Dataverse '.tab' ingest), 'restricted', 'role' and 'selected'. The dataset DOI and title are attached as attributes.

See Also

[list_borealis_pumf_catalogue()], [get_pumf()]

Examples

tryCatch(list_borealis_pumf_files("doi:10.5683/SP3/LG7WKC"),
         error = function(e) message(conditionMessage(e)))

List Statistics Canada PUMF datasets supported by canpumf

Description

Returns a tibble of all survey series and versions for which canpumf has download wrappers. Scrapes the StatCan website to discover Census versions; other series are hard-coded. Requires an internet connection.

Usage

list_canpumf_collection()

Value

A tibble with columns 'Title', 'Acronym', 'Version', 'Survey Number', and 'url'. The 'url' column contains the download URL, '"(EFT)"' for versions distributed via the Research Data Centre (EFT only), or the Borealis dataset page for versions canpumf loads from Borealis (see [list_borealis_pumf_catalogue()]). Pass 'Acronym' and 'Version' to [get_pumf()] to download a dataset.

See Also

[get_pumf()], [list_available_lfs_pumf_versions()]

Examples

collection <- list_canpumf_collection()
# Show all SFS versions
collection[collection$Acronym == "SFS", c("Acronym", "Version")]

List the contents of the local canpumf cache

Description

Scans the cache directory and returns a tibble describing every downloaded PUMF version — which raw files, parsed metadata, and DuckDB tables are present — along with their disk sizes.

Usage

list_pumf_cache(cache_path = getOption("canpumf.cache_path", tempdir()))

Arguments

Details

For LFS surveys the DuckDB is a single shared file ('LFS.duckdb') that accumulates all versions; its total size is reported in 'duckdb_mb' for every LFS row. Use [remove_pumf_cache()] to free disk space.

Value

A tibble with columns:

'series'

Survey series acronym.

'version'

Version string.

'has_raw'

'TRUE' if a zip or extracted data files are present.

'has_metadata'

'TRUE' if a parsed 'metadata/' directory exists.

'has_duckdb'

'TRUE' if a DuckDB table is built for this version.

'raw_mb'

Disk size of raw files in MB (excluding metadata and DuckDB).

'duckdb_mb'

Disk size of the DuckDB file in MB. For LFS this is the total shared 'LFS.duckdb' size, repeated for each version row.

Returns a zero-row tibble with the same column structure if the cache directory does not exist or is empty.

See Also

[remove_pumf_cache()], [get_pumf()]

Examples

list_pumf_cache()
# With an explicit cache path:
list_pumf_cache(cache_path = file.path(tempdir(), "pumf_cache"))

Overview of all built-in registry entries

Description

Overview of all built-in registry entries

Usage

list_pumf_registry()

Value

A tibble with one row per registered '(series, version)' and columns summarising the key configuration: 'file_mask', 'layout_mask', 'bsw_join_key', and 'data_fixups' (comma-separated fixup types present).

See Also

[pumf_registry()], [pumf_registry_entry()]

Examples

list_pumf_registry()

Crawl the full Statistics Canada PUMF catalogue (experimental)

Description

Scrapes the live StatCan "Public use microdata" listing and follows each survey to its product page to discover every PUMF series, its editions, and direct-download URLs. This is an exploratory counterpart to [list_canpumf_collection()], which returns only the curated set of surveys canpumf has tested download wrappers for.

Usage

list_statcan_pumf_catalogue(
  prefer = names(.statcan_format_tokens),
  max_surveys = NULL,
  surveys = NULL,
  verbose = TRUE,
  refresh = FALSE,
  cache_path = getOption("canpumf.cache_path")
)

Arguments

Details

The StatCan markup is irregular and this crawler is best-effort: surveys distributed only by Electronic File Transfer (EFT) report 'url = "(EFT)"', and some products may not be parsed. When an edition is offered in several formats the one highest in 'prefer' is kept (CSV/flat-text first).

Value

A tibble with one row per discovered edition: 'catalogue_id', 'Acronym', 'SeriesTitle', 'Title', 'survey_url', 'edition', 'format', 'url', and 'product_url'. 'SeriesTitle' is the plain-language series name matching the acronym (the catalogue title with the edition-specific tail and "Public Use Microdata File" boilerplate stripped). 'Title' is edition-specific: StatCan's own per-edition catalogue title where it carries one, otherwise — for *umbrella* products whose catalogue title is only the series name (e.g. the consolidated General Social Survey, or a census year's individuals/hierarchical pair) — a synthesised '"<series> — <edition>"', where the structural edition descriptor disambiguates colliding years (GSS '"Cycle 16 (2002)"', census '"2021 (individuals)"'). 'edition' remains the reference period/variant. ‘survey_url' is the survey’s catalogue overview page (the L2 page the crawler followed); 'url'/'product_url' point at the individual edition's download and product page. 'Acronym' and 'SeriesTitle' are derived from the title since StatCan exposes no such field; they match the curated [list_canpumf_collection()] values for most surveys but are best-effort (Census 'Acronym' is hard-coded to '"Census"'). Surveys with no downloadable file get a single row with 'url = "(EFT)"'.

See Also

[list_canpumf_collection()], [get_pumf()]

Examples

# Quick look at the first 5 surveys
tryCatch(head(list_statcan_pumf_catalogue(max_surveys = 5)),
         error = function(e) message(conditionMessage(e)))

Merge metadata from multiple parser outputs

Description

Sources are applied in priority order: spss_mono > spss_split > sas_cards > lfs_csv > cpss_csv. For each variable / code, the highest-priority source provides the English label; French labels fill in from lower-priority sources when missing. Conflicting missing ranges or English code labels emit warnings.

Usage

merge_metadata(parsed_list)

Arguments

Value

Single merged canonical metadata list.

Check whether canonical metadata exists for a version directory

Description

Check whether canonical metadata exists for a version directory

Usage

metadata_exists(version_dir, bare = FALSE)

Arguments

Value

Logical.

Open PUMF documentation in the browser

Description

Scans the cached version directory for PDF documentation files and opens them interactively. If no PDFs are found, falls back to small text files (filtering out large FWF data files by size). When multiple candidate files exist, an interactive menu lets you choose which to open, with "Open all" as the last option. In non-interactive mode the first preferred-language file is opened automatically.

Usage

open_pumf_documentation(
  series = NULL,
  version = NULL,
  lang = NULL,
  cache_path = getOption("canpumf.cache_path", tempdir()),
  pumf_series = NULL,
  pumf_version = NULL,
  pumf_cache_path = NULL
)

Arguments

Details

After opening documentation, emits a message listing any manual registry overrides (sentinel values, forced-numeric columns, column swaps, etc.) that were applied at import so values can be interpreted correctly.

Value

Invisibly, the file path(s) of the opened documentation, or 'invisible(NULL)' when no documentation is found or data has not been downloaded yet.

See Also

[get_pumf()], [pumf_metadata()]

Examples

if (interactive()) {
# Open by series and version
open_pumf_documentation("SFS", "2019")

# Open from an existing tbl (reads provenance automatically)
sfs <- get_pumf("SFS", "2019")
open_pumf_documentation(sfs)
close_pumf(sfs)

# French documentation
open_pumf_documentation("SFS", "2019", lang = "fra")
}

Parse a CPSS variables.csv into canonical metadata

Description

The CPSS variables.csv is a single bilingual CSV. Variable rows have the Variable column filled; code rows have Variable empty and the Code column filled.

Usage

parse_cpss_csv(variables_path, encoding = "Latin1")

Arguments

Value

Named list with elements variables, codes, and layout (always NULL for CPSS CSV data).

Parse an LFS codebook CSV into canonical metadata

Description

The LFS codebook is a single bilingual CSV with rows alternating between variable definitions (Field_Champ filled) and code values (Field_Champ NA).

Usage

parse_lfs_codebook(codebook_path, encoding = "CP1252")

Arguments

Value

Named list with elements variables, codes, and layout (always NULL for LFS CSV data).

Parse a StatCan bilingual PDF frequency codebook

Description

Extracts variable names/labels and code-value labels from the bilingual PDF codebook shipped with surveys such as CPSS 1, which (unlike CPSS 2-6) ships no machine-readable variables.csv. The codebook lists each variable as a “Variable Name:/Concept:” block followed by an “Answer Categories” frequency table whose Code column supplies the value labels. Positions in the codebook are not used; this parser produces only variables and codes (no layout).

Usage

parse_pdf_codebook(eng_pdf, fra_pdf = NULL)

Arguments

Value

Named list with elements variables, codes, layout (always NULL), or NULL if no variable blocks were found.

Parse a StatCan PDF Data Dictionary for variable and code labels

Description

Extracts variable long-names and code-value labels from the bilingual PDF data dictionaries shipped with some older StatCan PUMF releases (e.g. SFS 1999). Positions in the PDF do not match the PUMF flat file; this parser produces only variables and codes (no layout).

Usage

parse_pdf_dictionary(eng_pdf, fra_pdf = NULL)

Arguments

Value

Named list with elements variables, codes, layout (always NULL).

Parse a StatCan PDF frequency data dictionary

Description

Extracts variable names, variable labels, value labels and per-code frequencies from the data-dictionary appendix of a StatCan PUMF user guide (GSS cycles, SGVP, PALS, SFS). Each variable appears as a “Variable Name:/Position:” block followed by a frequency table whose rows are <code> <label> <FREQ> <WTD>.

Usage

parse_pdf_freq_codebook(eng_pdf, fra_pdf = NULL)

Arguments

Details

Unlike the other metadata parsers this one also returns the frequencies, which pumf_parse_metadata uses to validate the parse against the actual microdata before allowing any label to be repaired from it.

Every returned table carries a block id – the index of the guide block it came from. A guide covering several linked modules documents a shared key once per module, so name alone is not unique; the block id plus the position/length columns on variables are what .pumf_pdf_select_blocks uses to pick the block belonging to the module being parsed. Positions are used only for that disambiguation – layout always comes from the command file.

Value

Named list with elements variables (plus block, position, length), codes, layout (always NULL), freqs (name/val/freq/block) and ranges (name/lo/hi/freq/block, for continuous variables), or NULL if no variable blocks were found.

Parse a set of SAS/SPSS reading-card files for PUMF metadata

Description

Handles surveys where metadata is stored in reading-card files: .lay (column positions), .lbe (English variable labels), .cde (English value labels), .mvs (missing values), with optional .lbf / .cdf for French.

Usage

parse_sas_cards(cards_dir, layout_mask = NULL, encoding = "Latin1")

Arguments

Details

The .lay file supports two sub-formats detected automatically:

Reading-card

Lines like NAME start - end (A)

SAS input

Lines like @pos NAME \$CHAR6.

Value

A list with elements variables (name, label_en, label_fr, type, missing_low, missing_high), codes (name, val, label_en, label_fr), and layout (name, start, end; NULL when no .lay file is found).

Parse an ODESI-generated SAS setup file

Description

The Borealis/ODESI deposits of older StatCan PUMFs (e.g. the monthly Labour Force Survey 1976-2005, catalogue 71M0001XCB) ship one machine-generated SAS program per language:

Usage

parse_sas_odesi(eng_path, fra_path = NULL, encoding = "CP1252")

Arguments

Details

“' PROC FORMAT LIBRARY=LIBRARY ; Value V4_F 1='Employed, at work' ; DATA OUT.x; INFILE 'x.txt' LRECL = 134; INPUT REC_NUM 1-5 SURVYEAR 6-9 ... FWEIGHT 131-134 ; FORMAT LFSSTAT V4_F. ; LABEL LFSSTAT='Respondent Labour Force Status' “'

Formats are tied to variables by the 'FORMAT var fmt.' statements, so a French program (same format names, same variables) supplies 'label_fr' by position-independent '(name, val)' matching.

Value

'list(variables, codes, layout)' in the canonical schema. Variables with a value format are typed '"character"', the rest '"numeric"'.

Parse a monolithic SPSS command file for PUMF metadata

Description

Handles Census-style monolithic SPSS files where a single .sps file contains DATA LIST, FORMATS, VARIABLE LABELS, VALUE LABELS, and optionally MISSING VALUES sections. Supports both the 2021 style (single-quoted labels, + continuation, /VAR headers in VALUE LABELS) and the 2016 style (double-quoted labels, / on its own line separating variable groups).

Usage

parse_spss_mono(eng_sps_path, fra_sps_path = NULL, encoding = "Latin1")

Arguments

Value

A list with elements variables (name, label_en, label_fr, type, missing_low, missing_high), codes (name, val, label_en, label_fr), and layout (name, start, end; NULL when no DATA LIST section is present).

Parse an SPSS .sav data file for embedded metadata

Description

Uses haven::read_sav() to extract variable labels, value labels, and SPSS format codes (which give type and decimal precision) without loading any data rows. This is the primary metadata source for surveys such as CIS 2016/2017 that ship only a .sav file and no separate SPSS command files.

Usage

parse_spss_sav(sav_path)

Arguments

Value

Named list with elements variables, codes, and layout (always NULL – .sav files embed metadata but use a binary record format that does not translate to start/end positions).

Parse a set of SPSS split command files for PUMF metadata

Description

Handles surveys where metadata is spread across separate files: *_i.sps (layout), *vare.sps (English variable labels), *vale.sps (English value labels), *miss.sps (missing values), with optional *varf.sps / *valf.sps for French.

Usage

parse_spss_split(layout_dir, layout_mask = NULL, encoding = "Latin1")

Arguments

Value

A list with elements variables (name, label_en, label_fr, type, missing_low, missing_high), codes (name, val, label_en, label_fr), and layout (name, start, end; NULL if no layout file is found).

Build a labeled DuckDB table for a PUMF version

Description

Reads the canonical metadata from 'metadata/', reads the raw data file, optionally joins bootstrap weights, applies code labels as factors, converts numeric columns, and writes to a '.duckdb' file.

Usage

pumf_build_duckdb(
  version_dir,
  series,
  version,
  lang = "eng",
  layout_mask = NULL,
  file_mask = NULL,
  refresh = FALSE,
  db_path = NULL,
  meta_subdir = NULL,
  data_fixups = NULL,
  bsw_override = NULL
)

Arguments

Details

Skips re-building if the named table already exists in the DuckDB file and 'refresh = FALSE'. Passing 'refresh = TRUE' drops and rewrites the table without re-downloading or re-extracting raw data.

Returns the db path and table name invisibly. Call [pumf_open_duckdb()] to open a read-only connection and get a lazy 'dplyr::tbl()'. Keeping Stage 3 and connection-opening separate prevents DuckDB file-lock conflicts when building multiple language tables for the same survey in one session.

Value

Invisibly, a named list with 'db_path' and 'table_name'.

Inspect the PDF-versus-microdata frequency validation

Description

Companion to [pumf_label_repairs()]. Reports, per variable, whether the frequencies printed in the survey's PDF data dictionary reconcile against a tabulation of the actual data file. This is the evidence 'canpumf' uses to decide whether a label from the guide may be trusted.

Usage

pumf_freq_validation(tbl)

Arguments

Details

'status' is one of:

'validated'

Every documented code's count matches the data exactly, and the data holds no undocumented values.

'continuous'

The documented sentinel codes match and the remaining values are accounted for by the guide's 'lo : hi' range row – a continuous variable, correctly parsed.

'mismatch'

Counts disagree; nothing from the guide is used for this variable.

'unchecked'

The check could not be run: no data file, no overlapping codes, or – common for a multi-module survey – the variable belongs to a sibling module and is absent from this one's data file. Also used when the guide's field positions reproduce the command file's layout but none of its counts reproduce the data (note '"guide frequencies use a different population"'): the guide is the right document, its tables were simply tabulated on another base, so the counts are treated as absent rather than as evidence against the parse.

A guide corroborated by *neither* channel – neither its counts nor its field positions – is rejected as the wrong document. Its validation table is still returned (it is the evidence for that decision), but the repair ledger is empty and no label from it is used.

Value

A tibble with columns 'block' (the guide block the check ran on), 'name', 'status', 'n_codes', 'n_matched' and 'note'; zero rows when the survey ships no parseable PDF dictionary.

See Also

[pumf_label_repairs()]

Examples

gss <- get_pumf("GSS", "Cycle 16 (2002)")
if (!is.null(gss)) {
  table(pumf_freq_validation(gss)$status)
  close_pumf(gss)
}

Inspect label repairs and divergences found against the PDF data dictionary

Description

Statistics Canada's PUMF command files routinely ship truncated value and variable labels – hard cuts at 60 characters, dropped leading text, dropped interior text. The damage is upstream of the command files (SAS, SPSS and Stata carry byte-identical text), but the same survey's user guide contains a data dictionary with the full label text.

Usage

pumf_label_repairs(tbl, action = NULL)

Arguments

Details

For surveys that ship such a guide, 'canpumf' parses it during metadata preparation, validates it against the microdata using the per-code frequencies the guide prints, and then repairs labels the guide demonstrably extends. This function returns the ledger of what it found: every divergence between the command file and the guide, whether or not it was acted on.

'action' is one of:

'repaired'

The command-file label was replaced, because the guide's text extends it *and* the command-file label carries a damage signature: either it sits at the width the command file's labels were hard-cut to, or it is what a dropped prefix leaves behind (a strict suffix of the guide's text). A guide that merely words a label differently is 'flagged' instead, as is one whose extra text is an annotation prepended to a label the command file has in full – recognised by the 'Key: value' shape of the dropped text, which running prose lost to truncation does not have.

'filled'

The command file had no label at all; the guide supplied one.

'flagged'

Recorded but not acted on – the two simply differ, or the variable's frequencies contradicted the data file, or the guide documents a code the command file never declared. Read 'reason' for which.

The ‘validation' column carries the variable’s frequency-check status ('validated', 'continuous', 'unchecked', 'mismatch', or 'not documented'), so a repair corroborated against the microdata can be told apart from one the check simply could not reach. See [pumf_freq_validation()].

Value

A tibble with columns 'kind' ('"variable"'/'"code"'), 'name', 'val', 'lang', 'label_command_file', 'label_pdf', 'action', 'reason' and 'validation'. Zero rows when the survey ships no parseable PDF dictionary or nothing diverged.

See Also

[pumf_var_labels()], [pumf_freq_validation()]

Examples

gss <- get_pumf("GSS", "Cycle 16 (2002)")
if (!is.null(gss)) {
  pumf_label_repairs(gss, action = "repaired")
  close_pumf(gss)
}

Locate or download a PUMF version directory

Description

Ensures the version directory exists and its zip has been extracted. With 'refresh = TRUE', clears the DuckDB file(s) and 'metadata/' subdirectory so that Stages 2 and 3 re-run, but does **not** re-download or re-extract raw data.

Usage

pumf_locate_or_download(
  series,
  version,
  cache_path = getOption("canpumf.cache_path", tempdir()),
  refresh = FALSE,
  redownload = FALSE
)

Arguments

Details

For EFT-only surveys (older Census years), stops with an informative error asking the user to deposit the zip manually.

Value

The version directory path, invisibly.

Download and parse PUMF metadata without building a DuckDB table

Description

Runs Stage 1 (locate or download) and Stage 2 (parse metadata) and returns the full bilingual canonical metadata. Both 'label_en' and 'label_fr' columns are always returned regardless of language. This is useful for inspecting variable definitions and code labels before loading data with [get_pumf()].

Usage

pumf_metadata(
  series,
  version,
  cache_path = getOption("canpumf.cache_path", tempdir()),
  refresh = FALSE,
  redownload = FALSE,
  registry = NULL
)

Arguments

Value

A named list with three elements:

'variables'

Tibble with columns 'name', 'label_en', 'label_fr', 'type', 'decimals', 'missing_low', 'missing_high'.

'codes'

Tibble with columns 'name', 'val', 'label_en', 'label_fr', mapping numeric codes to their labels.

'layout'

Tibble with columns 'name', 'start', 'end' for fixed-width data files; 'NULL' for CSV-format surveys.

Returns 'invisible(NULL)' with an informative message if the data must be downloaded but Statistics Canada is unreachable.

See Also

[get_pumf()], [pumf_var_labels()]

Examples

meta <- pumf_metadata("SFS", "2019")
if (!is.null(meta)) {
  meta$variables
  meta$codes[meta$codes$name == "PEFAMID", ]
}

Open a sibling module of a multi-module survey

Description

Some surveys ship several linked fixed-width files that share a respondent key (e.g. GSS cycle 16, "Aging and Social Support", 2002, whose MAIN, CG4, CG6 and CR files all join on 'RECID', with the person weight 'WGHT_PER' living only in MAIN). ‘get_pumf()' returns the survey’s primary module; 'pumf_module()' returns one of its sibling modules **on the same DuckDB connection**, so the two tbls are joinable on the shared key without opening a second connection.

Usage

pumf_module(tbl, module)

Arguments

Value

A lazy 'dplyr::tbl()' for the requested module, backed by the same connection as 'tbl'.

Examples

main <- get_pumf("GSS", "Cycle 16 (2002)") # primary module (MAIN), has WGHT_PER
if (!is.null(main)) {
  cg4 <- pumf_module(main, "CG4")          # caregiving module, same connection
  dplyr::left_join(main, cg4, by = "RECID")
  close_pumf(main)
}

Open a DuckDB table as a lazy dplyr tbl

Description

Opens a connection to the '.duckdb' file produced by [pumf_build_duckdb()] and returns a lazy 'dplyr::tbl()'. Use [close_pumf()] to release the connection when done.

Usage

pumf_open_duckdb(db_path, table_name, read_only = TRUE)

Arguments

Value

A lazy 'dplyr::tbl()'.

Parse all metadata from a PUMF version directory

Description

Detects every parseable command-file format in version_dir, runs all applicable parsers, merges the results into the canonical schema, and writes metadata/variables.csv, metadata/codes.csv (and optionally metadata/layout.csv) under version_dir.

Usage

pumf_parse_metadata(
  version_dir,
  layout_mask = NULL,
  metadata_encoding = NULL,
  refresh = FALSE,
  meta_subdir = NULL,
  file_mask = NULL
)

Arguments

Details

Idempotent: skips parsing if metadata/variables.csv already exists and refresh = FALSE.

Value

metadata_dir path invisibly.

Inspect a survey's registry configuration

Description

Returns the resolved configuration entry for a '(series, version)' pair: the built-in registry entry when one exists, otherwise an all-default entry. Useful for understanding the parsing strategy and overrides applied to a survey, and as a template for [pumf_registry_entry()].

Usage

pumf_registry(series, version)

Arguments

Value

A classed '"pumf_registry_entry"' list of all configuration fields.

See Also

[pumf_registry_entry()], [list_pumf_registry()], [get_pumf()]

Examples

pumf_registry("SFS", "2019")

Construct a custom PUMF registry entry

Description

Builds a survey-configuration patch that can be passed to [get_pumf()] (or [pumf_metadata()]) via the 'registry' argument to drive parsing and building for a survey that is not in the built-in registry, or to override specific fields of one that is.

Usage

pumf_registry_entry(
  layout_mask = NULL,
  bsw_mask = NULL,
  bsw_file_mask = NULL,
  bsw_join_key = NULL,
  bsw_drop_cols = NULL,
  bsw_strata = NULL,
  file_mask = NULL,
  data_encoding = NULL,
  metadata_encoding = NULL,
  data_fixups = NULL,
  bundled_eng_sps = NULL,
  bundle_source = NULL,
  bundle_sps_mask = NULL,
  doc_mask = NULL,
  download_format = NULL,
  borealis = NULL,
  ...
)

Arguments

Details

Only the arguments you actually supply are recorded; unspecified fields fall back to the built-in entry (when overriding a known survey) or to the pipeline defaults (for a new survey). This makes the result a *patch* rather than a full replacement. Use [pumf_registry()] to inspect an existing entry as a starting template.

The custom registry covers parsing and building configuration, plus an optional Borealis source ('borealis'); it does not provide a StatCan download URL. For a survey not in [list_canpumf_collection()], either point 'borealis' at the dataset on Borealis, or deposit the raw zip (or extracted files) under '<cache_path>/<series>/<version>/' first, then call 'get_pumf(series, version, registry = ...)'.

Value

A classed '"pumf_registry_entry"' list containing only the supplied fields.

See Also

[pumf_registry()], [get_pumf()], [list_pumf_registry()]

Examples

## Not run: 
# New CSV survey not yet in the registry (raw files already in the cache):
entry <- pumf_registry_entry(
  file_mask   = "DATA\\.csv",
  data_fixups = list(force_numeric = "WEIGHT"))
get_pumf("NEWSURVEY", "2025", registry = entry)

## End(Not run)

List all registered survey keys

Description

List all registered survey keys

Usage

pumf_registry_keys()

Value

character vector of '"series/version"' keys

Look up survey registry configuration

Description

Returns the configuration entry for a given survey series and version, or 'NULL' if the survey falls back to auto-detection.

Usage

pumf_registry_lookup(series, version)

Arguments

Value

named list of configuration fields, or 'NULL' if not in registry

Resolve version aliases

Description

Canonicalises user-supplied version strings for Census of Population. Any string starting with a four-digit year is parsed flexibly: the file type is detected by grepping for "hierarchical", "household", or "famil" (defaulting to "individuals"), and CMA vs provincial by grepping for "cma". The registry is then probed to find the correct canonical format for that year (e.g. '"1971 (households, CMA)"', '"1986 (families)"', or '"2001 (households)"').

Usage

pumf_resolve_version(
  series,
  version,
  cache_path = getOption("canpumf.cache_path", tempdir())
)

Arguments

Details

The 1971–1986 vintages exist twice: as StatCan EFT bundles that must be deposited by hand ('"1971/individuals_cma"', '"1986/families"', ...) and as Borealis downloads ('"1971 (individuals, CMA)"', '"1986 (families)"', ...). A loose request resolves to the EFT key when that year's bundle is present in 'cache_path', and to the Borealis key otherwise. Adding '"EFT"' or '"Borealis"' to the version string picks one explicitly, and an exact registry key is always returned unchanged.

Examples of accepted inputs (case-insensitive keywords): - '"2021"' -> '"2021 (individuals)"' - '"1971"' -> '"1971 (individuals, provincial)"' (or '"1971/individuals_prov"' with the EFT bundle deposited) - '"1971 households CMA"' -> '"1971 (households, CMA)"' - '"1986 families EFT"' -> '"1986/families"'

Value

canonical version string (or 'NULL' if 'version' was 'NULL')

Run the full three-stage PUMF pipeline for one survey version

Description

Convenience wrapper that: 1. Looks up the survey registry entry for '(series, version)'. 2. Calls [pumf_locate_or_download()] (Stage 1). 3. Calls [pumf_parse_metadata()] (Stage 2) with the registry's 'layout_mask' and 'metadata_encoding'. 4. Calls [pumf_build_duckdb()] (Stage 3). 5. Returns a lazy 'dplyr::tbl()' via [pumf_open_duckdb()].

Usage

pumf_run_pipeline(
  series,
  version,
  lang = "eng",
  cache_path = getOption("canpumf.cache_path", tempdir()),
  refresh = FALSE,
  redownload = FALSE,
  read_only = TRUE
)

Arguments

Details

Each stage is idempotent: subsequent calls reuse cached results unless 'refresh = TRUE'.

Value

A lazy 'dplyr::tbl()' backed by a DuckDB connection.

Retrieve variable labels as a tibble

Description

Returns a tibble mapping short coded column names to their bilingual human-readable variable labels. Use this as a quick reference without renaming the table itself; to rename, use [label_pumf_columns()].

Usage

pumf_var_labels(tbl)

Arguments

Value

A tibble with columns 'name' (coded column name), 'label_en' (English label), and 'label_fr' (French label). Rows follow survey-metadata order.

See Also

[label_pumf_columns()], [get_pumf()]

Examples

sfs <- get_pumf("SFS", "2019")
if (!is.null(sfs)) {
  pumf_var_labels(sfs)
  close_pumf(sfs)
}

Read canonical PUMF metadata from CSV files

Description

Read canonical PUMF metadata from CSV files

Usage

read_metadata(metadata_dir)

Arguments

Value

A list with elements variables, codes, and layout (NULL when no layout.csv is present).

Remove bootstrap weight tables and views from a PUMF DuckDB database

Description

Drops the bootstrap weight table(s) created by [add_bootstrap_weights()] and their companion VIEWs from the DuckDB file. When all BSW tables have been removed and the main survey table has a 'pumf_row_id' column (added automatically by [add_bootstrap_weights()] when no natural key was available), that column is also dropped.

Usage

remove_bootstrap_weights(tbl, weight_col = NULL)

Arguments

Details

Like [add_bootstrap_weights()], this function requires brief exclusive write access: the read-only connection backing 'tbl' is shut down, the tables are dropped, and a fresh read-only connection is returned.

Value

A lazy 'dplyr::tbl()' backed by the original physical survey table (without BSW columns), with a fresh read-only DuckDB connection.

See Also

[add_bootstrap_weights()], [bsw_info()], [get_pumf()]

Examples

sfs <- get_pumf("SFS", "2019")
if (!is.null(sfs)) {
  sfs_bsw <- add_bootstrap_weights(sfs, weight_col = "PWEIGHT", seed = 1L)
  # Remove only the PWEIGHT BSW table
  sfs_clean <- remove_bootstrap_weights(sfs_bsw, weight_col = "PWEIGHT")
  close_pumf(sfs_clean)
}

Remove a PUMF version from the local cache

Description

Deletes the DuckDB table (and optionally the raw zip and extracted files) for one cached PUMF version.

Usage

remove_pumf_cache(
  series,
  version,
  keep_raw = TRUE,
  cache_path = getOption("canpumf.cache_path", tempdir())
)

Arguments

Details

With the default 'keep_raw = TRUE', only the DuckDB and parsed 'metadata/' are removed; the raw zip and extracted data are left intact so that [get_pumf()] can rebuild without re-downloading. Set 'keep_raw = FALSE' to delete everything, freeing the full disk space.

For LFS surveys the DuckDB is shared across all versions. Removing one version deletes only that version's rows from the shared 'LFS.duckdb'; if it was the last loaded version the shared database file is also deleted.

Value

Invisibly 'NULL'.

See Also

[list_pumf_cache()], [get_pumf()]

Examples

# Remove only DuckDB and metadata, keep raw files for quick rebuild:
remove_pumf_cache("SFS", "2019")

# Remove everything including raw files:
remove_pumf_cache("SFS", "2019", keep_raw = FALSE)

Select language-specific labels from a metadata list

Description

Picks label_en or label_fr based on lang and adds a label column to variables and codes. When lang = "fra" and individual label_fr values are NA, those entries fall back to label_en and a warning is emitted listing the affected variable names.

Usage

select_labels(metadata, lang = "eng")

Arguments

Value

Modified metadata list with an additional label column in variables and codes.

Write canonical PUMF metadata to CSV files

Description

Write canonical PUMF metadata to CSV files

Usage

write_metadata(metadata, metadata_dir)

Arguments

Value

metadata_dir invisibly.