LFS

The LFS is one of the most-used PUMF series, since January 2021 the LFS PUMF is now easily available for direct download instead of needing to request it via EFT. This makes it very easy to integrate the LFS into reproducible workflows.

The canpumf package has two functions to facilitate access to the LFS PUMF. The first lists all LFS pumf versions that are available for direct download.

The second one fetches and loads the LFS data. For example, to download the LFS pumf for 2022 we use

By default the data is stored in the temporary session path, generally we want to make sure that the canpumf.cache_path option is set to a path to permanently cache the data.

Values come labelled, but columns are not. People working regularly with the LFS data will likely want to keep the short default column names, they can be converted to human readable column labels using label_pumf_columns function.

With this we can do some simple descriptive analysis. We could use the add_bootstrap_weights function to add bootstrap weights if desired. We focus in on February 2022 and add bootstrap weights. By default this adds 500 weights, this will take time to generate for the full LFS sample. For LFS data the bootstrap weight generation will automatically stratify the generation by year and month. Here we shortcut this by only generating bootstrap weights for the February 2022 data after calling collect, so they just get generated in memory for the resulting tibble.

For this vignette we look at gender-specific labour force status statistics for the 20 to 64 year old population, computing age-adjusted rates to even out age-specific effects.

We can similarly compute the age-adjusted participation rate by gender and marital status.

Narrowing it down a bit to only look at the share of the population employed and at work in February 2022 drops these shares a bit.

It’s good practice to close the database connection after being done with a specific task.

Derived connections, like the one to the February 2022 table, will automatically be closed too.

Timelines

LFS data can also easily be accessed across time.

lfs_pumf <- get_pumf("LFS", refresh="auto")

We can now easily extract time series data, we want to perform as many operations as possible at the database level. There are several convenience functions when working with the LFS data, one is add_lfs_SURVDATE which adds a SURVDATE column based on the survey year and month.

unemployment_stats <- lfs_pumf |> 
  filter(LFSSTAT !="Not in labour force") |>
  filter(AGE_12 %in% c("25 to 29 years","30 to 34 years", "35 to 39 years")) |>
  mutate(jd=case_when(is.na(DURJLESS) ~ "Not applicable",
                      DURJLESS<12 ~ "Less than one year",
                      TRUE ~ "One year or more")) |>
  add_lfs_SURVDATE() |>
  summarize(Count=sum(FINALWT),.by=c(SURVDATE,jd,AGE_12)) |>
  mutate(Share=Count/sum(Count),.by=c(SURVDATE,AGE_12)) |>
  filter(jd!="Not applicable")


unemployment_stats |>
  ggplot(aes(x=SURVDATE,y=Share,colour=AGE_12)) +
  geom_line() +
  facet_wrap(~jd) +
  scale_y_continuous(labels=scales::percent_format()) +
  labs(title="Unemployment by duration of unemployment",
       y="Unemployment rate",x=NULL,
       colour="Age group",
       caption="StatCan LFS (PUMF)")

Before plotting we could call collect, but this does not need to be done explicitly.

Because the data is efficiently organised in DuckDB, this query runs quite fast despite no explicit indexing of the database, taking less than half a second.

microbenchmark::microbenchmark(collect(unemployment_stats)) |> 
  boxplot()
#> Warning in microbenchmark::microbenchmark(collect(unemployment_stats)): less
#> accurate nanosecond times to avoid potential integer overflows

The SEX variable has been recategorized into the GENDER concept starting in 2011, older LFS PUMF data still uses SEX. We can harmonize this by coalescing the values to create a new GENDER_SEX column as done by the convenience function add_lfs_GENDER_SEX.

lfs_pumf |> 
  filter(LFSSTAT !="Not in labour force") |>
  add_lfs_SURVDATE() |>
  add_lfs_GENDER_SEX() |>
  summarise(Count=sum(FINALWT),.by=c(SURVDATE,LFSSTAT,GENDER_SEX)) |>
  mutate(Share=Count/sum(Count),.by=c(SURVDATE,GENDER_SEX)) |>
  filter(LFSSTAT=="Unemployed") |>
  ggplot(aes(x=SURVDATE,y=Share,colour=GENDER_SEX)) +
  geom_line() +
  scale_y_continuous(labels=scales::percent_format()) +
  labs(title="Unemployment sex/gender",
       y="Unemployment rate",x=NULL,
       colour="Gender",
       caption="StatCan LFS (PUMF)")

lfs_pumf |> close_pumf()

The long timeline, 1976 onward

Statistics Canada posts the LFS PUMF for 2006 onward. The monthly files for January 1976 to December 2005 are only publicly available through the Borealis Dataverse (ODESI). They use the older, pre-2017 file layout. canpumf loads them as a separate series, "LFS_HIST", which works just like "LFS": every month you load is appended to one shared database.

lfs_hist_1995_06 <- get_pumf("LFS_HIST", "1995-06")  # one month
lfs_hist_1995_06 |> 
  count(LFSSTAT, wt = FWEIGHT) |>
  collect()
#> # A tibble: 6 × 2
#>   LFSSTAT                             n
#>   <fct>                           <dbl>
#> 1 Employed, at work            12874136
#> 2 Employed, absent from work     731588
#> 3 Unemployed, temporary layoff    81243
#> 4 Unemployed, job searcher      1240976
#> 5 Unemployed, future start        42304
#> 6 Not in labour force           7677684

A year ("1995") loads all twelve months, and refresh = "auto" loads every month that is not yet in the database. The first full load downloads all 360 monthly files and takes a couple of hours. After that everything is read from the cache.

lfs_hist <- get_pumf("LFS_HIST", refresh = "auto")

The two series cannot be stacked as they are. The column sets differ, some variables are coded differently, and the current LFS stores hours in tenths and wages in cents. get_lfs_timeline() does that work. It combines whatever has been loaded of "LFS_HIST" and "LFS" into one lazy table with a curated set of common variables, and SOURCE records which series a row came from. The two databases are opened read-only, and nothing is copied or loaded.

By default the timeline uses only what has already been loaded. With refresh = "auto" it first loads every available month of either series that is not yet in the cache. An analysis script that starts with get_lfs_timeline(refresh = "auto") therefore always picks up the latest LFS release. When everything is current this only checks the list of available versions.

lfs_tl <- get_lfs_timeline(refresh = "auto")
#> LFS timeline: LFS_HIST 1976-01..2005-12 (360 versions); LFS 2006..2026-08 (28 versions).
pumf_var_labels(lfs_tl)
#> # A tibble: 57 × 3
#>    name     label_en                                           label_fr         
#>    <chr>    <chr>                                              <chr>            
#>  1 SOURCE   Source series                                      Série source     
#>  2 SURVYEAR Survey year                                        Année d'enquête  
#>  3 SURVMNTH Survey month                                       Mois de l'enquête
#>  4 PROV     Province                                           Province         
#>  5 AGE_12   Five-year age group of respondent                  L'âge du réponda…
#>  6 AGE_6    Age in 2 and 3 year groups, 15 to 29               L'âge des person…
#>  7 EDUC     Highest educational attainment                     Plus haut niveau…
#>  8 MJH      Single or multiple jobholder                       Personnes ayant …
#>  9 EVERWORK Identifies if a person has worked in the last year Indique si une p…
#> 10 FTPTLAST Full- or part-time status of last job              Situation du der…
#> # ℹ 47 more rows

Variables with identical codes in both series use the current LFS labels. The rest are recoded to a common scheme:

LFSSTAT collapses the historical breakdown of unemployment.
GENDER_SEX combines sex and gender, in the same way as add_lfs_GENDER_SEX().
MARSTAT has four categories, because the files before November 1999 have only four.
CMA distinguishes Montréal, Toronto, Vancouver and the rest, and is NA before 1987, when the older files do not identify CMAs.
SCHOOLN, AGYOWNK and the industry groups (NAICS_18) are also recoded.

Hours are in hours and wages in dollars throughout, and the weight is FINALWT. All the usual tools work on the result, including add_lfs_SURVDATE() and label_pumf_columns().

To get a long series of the unemployment rate, we again do the heavy lifting in the database and only collect the monthly totals.

lf_monthly <- lfs_tl |>
  filter(LFSSTAT != "Not in labour force") |>
  add_lfs_SURVDATE() |>
  summarise(labour_force = sum(FINALWT),
            unemployed = sum(FINALWT[LFSSTAT == "Unemployed"], na.rm = TRUE),
            .by = c(SURVDATE, GENDER_SEX)) |>
  mutate(rate = unemployed / labour_force) |>
  collect()

lf_monthly |>
  ggplot(aes(x = SURVDATE, y = rate, colour = GENDER_SEX)) +
  geom_line(alpha = 0.3) +
  geom_smooth(method = "loess", span = 0.05, se = FALSE, linewidth = 0.8) +
  geom_vline(xintercept = as.Date("2006-01-01"), linetype = "dashed") +
  scale_y_continuous(labels = scales::percent) +
  labs(title = "Unemployment rate by gender/sex, 1976 onward",
       subtitle = "Monthly, not seasonally adjusted; dashed line: LFS_HIST to LFS",
       x = NULL, y = "Unemployment rate", colour = NULL,
       caption = "StatCan LFS PUMF (1976-2005 via Borealis/ODESI)")
#> `geom_smooth()` using formula = 'y ~ x'

The recessions of the early 1980s and 1990s, 2008-09 and 2020 all show up clearly, and the series is continuous across the 2005/2006 seam.

Longer timelines bring out structural change. The participation rate of women in their core working years rose from about half to over 80%.

core_age <- c("25 to 29 years", "30 to 34 years", "35 to 39 years", "40 to 44 years",
              "45 to 49 years", "50 to 54 years")

participation <- lfs_tl |>
  filter(AGE_12 %in% core_age) |>
  summarise(population = sum(FINALWT),
            labour_force = sum(FINALWT[LFSSTAT != "Not in labour force"], na.rm = TRUE),
            .by = c(SURVYEAR, GENDER_SEX)) |>
  mutate(rate = labour_force / population) |>
  collect()

participation |>
  ggplot(aes(x = SURVYEAR, y = rate, colour = GENDER_SEX)) +
  geom_line() +
  scale_y_continuous(labels = scales::percent) +
  labs(title = "Labour force participation of 25 to 54 year olds",
       subtitle = "Pooled monthly samples of each year",
       x = NULL, y = "Participation rate", colour = NULL,
       caption = "StatCan LFS PUMF (1976-2005 via Borealis/ODESI)")

Variables that enter the survey later are NA for earlier years, so a filter on them restricts the timeline automatically. The three large CMAs are identified from 1987, and hourly wages are collected from 1997 on.

lfs_tl |>
  filter(LFSSTAT %in% c("Employed, at work", "Employed, absent from work"),
         !is.na(HRLYEARN)) |>
  summarise(wage = sum(HRLYEARN * FINALWT) / sum(FINALWT),
            .by = c(SURVYEAR, CMA)) |>
  collect() |>
  ggplot(aes(x = SURVYEAR, y = wage, colour = CMA)) +
  geom_line() +
  scale_y_continuous(labels = scales::dollar) +
  labs(title = "Average usual hourly wage of employees",
       subtitle = "Nominal dollars",
       x = NULL, y = NULL, colour = NULL,
       caption = "StatCan LFS PUMF (1997-2005 via Borealis/ODESI)")

Some caveats apply when working across the full range:

Population rebasing. The weights of each era reflect the population estimates in use when the files were last revised: 1976–86 original, 1987–95 rebased to the 2001 Census, 1996–2000 to 2006, 2001–05 to 2011, and the current LFS to more recent Censuses. The population series runs smoothly across these boundaries in the public-use files, but population counts from different eras are not strictly comparable. Rates are much less affected than counts.
1985–86. Estimates computed from LFS_HIST match the published table 14-10-0287 to rounding, except for January 1985 to November 1986. For those months the public-use files give about 1.2% more unemployed than the published series.
Variables outside the common schema. Occupation, the spouse variables and other detail only available in one series are not part of the timeline. Use get_pumf("LFS_HIST") or get_pumf("LFS") for those.

As before, close the connections when done.

close_pumf(lfs_tl)
close_pumf(lfs_hist)
close_pumf(lfs_hist_1995_06)