Package {sage}

Small-Area Global Elections (SAGE) Data Retrieval

Description

These helpers fetch partitioned Parquet from the SAGE public release (default: a Google Cloud Storage bucket; override with sage_set_source() or the SAGE_SOURCE environment variable) and let users pull (country, years, columns) slices without reading the giant per-country RDS files.

Details

Remote reads go over plain HTTPS (one partition file at a time, guided by the ⁠_index.csv⁠ manifest), so the package works with any build of arrow – it does not require Arrow's S3/GCS cloud filesystem to be compiled in.

When the source is the public bucket and arrow does have GCS support compiled in, sage_columns() reads Parquet schemas through Arrow's random-access reader instead of downloading whole partitions. That is an optimization only: it falls back to the HTTPS download path whenever the cloud filesystem or an individual read is unavailable.

Source layout

The release is a hive-partitioned Parquet dataset:

<source>/
  country=Afghanistan/year=2014/part-0.parquet
  country=Afghanistan/year=2019/part-0.parquet
  country=Albania/year=2009/part-0.parquet
  ...
  _index.csv

so a (country, year) slice is a single partition read.

Offline excerpt

Every example in this package runs against a small excerpt of the archive (Iceland and Samoa) shipped under extdata/mini_archive, so no network access is required to try the interface:

sage_set_source(system.file("extdata", "mini_archive", package = "sage"))

Point sage_set_source() back at "gs://sage-archive/parquet" to use the full release.

List the columns available in the dataset

Description

Not every column is populated for every country; this returns the union of columns over the largest partition of each of several countries, which is what surfaces the optional deep-hierarchy columns (NAME4, NAME5, ...). The answer is cached for the session, and is discarded when sage_set_source() changes the source.

Usage

sage_columns()

Value

A character vector of column names, beginning with "country" and "year".

See Also

sage_load(), whose columns argument takes these names.

Examples

sage_set_source(system.file("extdata", "mini_archive", package = "sage"))
sage_columns()

List countries available in the current source

Description

Reads the _index.csv manifest that ships with the archive, so this costs one small request rather than a scan of the bucket.

Usage

sage_countries()

Value

A character vector of country names, sorted.

See Also

sage_years(), sage_load()

Examples

sage_set_source(system.file("extdata", "mini_archive", package = "sage"))
sage_countries()

Load a (country, years, columns) slice of SAGE

Description

Reads only the partitions the request needs, one file per (country, year), and returns them as a single tibble.

Usage

sage_load(
  country,
  years = NULL,
  columns = NULL,
  election_type = NULL,
  min_match_confidence = NULL,
  drop_all_na = TRUE
)

Arguments

Value

A tibble, with country and year as the first two columns.

See Also

sage_columns() for the available columns, sage_polygons() for boundaries.

Examples

sage_set_source(system.file("extdata", "mini_archive", package = "sage"))
is_2020 <- sage_load("Iceland", years = 2020)
dim(is_2020)
names(is_2020)

# Restrict to a few columns; country and year are always kept.
sage_load("Iceland", years = 2020, columns = c("party", "votes"))

## Not run: 
# Against the full release (requires network access).
sage_set_source("gs://sage-archive/parquet")
de_2021 <- sage_load("Germany", years = 2021, columns = c("party", "votes", "NAME3"))
all_de_high <- sage_load("Germany", min_match_confidence = "high")

## End(Not run)

Load the polygon set for one country

Description

SAGE ships a parallel polygons/ tree (one geoparquet per country) alongside the main parquet release; the polygons carry only the NAME-key + year + geometry columns and are deduplicated to one row per unique polygon. Use this helper when you need boundaries for choropleth mapping; the main sage_load() only returns lat/long.

Usage

sage_polygons(country, years = NULL)

Arguments

Details

Requires the sf and sfarrow packages.

Value

An sf data frame, or NULL invisibly if the source has no polygons for the country.

See Also

sage_load() for the vote data these boundaries join to.

Examples

## Not run: 
# Requires network access, plus the sf and sfarrow packages; polygons are
# not part of the offline excerpt.
sage_set_source("gs://sage-archive/parquet")
is_poly <- sage_polygons("Iceland", years = 2021)

## End(Not run)

Load the per-candidate vote sidecar

Description

Some countries report votes for individual candidates alongside the party totals in the main release. This reads that sidecar. It is available for Germany (Erststimme by Wahlkreis) and the Netherlands (preference votes by stembureau); any other country is an error.

Usage

sage_preference_votes(country)

Arguments

Value

A tibble of candidate-level votes.

See Also

sage_load() for the main party-level release.

Examples

## Not run: 
# Requires network access; the sidecar is not part of the offline excerpt.
sage_set_source("gs://sage-archive/parquet")
de_cand <- sage_preference_votes("Germany")

## End(Not run)

Set the source URL or path for the SAGE archive

Description

Sets the archive every other function in the package reads from. The setting is global to the session and persists until it is changed again; unset, the package falls back to the SAGE_SOURCE environment variable and then to the public bucket.

Usage

sage_set_source(source)

Arguments

Value

The source, invisibly.

See Also

sage_load() to read a slice from the source that is set here.

Examples

# Point the package at the offline excerpt that ships with it.
sage_set_source(system.file("extdata", "mini_archive", package = "sage"))
sage_countries()

## Not run: 
# The full public release (requires network access).
sage_set_source("gs://sage-archive/parquet")

## End(Not run)

List election years available for a country

Description

List election years available for a country

Usage

sage_years(country)

Arguments

Value

An integer vector of years, sorted. A country with no partitions in the source gives a zero-length vector.

See Also

sage_countries(), sage_load()

Examples

sage_set_source(system.file("extdata", "mini_archive", package = "sage"))
sage_years("Iceland")