Package {s3pe}

Title:	A Small, Dependable Client for S3 Object Stores
Version:	0.3.0
Description:	Read and write data on any S3-compatible object store ('AWS S3', 'MinIO', 'Garage', 'Ceph', ...). read_s3() and write_s3() move data frames, lists and files, with the format chosen by the key's extension ('CSV', 'JSON', 'Parquet', 'RDS', 'GeoJSON'); list, inspect and delete objects with the same small vocabulary. Requests are signed with AWS Signature Version 4 over the real hash of the payload, transient failures are retried with exponential backoff, and every transfer is checked against the server's ETag and content length. Endpoint and credentials come from the environment and are never written to disk.
License:	MIT + file LICENSE
URL:	https://github.com/StrategicProjects/s3pe, https://strategicprojects.github.io/s3pe/r/
BugReports:	https://github.com/StrategicProjects/s3pe/issues
Depends:	R (≥ 4.1)
Imports:	cli, httr2 (≥ 1.0.0), jsonlite, openssl, readr, rlang (≥ 1.1.0), tibble
Suggests:	knitr, nanoparquet, readxl, rmarkdown, sf, testthat (≥ 3.0.0), withr
VignetteBuilder:	knitr
Config/Needs/website:	tidyverse/tidytemplate, pkgdown
Config/testthat/edition:	3
Encoding:	UTF-8
Language:	en-US
Config/roxygen2/version:	8.0.0
NeedsCompilation:	no
Packaged:	2026-10-24 01:02:58 UTC; leite
Author:	André Leite [aut, cre], Marcos Wasiliew [aut], Hugo Vasconcelos [aut], Carlos Amorim [aut], Diogo Bezerra [aut], Júlia Nascimento Barreto [aut]
Maintainer:	André Leite <leite@castlab.org>
Repository:	CRAN
Date/Publication:	2026-11-07 17:30:02 UTC

s3pe: A Small, Dependable Client for S3 Object Stores

Description

Read and write data on any S3-compatible object store ('AWS S3', 'MinIO', 'Garage', 'Ceph', ...). read_s3() and write_s3() move data frames, lists and files, with the format chosen by the key's extension ('CSV', 'JSON', 'Parquet', 'RDS', 'GeoJSON'); list, inspect and delete objects with the same small vocabulary. Requests are signed with AWS Signature Version 4 over the real hash of the payload, transient failures are retried with exponential backoff, and every transfer is checked against the server's ETag and content length. Endpoint and credentials come from the environment and are never written to disk.

Author(s)

Maintainer: André Leite leite@castlab.org (ORCID)

Authors:

• André Leite leite@castlab.org (ORCID)

• Marcos Wasiliew marcos.wasiliew@gmail.com (ORCID)

• Hugo Vasconcelos hugo.vasconcelos@ufpe.br (ORCID)

• Carlos Amorim carlos.agaf@ufpe.br (ORCID)

• Diogo Bezerra diogo.bezerra@ufpe.br (ORCID)

• Júlia Nascimento Barreto juliabarreto@gd.seplag.pe.gov.br (ORCID)

See Also

Useful links:

• https://github.com/StrategicProjects/s3pe

• https://strategicprojects.github.io/s3pe/r/

• Report bugs at https://github.com/StrategicProjects/s3pe/issues

List the buckets the credentials can see

Description

List the buckets the credentials can see

Usage

buckets_s3(config = s3_config())

Arguments

Value

A tibble with name and created (UTC).

Examples

buckets_s3()

Copy an object inside the store

Description

A server-side CopyObject: nothing is downloaded. The copy keeps the source's content type and metadata.

Usage

copy_s3(from, to, bucket = NULL, config = s3_config())

Arguments

Value

Invisibly, a one-row tibble: bucket, key, size, etag, elapsed of the copy.

Examples

copy_s3("etl/sales.parquet", "archive/sales-2026.parquet", bucket = "my-bucket")
copy_s3("s3://staging/report.pdf", "s3://public/report.pdf")

Delete objects

Description

One key is a DELETE; several keys go in batches of 1000 (DeleteObjects). Deleting an object that does not exist is not an error; an object the store refuses to delete is.

Usage

delete_s3(key, bucket = NULL, config = s3_config())

Arguments

Value

The number of keys deleted, invisibly.

Examples

delete_s3("report/draft.csv", bucket = "my-bucket")
delete_s3(list_s3("tmp/", bucket = "my-bucket")$key, bucket = "my-bucket")

Download an object to a file

Description

Streams the object to disk. It is written next to its final name with the suffix .s3pe-part and renamed when complete and verified, so an interrupted download leaves nothing at path.

Usage

download_s3(key, path, bucket = NULL, overwrite = FALSE, config = s3_config())

Arguments

Value

Invisibly, a one-row tibble: bucket, key, size, etag, elapsed.

Examples

download_s3("data/regions.geojson", "regions.geojson", bucket = "my-bucket")

Does an object exist?

Description

Does an object exist?

Usage

exists_s3(key, bucket = NULL, config = s3_config())

Arguments

Value

TRUE or FALSE. Anything but a plain "not found" (no credentials, access denied, server down) is an error.

Examples

exists_s3("data/manifest.json", bucket = "my-bucket")

Metadata of an object

Description

A HEAD request: size, content type, modification time, ETag and user metadata, without downloading the object.

Usage

head_s3(key, bucket = NULL, config = s3_config())

Arguments

Value

A one-row tibble (key, size, content_type, last_modified, etag, and metadata, a list column holding the named list of user metadata), or NULL when the object does not exist.

Examples

head_s3("data/manifest.json", bucket = "my-bucket")

List the objects under a prefix

Description

Every object whose key starts with prefix, across all pages of the listing. Reading a bucket's manifest (metadata.json, manifest.json) is usually faster than listing it, when the publisher writes one last: it says what is there, and it is there only when everything else is.

Usage

list_s3(prefix = NULL, bucket = NULL, delimiter = NULL, config = s3_config())

Arguments

Value

A tibble with one row per object: key, size (bytes), last_modified (UTC) and etag. Zero rows when nothing matches.

Examples

list_s3("data/", bucket = "my-bucket")
list_s3("s3://my-bucket/series/")
list_s3("series/", bucket = "my-bucket", delimiter = "/")   # one level

A pre-signed URL

Description

A URL anyone can use for expires seconds, without credentials, to read (GET) or write (PUT) the object. The signature is in the query string (AWS Signature Version 4).

Usage

presign_s3(
  key,
  bucket = NULL,
  expires = 3600,
  method = c("GET", "PUT"),
  config = s3_config()
)

Arguments

Value

The URL, a string.

Examples

presign_s3("report/report.pdf", bucket = "my-bucket", expires = 3600)

Read an object into R

Description

Downloads an object and parses it according to its extension (or format):

Usage

read_s3(key, bucket = NULL, ..., format = NULL, config = s3_config())

Arguments

Details

The byte count is checked against the server's Content-Length and, when the ETag is an MD5, against the MD5 of what arrived.

Value

The parsed object, as in the table.

Examples

meta <- read_s3("data/manifest.json", bucket = "my-bucket")
meta$ultimo_mes

sales <- read_s3("s3://my-bucket/series/sales.csv.gz")

# the bytes, whatever the extension
png <- read_s3("report/figure1.png", bucket = "my-bucket", format = "raw")

Configure the connection to the store

Description

Collects endpoint, region, credentials and default bucket. Anything not given as an argument comes from an option, then from the environment, then from the default:

Usage

s3_config(
  endpoint = NULL,
  region = NULL,
  access_key = NULL,
  secret_key = NULL,
  session_token = NULL,
  bucket = NULL,
  timeout = NULL,
  max_tries = NULL,
  multipart_threshold = NULL,
  part_size = NULL
)

Arguments

Details

Keep credentials in ⁠~/.Renviron⁠ (R reads it at start-up), never in scripts. The endpoint has no default: set it once per machine (S3PE_ENDPOINT) or per session (use_endpoint()). URLs are always path-style (endpoint/bucket/key), which every S3-compatible store accepts.

Value

An object of class s3pe_config. Printing it never shows the secret key.

Examples

s3_config()
cfg <- s3_config(endpoint = "https://s3.example.org", bucket = "my-bucket")
list_s3("data/", config = cfg)

Mirror a directory to a prefix, or a prefix to a directory

Description

sync_up_s3() uploads every file under dir that is missing from the prefix or differs from it; sync_down_s3() downloads every object under the prefix that is missing locally or differs. "Differs" is decided by size and ETag: the local file's MD5, or its multipart ETag when it is larger than the config's multipart_threshold (computed with the same part_size, so files this package uploaded compare equal).

Usage

sync_up_s3(
  dir,
  prefix,
  bucket = NULL,
  delete = FALSE,
  last = NULL,
  config = s3_config()
)

sync_down_s3(prefix, dir, bucket = NULL, delete = FALSE, config = s3_config())

Arguments

Details

With delete = TRUE, what exists on the destination but not on the source is removed (remote objects in one batch; local files with unlink()). last names the file to upload after every other, so a manifest is there only when the batch is complete.

Value

Invisibly, a list of class s3pe_sync with transferred, skipped and deleted (relative paths) and elapsed (seconds).

Examples

sync_up_s3("output/", "reports/2026/", bucket = "my-bucket", last = "manifest.json")
sync_down_s3("s3://my-bucket/reports/2026/", "reports-2026/")

Upload a file

Description

Streams a file to the store (two passes over the file: one to hash it, one to send it). The server's ETag is compared with the file's MD5. Files larger than the config's multipart_threshold (64 MiB by default) go up as a multipart upload, read part by part, aborted on any failure.

Usage

upload_s3(
  path,
  key = NULL,
  bucket = NULL,
  content_type = NULL,
  metadata = NULL,
  headers = NULL,
  config = s3_config()
)

Arguments

Value

Invisibly, a one-row tibble: bucket, key, size, etag, elapsed.

Examples

upload_s3("output/panel.parquet", "etl/", bucket = "my-bucket")
upload_s3("figure1.png", "s3://my-bucket/report/figure1.png")
upload_s3("report.pdf", "public/", bucket = "my-bucket",
          headers = c("Content-Disposition" = "inline"))

Set the default bucket or endpoint for the session

Description

Shortcuts for options(s3pe.bucket = ) and options(s3pe.endpoint = ): after use_bucket("my-bucket") every verb accepts bare keys (read_s3("data/manifest.json")); after use_endpoint("https://s3.example.org") requests go to that store for the rest of the session. NULL clears the setting.

Usage

use_bucket(bucket)

use_endpoint(endpoint)

Arguments

Value

The previous value, invisibly.

Examples

use_endpoint("https://s3.example.org")
use_bucket("my-bucket")
read_s3("data/manifest.json")

Write an R object to the store

Description

Serialises x according to the key's extension (or format) and uploads it:

Usage

write_s3(
  x,
  key,
  bucket = NULL,
  ...,
  format = NULL,
  content_type = NULL,
  metadata = NULL,
  headers = NULL,
  config = s3_config()
)

Arguments

Details

The content type is set from the extension unless content_type is given. After the upload the server's ETag is compared with the MD5 of what was sent. Bodies larger than the config's multipart_threshold (64 MiB by default) go up as a multipart upload, part by part, aborted on any failure.

Value

Invisibly, a one-row tibble: bucket, key, size (bytes), etag, elapsed (seconds).

Examples

write_s3(mtcars, "report/mtcars.csv", bucket = "my-bucket")
write_s3(list(generated_at = Sys.time()), "report/manifest.json", bucket = "my-bucket")
write_s3(readBin("figure1.png", "raw", 1e7), "s3://my-bucket/report/figure1.png")
write_s3(mtcars, "report/mtcars.csv", bucket = "my-bucket",
         metadata = c(author = "ana"), headers = c("Cache-Control" = "max-age=3600"))