Seamless AWS cloud bursting for parallel R workloads
staRburst lets you run parallel R code on AWS with zero infrastructure management. Scale from your laptop to 100+ cloud workers with a single function call. Supports both EC2 (recommended for performance and cost) and Fargate (serverless) backends.
starburst_map()
function - no new concepts to learnFrom CRAN:
install.packages("starburst")Development version (from GitHub):
remotes::install_github("scttfrdmn/starburst")library(starburst)
# One-time setup (~2 min to provision; first run also builds the worker image,
# +5-10 min once)
starburst_setup()
# Run parallel computation on AWS (each task should be real work — seconds+ —
# so it's worth shipping; batch tiny items first). Console output is illustrative:
results <- starburst_map(
inputs, # e.g. a list of scenarios / parameter sets
function(x) expensive_computation(x),
workers = 50
)
#> [Starting] Starting starburst cluster with 50 workers
#> [Status] Processing 200 items with 50 workers
#> [Starting] Submitting 200 tasks...
#> [Wait] Progress: 200/200
#> [OK] Completed
#> [Cost] Estimated cost: (printed per run)library(starburst)
# Define simulation
simulate_portfolio <- function(seed) {
set.seed(seed)
returns <- rnorm(252, mean = 0.0003, sd = 0.02)
prices <- cumprod(1 + returns)
list(
final_value = prices[252],
sharpe_ratio = mean(returns) / sd(returns) * sqrt(252)
)
}
# Each simulation is tiny (sub-millisecond), so BATCH them: 100 tasks of 100
# simulations each, not 10,000 one-simulation tasks. (Thousands of tiny tasks are
# an anti-pattern — see the "Workload Shapes" guide for measured numbers.)
batches <- split(1:10000, ceiling(seq_along(1:10000) / 100))
results <- starburst_map(
batches,
function(seeds) lapply(seeds, simulate_portfolio),
workers = 50
)
results <- unlist(results, recursive = FALSE) # flatten to 10,000 results
# Extract results
final_values <- sapply(results, function(x) x$final_value)
sharpe_ratios <- sapply(results, function(x) x$sharpe_ratio)
# Summary
mean(final_values) # Average portfolio outcome
quantile(final_values, c(0.05, 0.95)) # Risk rangeFor real, measured performance (when the cloud wins, when it loses, and how to size tasks/workers), see the Workload Shapes and Performance guides.
# Create cluster once
cluster <- starburst_cluster(workers = 50, cpu = 4, memory = "8GB")
# Run multiple analyses
results1 <- cluster$map(dataset1, analysis_function)
results2 <- cluster$map(dataset2, processing_function)
results3 <- cluster$map(dataset3, modeling_function)
# All use the same Docker image and configuration# For memory-intensive workloads
results <- starburst_map(
large_datasets,
memory_intensive_function,
workers = 20,
cpu = 8,
memory = "16GB"
)
# For CPU-intensive workloads
results <- starburst_map(
cpu_tasks,
cpu_intensive_function,
workers = 50,
cpu = 4,
memory = "8GB"
)Run long jobs and disconnect - results persist in S3:
# Start a detached session (EC2 + Spot by default)
session <- starburst_session(workers = 50)
# Fan work out by submitting one task per input
task_ids <- lapply(inputs, function(input) {
session$submit(quote(expensive_function(input)),
globals = list(input = input))
})
session_id <- session$session_id
# Disconnect - job continues running.
# Later (hours/days), reconnect from a fresh R session:
session <- starburst_session_attach(session_id)
status <- session$status() # Check progress
results <- session$collect(wait = TRUE) # Collect results (in submission order)
# Cleanup when done
session$cleanup().x) — there is no automatic chunking; batch small
items yourself if per-task overhead matterslaunch_type = "FARGATE"; falls back to waves if
quota-limitedqs2 format# Set an hourly cost ceiling (USD/hour)
starburst_config(
max_hourly_cost = 10, # Jobs estimated over $10/hour won't start
cost_alert_threshold = 5 # Warn at $5/hour
)
# Costs shown transparently
results <- starburst_map(data, fn, workers = 100)
#> [Starting] Starting starburst cluster with 100 workers
#> [OK] Completed in 1380.0 seconds
#> [Cost] Estimated cost: $1.34When using the Fargate backend, staRburst automatically handles AWS Fargate vCPU quota limits (EC2 uses your standard On-Demand/Spot limits):
results <- starburst_map(data, fn, workers = 100, cpu = 4, launch_type = "FARGATE")
#> [!] Requested 100 workers (400 vCPUs) but quota allows 25 workers (100 vCPUs)
#> [!] Using 25 workers insteadYour work still completes, just with fewer workers. You can request quota increases through AWS Service Quotas.
starburst_map(.x, .f, workers, ...) - Parallel map over
datastarburst_cluster(workers, cpu, memory) - Create
reusable clusterstarburst_setup() - Initial AWS configurationstarburst_config(...) - Update configurationstarburst_status() - Check cluster statusstarburst_config(
max_hourly_cost = 10, # USD/hour rate ceiling
cost_alert_threshold = 5 # USD/hour warning
)Full documentation available at starburst.ing
| Feature | staRburst | RStudio Server on EC2 | Coiled (Python) |
|---|---|---|---|
| Setup time | 2 minutes | 30+ minutes | 5 minutes |
| Infrastructure management | Zero | Manual | Zero |
| Learning curve | Minimal | Medium | Medium |
| Auto scaling | Yes | No | Yes |
| Cost optimization | Automatic | Manual | Automatic |
| R-native | Yes | Yes | No (Python) |
AWS_PROFILE setstarburstECSExecutionRole - for ECS/ECR accessstarburstECSTaskRole - for S3 accessFor detailed setup instructions, see the Getting Started guide.
launch_type/instance_type/use_spot)
on starburst_map()/starburst_cluster()starburst_setup() provisions the
default capacity provider)starburst_map,
starburst_cluster)future backend integration
(future.apply, furrr,
targets)Contributions welcome! See the GitHub repository for contribution guidelines.
Apache License 2.0 - see LICENSE
Copyright 2026 Scott Friedman
@software{starburst,
title = {staRburst: Seamless AWS Cloud Bursting for R},
author = {Scott Friedman},
year = {2026},
version = {0.3.9},
url = {https://starburst.ing},
license = {Apache-2.0}
}Built using the paws AWS SDK for R.
Container management with renv and rocker.
Inspired by Coiled for Python/Dask.