---
title: "Working with Livy (Spark)"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Working with Livy (Spark)}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = FALSE)
```

Apache Spark processes data using compute that runs in Fabric. *Livy* is the
service that lets an R program submit Spark code and receive its status and
output. The Spark code runs on Fabric; it does not run in your local R process.

Use Spark when a transformation is too large for one computer, needs a
Spark-specific library or format, or already exists as a Spark application.
For a small table read, SQL or a direct Lakehouse reader is usually simpler and
starts faster. This guide begins with one small statement, then moves to reusable
sessions, complete applications, and shared environments.

## Before the first call

You need a Fabric workspace on supported capacity, a Lakehouse with a Livy
endpoint, and the *tenant admin setting for the Livy API enabled*. 

A delegated caller needs all four of these Microsoft Entra scopes:

- `Lakehouse.Execute.All`
- `Lakehouse.Read.All`
- `Code.AccessFabric.All`
- `Code.AccessStorage.All`

With delegated sign-in and `audience = NULL`, 'fabricQueryR' requests those
four scopes. The following `Code.*` scopes are optional; add one only when the
Spark code needs the corresponding Azure service at runtime:

| Optional scope | Runtime access |
|---|---|
| `Code.AccessAzureKeyvault.All` | Azure Key Vault secrets, keys, or certificates |
| `Code.AccessAzureDataLake.All` | Azure Data Lake Storage Gen1 |
| `Code.AccessAzureDataExplorer.All` | Azure Data Explorer (Kusto) |
| `Code.AccessSQL.All` | Azure SQL |

Client-credentials authentication instead uses the single Fabric/Power BI
`.default` audience selected by the package. Microsoft currently documents
service-principal (SPN) tokens for session jobs. Add that principal to the
workspace as a *Contributor*, and confirm that the tenant's Livy settings allow
it; the workspace role alone does not guarantee service-side acceptance.

Microsoft's current batch guide is internally inconsistent: its introduction
says Azure SPN is unsupported, while its authentication section gives a
certificate-based SPN example. 'fabricQueryR' can acquire and send an app-only
token, but it cannot override a service-side identity restriction. Verify
unattended batch authentication in the target tenant and use a delegated user
if Fabric rejects the SPN.

The delegated user must be a *Contributor* in the workspace containing the
Livy endpoint and data-source items.

Then, you can discover the Lakehouse item which you want to use Livy with:

```{r, eval = FALSE}
library(fabricQueryR)

workspaces <- fabric_workspaces()
matches <- Filter(
  \(x) identical(x$displayName, "Analytics workspace"),
  workspaces
)
stopifnot(length(matches) == 1L)
workspace <- matches[[1L]]
lakehouse <- workspace$lakehouses()[[1L]]
```

`$lakehouses()` is the workspace method for `fabric_lakehouses()`.

The discovered `FabricLakehouse` is a read-only R6 object. Its fields expose
the service metadata. `$livy_query()`, `$livy_session()`, and
`$livy_batch_submit()` correspond to `fabric_livy_query()`,
`fabric_livy_session()`, and `fabric_livy_batch_submit()`.

An explicit `audience` replaces the defaults rather than extending them. For
example, include all four required scopes when adding Azure SQL access. The
Lakehouse `$livy_query()` method below calls `fabric_livy_query()`:

```{r, eval = FALSE}
livy_scopes <- paste0(
  paste0("https", "://api.fabric.microsoft.com/"),
  c(
    "Lakehouse.Execute.All",
    "Lakehouse.Read.All",
    "Code.AccessFabric.All",
    "Code.AccessStorage.All",
    "Code.AccessSQL.All"
  )
)

# `$livy_query()` is the object interface to `fabric_livy_query()`
result <- lakehouse$livy_query(
  code = "SELECT * FROM external_sql_table",
  kind = "sql",
  audience = livy_scopes
)
```

If discovery cannot retrieve the endpoint in your environment, copy the
session-job connection string from *Lakehouse settings > Livy endpoint* and
pass that URL instead.

## Run one piece of Spark code

`$livy_query()` (`fabric_livy_query()`) is the simplest method. It starts a temporary session,
runs one statement, waits, and closes the session:

```{r, eval = FALSE}
result <- lakehouse$livy_query(
  kind = "sql",
  code = "SELECT 1 AS id, 'hello from Spark' AS message"
)

result$output$parsed
```

The returned `output$parsed` value is usually a tibble for tabular results, an
R object for JSON, or character output for printed text.

## Choose the language that matches the code

The `kind` argument tells Livy how to interpret `code`:

Microsoft recommends the latest generally available runtime for production,
currently [Fabric Runtime 2.0
(Spark 4.1)](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-2-0).
SparkR still works, but SparkR is deprecated upstream in Spark 4.x and may be
removed in a future Spark release. Microsoft Fabric distributes
[`sparklyr`](https://learn.microsoft.com/en-us/fabric/data-science/r-use-sparklyr)
for R-first workloads in notebooks and Spark job definitions. Microsoft does
not currently document `sparklyr` over an item-scoped Livy session. Choose
PySpark or Spark SQL when the remote workload must be independent of the
current SparkR runtime bridge.

| `kind` | Code language |
|---|---|
| `"sql"` | Spark SQL |
| `"sparkr"` | R: SparkR; an experimental sparklyr bridge is described below |
| `"pyspark"` | Python with Spark |
| `"spark"` | Scala |

For example, SparkR code can use the active Spark session and Lakehouse:

This again uses `$livy_query()` (`fabric_livy_query()`):

```{r, eval = FALSE}
result <- lakehouse$livy_query(
  kind = "sparkr",
  code = paste(
    "df <- sql('SELECT * FROM orders LIMIT 100')",
    "printSchema(df)",
    "showDF(df, numRows = 10)",
    sep = "\n"
  )
)
```

`sparklyr` is not another Livy `kind`. The following is an experimental
adaptation of Fabric's documented notebook/Spark-job `"synapse"` connection;
Microsoft does not document it for item-scoped Livy, and 'fabricQueryR' does
not currently live-test it. If you evaluate that bridge in your tenant,
sparklyr code submitted through Livy still uses `kind = "sparkr"`:

```{r, eval = FALSE}
result <- lakehouse$livy_query(
  kind = "sparkr",
  code = paste(
    "library(sparklyr)",
    "spark_version <- sparkR.version()",
    "config <- spark_config()",
    paste0(
      "sc <- spark_connect(master = 'yarn', version = spark_version, ",
      "spark_home = '/opt/spark', method = 'synapse', config = config)"
    ),
    "orders <- dplyr::tbl(sc, 'orders')",
    "print(dplyr::collect(head(orders, 10)))",
    "spark_disconnect(sc)",
    sep = "\n"
  )
)
```

Even when it works, this removes only the application-level dependency on
SparkR's DataFrame verbs. It remains dependent on the R interpreter and SparkR
JVM bridge, so validate it after every runtime upgrade and do not treat this
example as a supported production contract.

## Reuse a session for several statements

Starting Spark can take time. Use `$livy_session()` (`fabric_livy_session()`)
when sequential statements need to share variables or cached data:

```{r, eval = FALSE}
answer <- local({
  session <- lakehouse$livy_session()
  on.exit(session$close(), add = TRUE)

  session$wait()
  session$run("shared_value = 40", kind = "pyspark")
  answer <- session$run("print(shared_value + 2)", kind = "pyspark")
  answer
})
answer$output$parsed
```

Always close a session explicitly. R object cleanup does not make a network
request, so forgetting `$close()` can leave Spark running. The returned
`FabricLivySession` lifecycle methods (`$wait()`, `$run()`, and `$close()`) do
not have separate free-function wrappers.

A standard session is right for one R process running a sequence. High
concurrency is an advanced option for several isolated workloads that may share
underlying compute; it is not needed for a few statements in order.

## Submit a complete application file

Use `$livy_batch_submit()` (`fabric_livy_batch_submit()`) when the work is a
repeatable Python, R, or Java/Scala script stored in OneLake or ADLS.

Python and Java batch applications have produced the expected output in the
package's persistent Fabric sandbox. Standalone R batches remain experimental:
attempts have failed during Spark-context initialization, so successful R batch
execution has not been established. Validate the application's output in the
target runtime before relying on it. Passing interactive SparkR statements does
not verify this separate batch path.

For example, submit a Python application and wait for completion:

```{r, eval = FALSE}
batch <- lakehouse$livy_batch_submit(
  file = paste0(
    "abfss://", workspace$id,
    "@onelake.dfs.fabric.microsoft.com/",
    lakehouse$id,
    "/Files/jobs/daily_transform.py"
  ),
  name = "daily-transform",
  wait = TRUE,
  timeout = 1800
)

batch$result()
```

The application file must already be available through an ABFS or ABFSS path.
Percent-encode spaces and other URL-reserved characters in file path segments;
raw spaces and whitespace in the workspace or filesystem authority are invalid.
Upload the file with `$onelake_upload()` (`fabric_onelake_upload()`) first when necessary. With
`wait = FALSE`, the function returns a `FabricLivyBatch` object immediately;
call its `$wait()`, `$result()`, or `$logs()` methods later.

The `FabricLivyBatch` lifecycle methods (`$wait()`, `$result()`, `$logs()`,
`$status()`, and `$cancel()`) do not have separate free-function wrappers.

If a session, statement, or batch wait times out, the
`fabric_livy_timeout_error` condition keeps the exact live object in its
`handle` field. You can inspect `$status()` through that handle in the current
R process. Use `$close()` for a session and `$cancel()` for a statement or batch.
`fabric_livy_query()` attempts to close its temporary session on exit, including
after a timeout, so that session may already be closed. The kind-specific `session`, `statement`, or
`batch` field contains safe metadata for logging; after serialization, a handle
intentionally no longer carries its credential.

Livy SQL results have a service-side row limit. The Fabric sandbox returned
1,000 rows for `SELECT * FROM range(1001)` on 7 September 2026, with a
`truncated` flag. The package raises `fabric_livy_partial_error` for that flag
and retains the returned rows in `partial_data`. The limit can vary by service
configuration. Use bounded queries or write large results to OneLake and read
them through the table/file APIs for complete extraction.

## Use an Environment for repeatable configuration

A published Fabric Environment can hold Spark settings and libraries shared by
several runs. Discover it and pass its ID when the workload depends on that
configuration:

```{r, eval = FALSE}
environment <- workspace$environments()[[1L]]

result <- lakehouse$livy_query(
  kind = "pyspark",
  code = "print(spark.version)",
  environment_id = environment$id
)
```

`$environments()` is the workspace method for `fabric_environments()`. The
Lakehouse `$livy_query()` method calls `fabric_livy_query()`.