---
title: "Arrow Decimal Types"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Arrow Decimal Types}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
has_arrow <- requireNamespace("arrow", quietly = TRUE) &&
  requireNamespace("tibble", quietly = TRUE)
knitr::opts_chunk$set(eval = has_arrow)
# arrow can be built without Parquet support.
has_parquet <- has_arrow && arrow::arrow_with_parquet()
```

```{r, echo = FALSE, eval = !has_arrow, comment = NA}
message("The arrow package is not available; code output is not shown.")
```

Apache Arrow has exact decimal types — `decimal128(precision, scale)` and
`decimal256(precision, scale)` — and so does this package. They agree on what a
number *is*, so values can pass between them without losing a digit. What they
don't share is a memory layout: Arrow packs a decimal into a fixed-width
integer, while a `decimal` vector stores text and a shared scale
(`vignette("decimal-values")`).

The bridge between them is the **decimal string**. Both sides write and read
one exactly, which makes it a lossless interchange format, and the package uses
it in both directions for you. This vignette shows the crossings and where the
two type systems don't quite line up.

```{r setup}
library(decimal)
```

## A name collision to know about

`arrow` exports a `decimal()` function of its own — it builds an Arrow *type*,
not a vector — and attaching `arrow` masks this package's `decimal()`:

```r
library(arrow)
decimal(c("1.20", "2.30"))
#> Error: `precision` must be an integer
```

The masking runs whichever way you attach the two packages, so the reliable fix
is to qualify. This vignette never attaches `arrow`: every Arrow function below
is written `arrow::`, and decimal vectors are built with `decimal::decimal()`.
Adopting the same habit in a script that uses both packages will save you a
confusing error.

## From Arrow to decimal

Start with an Arrow array of exact decimals:

```{r}
a <- arrow::Array$create(
  c("100.05", "99999999999999999999.99", "0.01")
)$cast(arrow::decimal128(25, 2))
a
```

The obvious move is `as.vector()`. Don't — it converts through `double`:

```{r}
format(as.vector(a), digits = 22)
```

The first value picked up a tail of garbage, and the second lost its cents
entirely: `99999999999999999999.99` came back as `1e+20`. Twenty-two
significant digits don't fit in a double, which carries about sixteen.

`as_decimal()` takes the array directly and keeps every digit:

```{r}
as_decimal(a)
```

No `double` is involved. Arrow casts the column to text in its own exact
arithmetic, and that text is already the canonical form a `decimal` vector
stores, so the values move across without being re-parsed. That makes the exact
path about as fast as the lossy one.

### Scale comes along for free

An Arrow decimal type carries its own scale, and `as_decimal()` reads it off
the type rather than guessing from the text. That matters when a column happens
to hold only whole numbers — the declared cents survive anyway:

```{r}
whole <- arrow::Array$create(c("1", "2"))$cast(arrow::decimal128(9, 2))
as_decimal(whole)
attr(as_decimal(whole), "scale")
```

Passing `scale` overrides the type, rescaling as usual — exactly when the scale
grows, and by quantizing under the active `decimal_context()` when it shrinks:

```{r}
as_decimal(a, scale = 4)
```

### Nulls become `NA`

```{r}
as_decimal(arrow::Array$create(c("1.50", NA, "2.25"))$cast(arrow::decimal128(9, 2)))
```

### Chunked arrays work the same way

A column read from Parquet or a dataset is usually a `ChunkedArray` rather than
an `Array`. It has its own `as_decimal()` method, so no per-chunk bookkeeping is
needed:

```{r}
cs <- arrow::ChunkedArray$create(
  arrow::Array$create(c("1.25", "2.50"))$cast(arrow::decimal128(9, 2)),
  arrow::Array$create("3.75")$cast(arrow::decimal128(9, 2))
)
cs$num_chunks
as_decimal(cs)
```

### Integer columns

An Arrow integer column converts exactly too, at every width, because the
values cross as text rather than through `double`. An `int64` value beyond
2^53, which a double cannot hold, arrives intact:

```{r}
as_decimal(arrow::Array$create("9007199254740993")$cast(arrow::int64()))
```

## From decimal to Arrow

A `decimal` vector converts to Arrow on its own, so it becomes a decimal field
wherever arrow infers types — `arrow::arrow_table()`, `arrow::write_parquet()`,
`arrow::write_dataset()`:

```{r}
x <- decimal::decimal(c("1.25", "2.50", "-3.75"))
a <- arrow::as_arrow_array(x)
a$type
```

The type is an Arrow *extension type*. Its storage is a real `decimal128`,
with the vector's own scale and a precision inferred from the values present:
the widest one here needs three digits, one before the point and two after.
The storage is what a file carries, so Spark, DuckDB, pandas and every other
reader see an ordinary decimal column. The extension name is what lets arrow
hand the column back to this package on the way in, so in R it returns as a
`decimal` vector on every read path, `as.data.frame()` included:

```{r}
as.vector(a)
```

### Pinning the type

A tight precision derived from today's data may not fit tomorrow's, so for a
column you'll append to, pin a wider type. `arrow_decimal_type()` builds the
extension type with the precision and scale you choose:

```{r}
arrow::as_arrow_array(x, type = arrow_decimal_type(20, 2))$type
```

Passing a plain Arrow decimal type instead gives exactly that type, with no
extension:

```{r}
arrow::as_arrow_array(x, type = arrow::decimal128(20, 2))$type
```

Either way, Arrow refuses a cast that wouldn't fit rather than rounding
silently:

```{r, error = TRUE}
arrow::as_arrow_array(x, type = arrow::decimal128(2, 2))
```

### Plain fields, and when you want one

Arrow's compute engine does not operate on extension columns. A
`dplyr::filter()` or `summarise()` evaluated inside arrow on the decimal column
itself fails with "no kernel matching input types", while selecting,
collecting, and filtering on other columns work as usual. If you need
arrow-side arithmetic on the column, write it as a plain field: pass a plain
type as above, or turn the extension type off for every conversion:

```r
options(decimal.arrow_extension = FALSE)
```

The price of a plain field is the trip back. Arrow records an R column's
attributes in the schema and reapplies them blindly on read, so
`as.data.frame()` on a table built from a plain decimal field returns the
double arrow produced, wearing the `decimal` class. This package refuses to
format such an object rather than print rounded values. Read those tables
with `arrow_as_data_frame()`, described below, or drop the recorded
attributes first with `tab$ReplaceSchemaMetadata(NULL)`.

## Whole tables and Parquet files

A data frame with a decimal column becomes a table with a decimal field, and
comes back the same way:

```{r}
tab <- arrow::arrow_table(
  id = 1:3,
  amount = decimal::decimal(c("100.05", "0.01", "12.30"))
)
tab$schema$GetFieldByName("amount")$type$ToString()
tibble::as_tibble(as.data.frame(tab))
```

A tibble is used here because pillar prints the column's type, which makes it
easy to confirm the decimal survived the crossing.

Parquet preserves the Arrow type, so a file written with a decimal column reads
back as one, whether you take the data frame or the table:

```{r, eval = has_parquet}
path <- tempfile(fileext = ".parquet")
arrow::write_parquet(tab, path)
arrow::read_parquet(path)$amount

t2 <- arrow::read_parquet(path, as_data_frame = FALSE)
t2$schema$GetFieldByName("amount")$type$ToString()
as_decimal(t2$amount)
```

```{r, include = FALSE, eval = has_parquet}
unlink(path)
```

### Decimal columns written elsewhere

A Parquet file from Spark, DuckDB or pandas carries plain decimal fields with
no extension name, and `as.data.frame()` converts those to `double`.
`arrow_as_data_frame()` converts the decimal fields with `as_decimal()`
instead, each with the scale its type declares, and leaves every other column
to arrow:

```{r}
foreign <- arrow::arrow_table(
  id = 1:2,
  amount = arrow::Array$create(
    c("100.05", "99999999999999999999.99")
  )$cast(arrow::decimal128(25, 2))
)
tibble::as_tibble(arrow_as_data_frame(foreign))
```

To find the decimal fields in a schema you didn't write, check the field
types:

```{r}
types <- vapply(foreign$schema$fields, function(f) f$type$ToString(), character(1))
names(foreign)[grepl("^decimal", types)]
```

## Where the two type systems differ

Arrow's decimals are fixed-width integers with a scale, which makes them
narrower than a `decimal` vector in two ways worth planning around. Parquet
adds a third.

**Infinity and NaN have no Arrow decimal.** A `decimal` vector holds them
happily; the conversion reports which element it cannot represent rather than
inventing a value:

```{r, error = TRUE}
arrow::as_arrow_array(decimal::decimal(c("1.50", "NaN")))
```

If a column can contain them, keep it as a string in Arrow, or map them to `NA`
before converting.

**Precision is capped.** `arrow::decimal128()` allows at most 38 digits and
`arrow::decimal256()` at most 76; a `decimal` vector has no such limit. The
type is chosen for you — `decimal256()` when the values need it, an error when
even that is too narrow:

```{r}
big <- decimal::decimal(
  c("12345678901234567890.12345678", "0.10000000000000000001"),
  scale = 20
)
arrow::infer_type(big)
```

```{r, error = TRUE}
arrow::infer_type(decimal::decimal(strrep("9", 90)))
```

Read back through `double`, those two wide values would have been
`12345678901234567168` and `0.10000000000000000555` — the first wrong from its
seventeenth digit, the second not 0.1 at all. Through Arrow's decimal type,
they're exact:

```{r}
as_decimal(arrow::as_arrow_array(big))
```

**Parquet needs a scale of zero or more.** Arrow's decimal types accept a
negative scale, so a vector like `decimal::decimal("12300", scale = -2)`
converts to `decimal128(3, -2)` in memory, but Parquet's decimal type does not,
and `arrow::write_parquet()` refuses the column. Set the scale to zero before
writing:

```{r}
tens <- decimal::decimal(c("12300", "4500"), scale = -2)
arrow::infer_type(as_decimal(tens, scale = 0))
```