---
title: "Explainable Outlier Detection in Titanic dataset"
author: "David Cortes"
output:
    rmarkdown::html_document:
        theme: "spacelab"
        highlight: "kate"
        toc: true
        toc_float: true
vignette: >
    %\VignetteIndexEntry{Explainable Outlier Detection in Titanic dataset}
    %\VignetteEngine{knitr::rmarkdown}
    %\VignetteEncoding{UTF-8}
---
```{r, include = FALSE}
### Don't overload CRAN servers
### https://stackoverflow.com/questions/28961431/computationally-heavy-r-vignettes
is_check <- ("CheckExEnv" %in% search()) || any(c("_R_CHECK_TIMINGS_",
             "_R_CHECK_LICENSE_") %in% names(Sys.getenv()))
```

# Explainable Outlier Detection in Titanic dataset

This short notebook illustrates basic usage of the [OutlierTree](https://github.com/david-cortes/outliertree) library for explainable outlier detection using the Titanic dataset. For more details, you can check the package's documentation [at CRAN](https://cran.r-project.org/package=outliertree) or through R's help (e.g. `?outliertree::outlier.tree`). For a more interesting and interactive example, see the documentation of the main function (`outlier.tree`), which uses a larger dataset.

The dataset is very popular and can be downloaded from different sources, such as Kaggle or many university webpages. This vignette took it from the following link: https://github.com/jbryer/CompStats/raw/master/Data/titanic3.csv

The data comes bundled in the package so there is no need to download it from the link above.

### Loading the raw data

```{r message=FALSE}
library(magrittr)
library(dplyr)
library(kableExtra)
library(outliertree)
data("titanic")

titanic %>%
	head(5) %>%
	kable() %>%
	kable_styling()
```

### Pre-processing the data

```{r}
## Capitalize column names and some values for easier reading
capitalize <- function(x) gsub("^(\\w)", "\\U\\1\\E", x, perl=TRUE)
titanic %<>%
	rename_all(capitalize) %>%
	mutate(Sex = capitalize(Sex)) %>%
	rename(SibSp = Sibsp)

## Convert 'survived' to yes/no for easier reading
titanic %<>%
	mutate(Survived = recode(Survived, `1`="Yes", `0`="No"))

## Some columns are not useful, such as name (an ID), ticket number (another ID),
## or destination (too many values, many non-repeated)
titanic %<>%
	select(-Name, -Ticket, -Home.dest)

## Ordinal columns need to be passed as ordered factors
cols_ord <- c("Pclass", "Parch", "SibSp")
titanic %<>%
	mutate_at(.vars=cols_ord, function(.) factor(., ordered = TRUE))

## A look at the processed data
titanic %>%
	head(5) %>%
	kable() %>%
	kable_styling()
```

### Fitting a model

```{r, eval=FALSE}
library(outliertree)

## Fit model with default hyperparameters
otree <- outlier.tree(titanic)
otree
```
```{r, echo=FALSE, comment=NA}
library(outliertree)

## Fit model with default hyperparameters
otree <- outlier.tree(titanic, nthreads=1)
otree
```

### Examining the results more closely

```{r}
## Double-check the data (last 2 outliers)
titanic[c(1147, 1164), ]
```

```{r}
## Distribution of the group from which those two outliers were flagged
titanic %>%
	filter(Pclass == 3,
		   SibSp == 0,
		   Embarked == "Q") %$%
	Fare %>%
	hist(breaks = 100, col = "navy", xlab="Fare",
		 main="Distribution of Fare within cluster")
```

```{r comment=NA}
## Get the outliers in a manipulable format
predict(otree, titanic, outliers_print = 0)[[1147]]
```

```{r comment=NA}
## To programatically get all the outliers that were flagged
pred <- predict(otree, titanic, outliers_print = 0)
only_flagged <- pred[!is.na(sapply(pred, function(x) x$outlier_score))]
```


```{r comment=NA}
## To print selected rows only
print(pred, only_these_rows = 1147)
```
### Trying different hyperparameters

```{r, eval=FALSE}
## In order to flag more outliers, one can also experiment
## with lowering the threshold hyperparameters
outlier.tree(titanic, z_outlier = 6., outliers_print = 5)
```
```{r, echo=FALSE, comment=NA}
## In order to flag more outliers, one can also experiment
## with lowering the threshold hyperparameters
outlier.tree(titanic, z_outlier = 6., outliers_print = 5, nthreads=1)
```



```{r, eval=FALSE}
## One can also lower the gain threshold, but this tends
## to result in more spurious outliers which come from
## not-so-good splits (not recommended)
outlier.tree(titanic, z_outlier = 6., min_gain = 1e-6, outliers_print = 5)
```
```{r, echo=FALSE, comment=NA}
## One can also lower the gain threshold, but this tends
## to result in more spurious outliers which come from
## not-so-good splits (not recommended)
outlier.tree(titanic, z_outlier = 6., min_gain = 1e-6, outliers_print = 5, nthreads=1)
```

