---
title: "epitweetr: user documentation"
author: "European Centre for Disease Prevention and Control (ECDC)"
output: 
  - rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{epitweetr: user documentation}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
bibliography: "epitweetr_refs.bib"
---

```{r, echo = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

library(epitweetr)
package_name <- "epitweetr"
package_desc <- packageDescription(package_name)
pkg_version <- package_desc$Version
pkg_date <- package_desc$Date
authors_str <- gsub("^ *|(?<= ) |\n| *$", "", package_desc$Authors, perl = TRUE)
author_list <- eval(parse(text = authors_str))
pkg_authors <- paste(
  format(
    author_list, 
    include = c("given", "family", "email", "comment"), 
    braces = list(email = c("<", ">,<br />"), 
    comment = c("", ""))
  ), 
  collapse = "<br /><br />"
)
pkg_maintainer <- package_desc$Maintainer
pkg_license <- package_desc$License
pkg_url <- package_desc$URL

```


## Description

The `epitweetr` package allows you to automatically monitor trends of tweets by time, place and topic. This automated monitoring aims at early detecting public health threats through the detection of signals (e.g. an unusual increase in the number of tweets for a specific time, place and topic). The `epitweetr` package was designed to focus on infectious diseases, and it can be extended to all hazards or other fields of study by modifying the topics and keywords.

The general principle behind `epitweetr` is that it collects tweets and related metadata from the Twitter Standard API versions 1.1 (https://developer.twitter.com/en/docs/twitter-api/v1/tweets/search/overview) and 2.0 (https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-recent) according to specified topics and stores these tweets on your computer on a database that can operate to calculate statistics or as a search engine. `epitweetr` geolocalises the tweets and collects information on key words, URLs, hashtags within a tweet but also entities and context detected by the Twitter API 2.0. Tweets are aggregated according to topic and geographical location. Next, a signal detection algorithm identifies the number of tweets (by topic and geographical location) that exceeds what is expected for a given day. If a number of tweets exceeds what is expected, `epitweetr` sends out email alerts to notify those who need to further investigate these signals following the epidemic intelligence processes (filtering, validation, analysis and preliminary assessment).

The package includes  an interactive web application (Shiny app) with six pages: the dashboard, where a user can visualise and explore tweets (Fig 1), the alerts page, where you can view the current alerts and train machine learning models for alert classification on user defined categories (Fig 2), the geotag page, where you can evaluate the geolocation algorithm and provide annotations for improving its performance (Fig 3), the data protection page, where the user can search, anonymise and delete tweets from the epitweetr database to support data deletion requests (Fig 4), the configuration page, where you can change settings and check the status of the underlying processes (Fig 5), and the troubleshoot page, with automatic checks and hints for using `epitweetr` with all its functionalities (Fig 6). 

On the dashboard, users can view the aggregated number of tweets over time, the location of these tweets on a map and different most frequent elements found in or extracted from these tweets (words, hashtags, URLs, contexts and entities). These visualisations can be filtered by the topic, location and time period you are interested in. Other filters are available and include the possibility to adjust the time unit of the timeline, whether retweets/quotes should be included, what kind of geolocation types you are interested in, the sensitivity of the prediction interval for the signal detection, and the number of days used to calculate the threshold for signals. This information is also downloadable directly from this interface in the form of data, pictures, and/or reports.

More information on the methodology used is available [here](https://doi.org/10.1101/2021.03.25.21254342)

**Shiny app dashboard:**

<center>Fig 1: Shiny app dashboard figure</center>

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/dashboard.png")
```


**Shiny app alerts page:**

<center>Fig 2: Shiny app alerts page</center>

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/alerts.png")
```


**Shiny app geotag evaluation page:**

<center>Fig 3: Shiny app geotag evaluation page</center>

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/geotag_evaluation.png")
```

**Shiny app data protection page:**

<center>Fig 4: Shiny app data protection page</center>

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/data_protection.png")
```

**Shiny app configuration page:**

<center>Fig 5: Shiny app configuration page</center>

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/config.png")
```


**Shiny app troubleshoot page:**

<center>Fig 6: Shiny app troubleshoot page</center>

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/troubleshoot.png")
```

## Background
### Epidemic Intelligence at ECDC
    
Article 3 of the European Centre for Disease Prevention and Control (ECDC) funding regulation and the Decision No 1082/2013/EU on serious cross-border threats to health have established the detection of public health threats as a core activity of ECDC.

ECDC performs Epidemic Intelligence (El) activities aiming at rapidly detecting and assessing public health threats, focusing on infectious diseases, to ensure EU's health security. ECDC uses social media as part of its sources to early detect signals of public health threats. Until 2020, the monitoring of social media was mainly performed through the screening and analysis of posts from pre-selected experts or organisations, mainly in Twitter and Facebook.

More information and an online tutorial are available:

[EI sources](https://www.ecdc.europa.eu/en/threats-and-outbreaks/epidemic-intelligence)

[EI tutorial](https://eva.ecdc.europa.eu/enrol/index.php?id=519) 


### Monitoring social media trends

Some signals are not detected at all or are not detected early enough by the methods explained above. The automated monitoring of social media (e.g. analysis of social media trends) allows the detection of signals that may not be detected by the monitoring of pre-selected accounts in social media, and it improves the timeliness of signal detection.

The analysis of trends from social media by topic, time and place generates relevant signals for early detection.

In 2019, ECDC developed a prototype of a manual R-based tool for the early detection of public health threats from Twitter data. The `epitweetr` package is an extension of this prototype, to allow a wider geolocation of tweets and greater automatisation, which first version was published in October 2020.

### Objectives of epitweetr

The primary objective of `epitweetr` is to use the Twitter Standard Search API version 1.1 and Twitter Recent Search API version 2 in order to detect early signals of potential threats by topic and by geographical unit.

Its secondary objective is to enable the user through an interactive web interface to explore the trend of tweets by time, geographical location and topic, including information on top words and numbers of tweets from trusted users, using charts and tables.

## Hardware requirements

The minimum and suggested hardware requirements for the computer are in the table below:

  Hardware requirements                 Minimum        Suggested
  ------------------------------------- -------------- ------------------
  RAM Needed                            8GB            16GB recommended
  CPU Needed                            4 cores        12 cores
  Space needed for 3 years of storage   3TB            5TB

The CPU and RAM usage can be configured on the Shiny app configuration page (see section *The interactive user application (Shiny app)>The configuration page*). The RAM, CPU and space needed may depend on the amount and size of the topics you request in the collection process.

## Installation

`epitweetr` is conceived to be platform independent, working on Windows, Linux and Mac. We recommend that you use `epitweetr` on a computer that can be run continuously. You can switch the computer off, but you may miss some tweets if the downtime is large enough, which will have implications for the alert detection. 

**If you need to reinstall epitweetr after activating its tasks, you must restart the machine running epitweetr first.**

Before using `epitweetr`, the following items need to be installed:

### Prerequisites for running epitweetr

-   R version 3.6.3 or higher

-   Java 1.8 eg. openjdk version "1.8" [*https://www.java.com/download/*]. The 64-bit rather than the 32-bit version is preferred, due to memory limitations. In Mac, also the Java Development Kit
[*https://docs.oracle.com/javase/9/install/installation-jdk-and-jre-macos.htm*](https://docs.oracle.com/javase/9/install/installation-jdk-and-jre-macos.htm)]

-   If you are running it in Windows, you will also need Microsoft Visual C++, however in most cases it is likely to be pre-installed:

    -   Microsoft Visual C++ 2010 Redistributable Package (x64) [*https://www.microsoft.com/en-us/download/details.aspx?id=40784*]

### Prerequisites for some of the functionalities in epitweetr

-   Pandoc, for exporting PDFs and Markdown

    -   [*https://pandoc.org/installing.html*]
-
-   Tex installation (TinyTeX or MiKTeX) (or other TeX installation) for exporting PDFs

    -   Easiest: [*https://yihui.org/tinytex/*] install from R, logoff/logon required after installation

    -   [*https://miktex.org/download*] full installation required, logoff/logon required after installation

-   Machine learning optimisation (only for advanced users)

    -   Open Blas (BLAS optimizer), which will speed up some of the geolocation processes: [*https://www.openblas.net/*] Installation instructions: [*https://github.com/fommil/netlib-Java*]

    -   **or** Intel MKL ([*https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onemkl.html*])

-   A scheduler

    -   If using Windows, you need to install the R package: taskscheduleR

    -   If using Linux, you need to plan the tasks manually
    
    -   If using a Mac, you need to plan the tasls manually

### Extra prerequisites for R developers

If you would like to develop `epitweetr` further, then the following development tools are needed:

-   Git (source code control) [*https://git-scm.com/downloads*]

-   Sbt (compiling scala code) [*https://www.scala-sbt.org/download.html*]

-   If you are using Windows, then you will additionally need Rtools: [*https://cran.r-project.org/bin/windows/Rtools/*]

### External dependencies

`epitweetr` will need to download some dependencies in order to work. The tool will do this automatically the first time the alert detection process is launched. The Shiny app configuration page will allow you to change the target URLs of these dependencies, which are the following:

- *CRAN JARs*: Transitive dependencies for running Spark, Lucene and embedded scala code. *[https://repo1.maven.org/maven2]*

- *Winutils.exe* (Windows only) This is a Hadoop binary necessary for running SPARK locally on Windows *[https://github.com/steveloughran/winutils/raw/master/hadoop-3.0.0/bin/winutils.exe]*. 

Please note that during the dependency download you will be prompted, first to stop the embedded database and then enable it again. If you are on Windows and you have activated the tasks using the 'activate' buttons on the configuration page you can performs this tasks by disabling and enabling the tasks on the 'Windows Task Scheduler'. For more information see the section 'Setting up tweet collection and the alert detection loop'

### Installing epitweetr from CRAN

After installing all required dependencies listed in the section “Prerequisites for running epitweetr”, you can install `epitweetr`:

```{r, eval=FALSE, echo=TRUE}
install.packages(epitweetr)
```

#### Environment variables

Additionally, the R environment needs to know where the Java installation home is. To check this, type in the R console:

```{r, eval=FALSE, echo=TRUE}
Sys.getenv("JAVA_HOME")
```

If the command returns null or empty, then you will need to set the Java Home environment variable, for your operating system (OS), please see your specific OS instructions. In some cases, `epitweetr` can work without setting the Java Home environment variable.

The first time you run the application, if the tool cannot identify a secure password store provided by the operating system, you will see a pop-up window requesting a keyring password (Linux and Mac). This is a password necessary for storing encrypted Twitter credentials. Please choose a strong password and remember it. You will be asked for this password each time you run the tool. You can avoid this by setting a system environment variable named ecdc\_twitter\_tool\_kr\_password containing the chosen password.

#### Launching the epitweetr Shiny app

You can launch the `epitweetr` Shiny app from the R session by typing in the R console. Replace “data\_dir” with the designated data directory which is a local folder you choose to store tweets, time series and configuration files in:

```{r, eval=FALSE, echo=TRUE}
library(epitweetr)
epitweetr_app("data_dir")
```

Please note that the data directory entered in R should have '/' instead of '\\' (an example of a correct path would be 'C:/user/name/Documents'). This applies especially in Windows if you copy the path from the File Explorer.

Alternatively, you can use a launcher: In an executable .bat or shell file type the following, (replacing “data\_dir” with the designated data directory)

R --vanilla -e epitweetr::epitweetr\_app("data\_dir")

You can check that all requirements are properly installed in the troubleshoot page. More information is available in section *The interactive user application (Shiny app)>Dashboard:The interactive user interface for visualisation>The troubleshoot page*

### Migrating to epitweetr v2
Migrating epitweetr from previous versions to version 2.0 or higher is possible without any data loss. On this section we will describe the necessary steps to perform the migration. 

**This migration is not necessary if you are installing epitweetr for the first time**.

In epitweetr v2, we have redesigned the way how tweets and series are stored. On previous versions, tweets were saved as compressed JSON files and series as RDS data frames on 'tweets' and 'series' folder, respectively. In addition, we have moved to a different storage system allowing epitweetr to work as a search engine and allowing efficient updates, deletions and faster aggregation. For doing so, data is stored using Apache Lucene indexes in the 'fs' folder. Note that during migration, Twitter data are moved to the 'fs' folder and series are left as it is. Epitweetr reports will combine data from older and new storage system.

If you have an existing installation that contains data in the previous format, you have to migrate it following the steps detailed in this section. This applies to any epitweetr version before v2.0.0. You can also check this by looking in 'tweets/geo' or 'tweets/search' folders. If there is a json.gz file, migration is needed.

The migration steps are the following:

-   Make sure the 'data collection & processing' and 'requirements & alerts' pipelines are stopped (you can see the status on the shiny configuration page). If you are using Windows and you have launched the tasks using the activate buttons, you have to manually open the Windows task scheduler and disable these two epitweetr tasks.
-   Close the Shiny App.
-   Install the new epitweetr version (if you have not yet done it).
-   Go to the configuration page in the Shiny app.
-   Click on the 'Run dependencies' button.
-   Activate the 'Requirements and alerts' process. If you are on Windows, you can click on the activate button or you can run on a new R session executing the following command: 
     -     epitweetr::detect_loop("data\_dir")
-   In the line of the dependencies tasks under 'Requirements & alerts' table, there will be a message reporting the status of the migration and download of necessary dependencies from the selected maven mirror. It will indicate whether some of the epitweetr tasks must be stopped or activate it to complete migration and download of dependiencies.
-   First, it will download the dependencies for which the embedded epitweetr database should be stop. If the embedded database is running, a message will appear asking you to stop the database ('epitweetr database' pipeline).
-   After all dependencies have been downloaded, a message will appear asking to activate the embedded database ('epitweetr database'), please refer to the section 'Setting up tweet collection and the alert detection loop' for details.
-   Once the epitweetr database is running, the migration will start. Epitweetr will report the estimated remaining time to finish the migration. Be aware that depending on the size of your data, this can take several days for which the machine running epitweetr cannot be turned off.
-   As files are migrated to the new storage system, these are moved from the folder 'tweets/search' and 'tweets/geo' to 'tweets/search\_archive' and 'tweets/geo\_archive'.
-   Once the migration is done, you can backup and delete the \_archive folders.
-   You can test that your data has been successfully migrated by searching for tweets on particular periods using the 'Data protection' page.
-   Click on the button 'Run GeoNames' and 'Run languages' and wait for them to finish.
-   You can now activate the 'Data collection & processing'.

### Setting up tweet collection and the alert detection loop

In order to use `epitweetr`, you will need to collect and process tweets, run the epitweetr database and run the requirements and alerts pipeline. Further details are also available in subsequent sections of the user documentation. A summary of the steps needed is as follows:

-   Launch the Shiny app (from the R console) 
```{r, eval=FALSE, echo=TRUE}
library(epitweetr)
epitweetr_app("data_dir")
```

-   On the configuration page of the Shiny app, in the manual tasks of the "Requirements & alerts pipeline", click on "Run dependencies", "Run GeoNames" and "Run languages" (their status will change to “pending”). This allows the requirements & alerts pipeline to download the elements needed. As long as no languages are added and no updates are available in geonames.org, these tasks have to be run only the first time you install `epitweetr.

```{r echo=FALSE, fig.cap="", out.width = '50%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/run_tasks.png")
```

-   Set up the Twitter authentication using a Twitter account or a Twitter developer app, see section *Collection of tweets>Twitter authentication* for more details

-   If you want to use the Twitter API V2 you have to request access from the Twitter developer portal. More information on [*https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api*]

-   Activate the embedded database 

    -   Windows: Click on the “Epitweetr database” activate button

    -   Other platforms: In a new R session run the following command
    
```{r, eval=FALSE, echo=TRUE}
library(epitweetr)
fs_loop("data_dir")
```
   -   You can confirm that the embeded database is running if the "Embeded database" status is "Running" in the Shiny app configuration page and "true" in the Shiny app troubleshoot page.

-   Activate the tweet collection and data processing

    -   Windows: Click on the “Data collection & processing” activate button

    -   Other platforms: In a new R session run the following command
    
```{r, eval=FALSE, echo=TRUE}
library(epitweetr)
search_loop("data_dir")
```

   -   You can confirm that the tweet collection is running if the "Data collection & processing" status is "Running" on the Shiny app configuration page (green text in screenshot above) and "true" in the Shiny app troubleshoot page.
   
-   Activate the epitweetr database

    -   Windows: Click on the “epitweetr database” activate button

    -   Other platforms: In a new R session run the following command
    
```{r, eval=FALSE, echo=TRUE}
library(epitweetr)
fs_loop("data_dir")
```

   -   You can confirm that the epitweetr database is active if the "epitweetr database" status is "Running" on the Shiny app configuration page (green text in screenshot above) and "true" in the Shiny app troubleshoot page.

-   Activate the Requirements & alerts pipeline:

    -   Windows: Click on the “Requirements & alerts pipeline” activation button
    -   Other platforms: In a new R session run the following command
```{r, eval=FALSE, echo=TRUE}
library(epitweetr)
detect_loop("data_dir")
```

   -   You can confirm that the requirements & alerts pipeline is running if the "Requirements & alerts pipeline" status is "Running" in the Shiny app configuration page and "true" in the Shiny app troubleshoot page.

-   You will be able to visualise tweets after "Data collection & processing" and "epitweetr database" are activated and the languages task has finished successfully.

-   You can start working with the generated signals. **Happy signal detection!**

For more details you can go through the section *How does it work? General architecture behind `epitweetr`*, which describes the underlying processes behind the tweet collection and the signal detection. Also, the section *"The interactive Shiny application (Shiny app)>The configuration page"* describes the different settings on the configuration page.


## How does it work? General architecture behind epitweetr

The following sections describe in detail the above general principles. The settings of many of these elements can be configured in the Shiny app configuration page, which is explained in the section *The interactive Shiny application (Shiny app)>The configuration page*.

###  Collection of tweets


#### Use of the Twitter Standard Search API version 1.1 and Twitter Recent Search API version 2.0

`epitweetr` uses the Twitter Standard Search API version 1.1 and/or Twitter Recent Search API version 2.0. The advantage of these APIs is that these are a free service provided by Twitter enabling users of `epitweetr` to access tweets free of charge. The search API is not meant to be an exhaustive source of tweets. It searches against a sample of recent tweets published in the past 7 days and it focuses on relevance and not completeness. This means that some tweets and users may be missing from search results.

While this may be a limitation in other fields of public health or research, the `epitweetr` development team believe that for the objective of signal detection a sample of tweets is sufficient to detect potential threats of importance in combination with other type of sources.

Other attributes of the Twitter Standard Search API version 1.1 include:

-   Only tweets from the last 5–8 days are indexed by Twitter

-   A maximum of 180 requests every 15 minutes are supported by the Twitter Standard Search API (450 requests every 15 minutes if you are using the Twitter developer app credentials; see next section)

-   Each request returns a maximum of 100 tweets and/or retweets

Other attributes of the Twitter Recent Search API version 2.0  include:

-   Only tweets from the last week days are indexed by Twitter

-   A maximum of 300 requests every 15 minutes are supported

-   Each request returns a maximum of 100 tweets and/or retweets

-   500.000 tweets per month in the [essential access level](https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api#v2-access-level)

If you are using both endpoints `epitweetr` will alternate between them when the limits are hit.


#### Twitter authentication

You can authenticate the collection of tweets by using a **Twitter account** (this approach utilises the rtweet package app) or by using a **Twitter application**. For the latter, you will need a Twitter developer account, which can take some time to obtain, due to verification procedures. We recommend using a Twitter account via the rtweet package for testing purposes and short-term use, and the Twitter developer application for long-term use.

-   **Using a Twitter account:** delegated via rtweet (user authentication)

    -   You will need a Twitter account (username and password)

    -   The rtweet package will send a request to Twitter, so it can access your Twitter account on your behalf

    -   A pop-up window will appear where you can enter your Twitter user name and password to confirm that the application can access Twitter on your behalf. You will send this token each time you access tweets. If you are already logged in Twitter, this pop-up window may not appear and automatically take the credentials of the 'active' Twitter account in the machine
    
    -   You can only use Twitter API version 1.1

-   **Using a Twitter developer app:** via `epitweetr` (app authentication)

    -   If you have not done so already, you will need to create a Twitter developer account: [*https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api*]

    -   Follow the instuctions, answer the questions to activate the Twitter API v2 using Essential access.

    -   Next, you will create a Project and an associated developer App during the onboarding process, which will provide you a set of credentials that you will use to authenticate all requests to the API. 

    -   Make a note of your OAuth settings

        -   Add them to the configuration page in the Shiny app (see image below)

        -   With this information `epitweetr` can request a token at any time directly to Twitter. The advantage of this method is that the token is not connected to any user information and tweets are returned independently of any user context.
        
        -   With this app, you can perform 450 requests every 15 minutes instead of the 180 requests every 15 minutes that a Twitter account allows.  

        -   You can activate Twitter API version 2.0 in the config page
        
```{r echo=FALSE, fig.cap="", out.width = '50%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/twitter_authentication.png")
```

#### Topics and tweet collection queries

After the Twitter authentication, you need to specify a list of topics in `epitweetr` to indicate which tweets to collect. For each topic, you have one or more queries that `epitweetr` uses to collect the relevant tweets (e.g. several queries for a topic using different terminology and/or languages). 

A query consists of keywords and operators that are used to match tweet attributes. Keywords separated by a space indicate an AND clause. You can also use an OR operator. A minus sign before the keyword (with no space between the sign and the keyword) indicates the keyword should not be in the tweet attributes. While queries can be up to 512 characters long, best practice is to limit your query to 10 keywords and operators and limit complexity of the query, meaning that sometimes you need more than one query per topic. If a query surpasses this limit, it is recommended to split the topic in several queries. 

`epitweetr` comes with a default list of topics as used by the ECDC Epidemic Intelligence team at the date of package generation (15^th^ of December, 2021). You can view details of the list of topics in the Shiny app configuration page (see screenshot below). In addition, the colour coding in the downloadable file allows users to see if the query for a topic is too long (red colour) and the topic should be split in several queries.

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/topics.png")
```

On the configuration page, you can also download the list of topics, modify and upload it to `epitweetr`. The new list of topics will then be used for tweet collection and visible in the Shiny app. The list of topics is an Excel file (\*.xlsx) as it handles user-specific regional settings (e.g. delimiters) and special characters well. You can create your own list of topics and upload it too, noting that the structure should include at least:

-   The name of the topic, with the header “Topic” in the Excel spreadsheet. This name should include alphanumeric characters, spaces, dashes and underscores only. Note that it should start with a letter.

-   The query, with the header “Query” in the Excel spreadsheet. This is the query `epitweetr` uses in its requests to obtain tweets from the Twitter Standard Search API. See above for syntax and constraints of queries.

The topics.xlsx file additionally includes the following fields:

-   An ID, with the header “\#” in the Excel spreadsheet, noting a running integer identifier for the topic.

-   A label, with the header “Label” in the Excel spreadsheet, which is displayed in the drop-down topic menu of the Shiny app tabs.

-   An alpha parameter, with the header “Signal alpha (FPR)” in the Excel spreadsheet. FPR stands for "false positive rate". Increasing the alpha will decrease the threshold for signal detection, resulting in an increased sensitivity and possibly obtaining more signals. Setting this alpha can be done empirically and according to the importance and nature of the topic.

-   "Length\_charact" is an automatically generated field that calculates the length of all characters used in the query. This field is helpful as a request should not exceed 500 characters.

-   "Length\_word" indicates the number of words used in a request, including operators. Best practice is to limit your number of keywords to 10.

-   An alpha parameter, with the header “Outlier alpha (FPR)” in the Excel spreadsheet. FPR stands for “false positive rate”. This alpha sets the false positive rate for determining what an outlier is when downweighting previous outliers/signals. The lower the value, the fewer previous outliers will potentially be included. A higher value will potentially include more previous outliers.

-   "Rank" is the number of queries per topic

```{r echo=FALSE, fig.cap="", out.width = '80%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/Example_topics_list.png")
```

When uploading your own file, please modify the topic and query fields, but do not modify the column titles.


#### Scheduled plans to collect tweets

As a reminder, `epitweetr` is scheduled to make 180 requests (queries) to Twitter every 15 minutes with the user authentication (or 450 (v1.1) or 300 (v2) requests every 15 minutes if you are using Twitter developer app credentials depending on the API version you use). Each request can return 100 tweets. The requests return tweets and retweets. These are returned in JSON format, which is a light-weight data format.

In order to collect the maximum number of tweets, given the Standard Search API limitations, and in order for popular topics not to prevent other topics from being adequately collected, `epitweetr` uses “search plans” for each query.

The first “search plan” for a query will collect tweets from the current date-time backwards until 7 days (7 days because of the Standard Search API limitation) before the current “search plan” was implemented. The first “search plan” is the biggest, as no tweets have been collected so far.

All subsequent “search plans” are scheduled intervals that are set up in the configuration page of the `epitweetr` Shiny app (see section *The interactive Shiny app > the configuration page > General*). For illustration purposes, let us consider the search plans are scheduled at four-hour intervals. The plans collect tweets for a specific query from the current date-time back until four hours before the date-time when the current “search plan” is implemented (see image below). `epitweetr` will make as many requests (each returning up to 100 tweets) during the four-hour interval as needed to obtain all tweets created within that four-hour interval.

For example, if the “search plan” begins at 4 am on the 10^th^ of November 2021, `epitweetr` will launch requests for tweets corresponding to its queries for the four-hour period from 4 am to midnight on the 10^th^ of November 2021. `epitweetr` starts by collecting the most recent tweets (the ones from 4 am) and continues backwards. If during the four-hour time period between 4am and midnight the API does not return any more results, the “search plan” for this query is considered completed.

```{r echo=FALSE, fig.cap="", out.width = '80%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/Schedule_plan_image.png")
```

However, if topics are very popular (e.g. COVID-19 in 2020 and 2021), then the “search plan” for a query in a given four-hour window may not be completed. If this happens, `epitweetr` will move on to the “search plans” for the subsequent four-hour window, and put any previous incomplete “search plans” in a queue to execute when “search plans” for this new four-hour window are completed.

Each “search plan” stores the following information:

  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Field            Type        Description
  ---------------- ----------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  expected\_end    Timestamp   End DateTime of the current search window

  scheduled\_for   Timestamp   The scheduled DateTime for the next request. On plan creation this will be the current DateTime and after each request this value will be set to a future DateTime. To establish the future DateTime, the application will estimate the number of requests necessary to finish. If it estimates that N requests are necessary, the next schedule will be in 1/N of the remaining time.

  start\_on        Timestamp   The DateTime when the first request of the plan was finished

  end\_on          Timestamp   The DateTime when the last request of the plan was finished if that request reached a 100% plan progress.

  max\_id          Long        The max Twitter id targeted by this plan, which will be defined after the first request

  since\_id        Long        The last tweet id returned by the last request of this plan. The next request will start collecting tweets before this value. This value is updated after each requests and allows the Twitter API to return tweets before min\_time(p~i~)

  since\_target    Long        If a previous plan exists, this value stores the first tweet id that was downloaded for that plan. The current plan will not collect tweets before that id. This value allows the Twitter API to return tweets after p~i~-time\_back

  requests         Int         Number of requests performed as part of the plan

  progress         Double      Progress of the current plan as a percentage. It is calculated as (current\$max\_id - current\$since\_id)/(current\$max\_id - current\$since\_target). If the Twitter API returns no tweets the progress is set to 100%. This only applies for non error responses containing an empty list of tweets.
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

`epitweetr` will execute plans according to these rules:

-   `epitweetr` will detect the newest unfinished plan for each search query with the scheduled\_for variable located in the past.

-   `epitweetr` will execute the plans with the minimum number of requests already performed. This ensures that all scheduled plans perform the same number of requests.

-   As a result of the two previous rules, requests for topics under the 180 limit of the Twitter Standard Search API (or 450 if you are using Twitter developer app authentication) will be executed first and will produce higher progress than topics over the limit.

The rationale behind this is that topics with such a large number of tweets that the 4-hour search window is not sufficient to collect them, are likely to already be a known topic of interest. Therefore, priority should be given to smaller topics and possibly less well-known topics.

An example is the COVID-19 pandemic in 2020. In early 2020, there was limited information available regarding COVID-19, which allowed detecting signals with meaningful information or updates (e.g. new countries reporting cases or confirming that it was caused by a coronavirus). However, throughout the pandemic, this topic became more popular and the broad topic of COVID-19 was not effective for signal detection and was taking up a lot of time and requests for `epitweetr`. In such a case it is more relevant to prioritise the collection of smaller topics such as sub-topics related to COVID-19 (e.g. vaccine AND COVID-19), or to make sure you do not miss other events with less social media attention.

If search plans cannot be finished, several search plans per query may be in a queue:

```{r echo=FALSE, fig.cap="", out.width = '80%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/Search_plan_queues.png")
```

This design can have the draw back of slowing down big topics collection since epitweetr is trying to rebuil last 7 days of history. If you do not care about rebuilding history on a particular point of time you can click into the "Dismiss past tweets" button who will discard all plans and will start collecting new data.

### Geolocation

In a parallel process to the collection of tweets, `epitweetr` attempts to geolocate all collected tweets using a supervised machine learning process. This process runs in automatically after tweets are collected.

`epitweetr` stores two types of geolocation for a tweet: tweet location, which is geolocation information within the text of a tweet (or a retweeted or quoted tweet), and user location from the available metadata. For signal detection, the best location is used while in the dashboard both types can be visualised. 

#### Geolocation based on tweet location

The tweet location is extracted and stored by `epitweetr` based on the geolocation information found within a tweet text. In case of a retweet or quoted tweet, it will extract the geolocation information from the original tweet text that was retweeted or quoted. If neither is available, no tweet location is stored based on tweet text.

`epitweetr` identifies if a tweet text contains reference to a particular location by breaking down the tweet text into sets of words and evaluating those which are more likely to be a location by using a machine learning model. If several parts of the text are likely to be a location, epitweetr will chose the one closest to a topic. After the location candidate has been identified epitweetr matches these words against a reference database, which is geonames.org. This is a geographical database available and accessible through various web services, under a Creative Commons attribution license. The geoNames.org database contains over 25,000,000 geographical names. `epitweetr` uses by default those limited to currently existing ones and those with a known population (so just over 500,000 names). You can override this default on the Shiny app configuration page, by unchecking "Simplified geonames". The database also contains longitude and latitude attributes of localities and variant spellings (cross-references), which are useful for finding purposes, as well as non-Roman script spellings of many of these names.

The matches can be performed at any level of administrative hierarchy. The matching is powered by Apache Lucene, which is an open-source high-performance full-featured text search engine library.

To validate the candidate against geonames a score is associated with a greater probability that a match is correct. A score is:

-   Higher if unusual parts of the name are matched

-   Higher if several administrative levels are matched

-   Higher if location population is bigger

-   Higher for countries and cities vs administrative levels

-   Higher for capital letter acronyms like NY

-   Lower for words that are more likely to be other (non-geographical) kinds of words. For example “Fair Play” town in Colorado. This is achieved by using language models provided by fasttext.cc.

You can select which languages you would like to check for other kinds of words in, by selecting the active language desired within the configuration page of the Shiny app and clicking on the "+" icon:

```{r echo=FALSE, fig.cap="", out.width = '80%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/config_languages.png")
```

In addition, you can unselect languages by selecting the language within the configuration page of the Shiny app and clicking on the "-"  icon. 

**All default languages (English, French, Spanish and Portuguese) must be downloaded before adding new languages or deleting any of the default languages.**


A minimum score (the "geolocation threshold") can be globally set in the general settings on the configuration page to reduce the number of false positives (see image). All geolocations with a smaller score than the geolocation threshold will be discarded by the algorithm as tweet location. If there is more than one match over the minimum score, then the match with the highest score will be chosen.

The threshold is empirically chosen and can be evaluated against a human read of tweets and tweet locations, on the geotag evaluation page.

#### Geolocation based on user location

Different types of user locations are available from the metadata provided through the Twitter Standard Search API. epitweetr selects the best user location for the aggregated files using the following order:

  - the user’s exact or approximate location at time of the tweet (provided by the API)
  
  - if the user’s location is not available and the tweet is a retweet or a quoted tweet, then the user’s exact or approximate location at time of the retweet/quoted tweet is used (provided by the API)
  
  - if not available, then the user declared location is used
  
  - if not available, then the “home” in the public profile is used. 


With exact locations, the longitude and latitude are provided. If it is the estimated location, `epitweetr` calculates the longitude and latitude from GeoNames.org.

If user location information provided by the API is not available, then `epitweetr` will calculate longitude and latitude from the user declared location or the place name given in the user “public profile”, using GeoNames.org.

#### Improving and evaluating geolocation performance

On the Geotag page, `epitweetr` will allow you to download the data that was used to build the classifiers for location identificaion. The predefined annotations are based on location vs non location words. Locations are extracted from geonames.org database. Non location words are obtained from common words on the downloaded models that are not present on GeoNames. You will be able to add tweets to the annotation database and to manually annotate them until you reach the expected level of performance.

To help you on the evaluation process, `epitweetr` will calculate standard machine learning metrics for evaluating its capacity to identify the right location in texts. These metrics are separately calculated by different type of texts: tweet location, tweet text, tweet user description and total.


#### Stored geolocated tweet information

The geolocation of the match is stored as a country code (using the ISO 3166 standard) and a longitude and latitude associated with the exact geolocation in the aggregated data.

### Most frequent elements found in and extracted from tweets

Epitweetr counts five types of elements on tweets

  - Words: Words present in tweets. Words are then aggregated in the topwords time series

  - Hashtags: hashtags present in tweets. Hashtags are then aggregated in the hashtags time series

  - URLs: URLs present in tweets. URLs are then aggregated in the URLs time series

  - Entities: Entities are automatically extracted and identified by the Twitter API v2. These are then aggregated in the entities time series. This information is only available when the API v2 is activated and properly configured. As per [*Twitter documentation*](https://developer.twitter.com/en/docs/labs/annotations) "entities are comprised of people, places, products, and organizations [...]. They are programmatically assigned based on what is explicitly mentioned in the tweet text".  

  - Contexts: Contexts are automatically extracted and identified by the Twitter API v2. These are then aggregated in the contexts time series. This information is only available when the API v2 is activated and properly configured. As per [*Twitter documentation*](https://developer.twitter.com/en/docs/labs/annotations) "[Context are] derived from the analysis of a tweet’s text and will include a domain and entity pairing which can be used to discover tweets on topics that may have been previously difficult to surface [...]. [It uses] a list of 50+ domains to categorize tweets".  

Note that, differently to the other visualisation figures, the most frequent elements found in and extracted from tweets are always based on the geolocation related to “tweet location” and not to “user location” regardless of the filter selected in the dashboard. Topwords & hashtags and entities & context are shown in the same figure, respectively.

#### Aggregation of data

The aggregation process produces data on five subfolders of "fs": geolocated, country\_counts, topwords, urls, hashtags, entities and contexts. These folders are splitted on week subfolders and each contains a Lucene index with the aggregated information. This data can be extracted as a dataframe through the public package function 'get\_aggregates'

In the geolocated time series, the number of tweets or retweets are stored by topic, date, tweet text, tweet geolocation, tweet longitude and tweet latitude, and user geolocation, user longitude and user latitude. Each of these entries also has the country associated with the tweet text geolocation and the country associated with the user geolocation . Note that tweets without geolocation information are also included.

The country\_counts serie is used to create the trendline in the Shiny app. This is a smaller time serie, without the longitude and latitude information, and includes the number of tweets by hour within a day, by country (according to tweet location or user location), topic (see screenshot), and whether a tweet was a retweet or not. The known\_retweets and known\_original fields give the number of tweets or retweets from a list of “important users”. In this file, tweets without geolocation are also included. Including tweets without geolocation information enables you to view all tweets when selecting “world” as a region, regardless of whether geolocation was successful or not.

The aggregation by top element is stored in the topwords, URLs, hashtags, contexts and entities fs subfolders which contain the number of tweets and/or retweets by topic, top element, date, country of tweet location and whether a tweet was a retweet or not (see screenshot).

### Signal detection

The main objective of `epitweetr` is to detect signals in the observed data streams, i.e. counts in the aggregated time series that exceed what is expected. For detecting signals, `epitweetr` uses an extended version of the EARS (Early Aberration Reporting System) algorithm [@fricker_comparing_2008], which in what follows is denoted by `eears` (extended EARS). This algorithm is part of the R package `surveillance` [@salmon_monitoring_2016].

As a default it uses a moving window of the past seven days to calculate a threshold. If the count for the current day exceeds this threshold, then a signal is generated.

#### Details of the algorithm underlying signal detection

The `eears` algorithm is applied on the counts from the past seven 24-hour blocks prior to the current 24 hour block of the signal detection. The running mean and the running standard deviation are calculated:

$$
\overline{y}_{0} = \frac{1}{7}\sum_{t=-7}^{-1} y_{t} \quad\text{and}\quad  s_{0}^{2} = \frac{1}{7 - 1}\sum_{t=-7}^{-1}{(y_{t} - \overline{y}_{0})}^{2},
$$

where $y_{t}, t=\ldots, -2, -1, 0$ denotes the observed count data time series with time index $0$ denoting the current block. Furthermore, the time index $-7,\ldots, -1$ denote the seven blocks prior to the current block.

Under the null hypothesis of no spikes, it is assumed that the $y_t$ are identically and independently $N(\mu, \sigma^2)$ distributed with unknown mean $\mu$ and unknown variance $\sigma^2$. Hence, the upper limit of a simple one-sided $(1-\alpha)\times$ 100% plug-in prediction interval for $y_0$ based on $y_{-7},\ldots,y_{-1}$ is given as 
$$
U_{0} = {\overline{y}}_{0} + z_{1 - a} \times s_{0},
$$
where $z_{1 - a}$ is the (1 − α)- quantile of the standard normal distribution. An alert is raised if $y_{0} > U_{0}$ . If one uses α=0.025, then this corresponds to investigating, if $y_{0}$ exceeds the estimate for the mean plus 1.96 times the standard deviation. However, as pointed out by @allevius_prospective_2017, the correct approach would be to compare the observation to the upper limit of a two-sided 95% prediction interval for $y_{0}$, because this respects both the sampling variation of a new observation *and* the uncertainty originating from the parameter estimation of the mean and variance. Hence, the statistical appropriate form is to compute the upper limit by
$$
U_{0} = \overline{\ y_{0}} + t_{1 - a}(7 - 1)\times s_{0} \times   \sqrt{1 + \frac{1}{7}}.
$$

where $t_{1 - a}(k - 1)$ denotes the 1 − α quantile of the t-distribution with k − 1 degrees of freedom.

#### Downweighting previous signals

If previous signals are included without modification in the historic values when calculating the running mean and standard deviation for the signal detection, then the estimated mean and standard deviation might become too large. This may mean that important current signals will not be detected. To address this issue, `epitweetr` downweights previous signals, such that the mean and standard deviation estimation is adjusted for such outliers using an approach similar to that used in the @farrington_statistical_1996. Historic values that are not identified as previous signals are given a weight of "1". Similarly, historic values identified as signals are given a weight lower than one and a new fit is performed using these weights (scaled s.t. they again sum to 7 observations). Details on the downweighting procedure can be found in [Annex I](#annexI) of this user documentation.


#### Timing of signal detection

Signal detection is carried out based on “days”, which are moving windows of 24 hours, moving according to the detect span (see also section *The interactive user application (Shiny app) > The configuration page > General*). The baseline is calculated on these “days” from -1 to -8 (if the current "day" is zero).

Signals are generated according to the detect span (see section *The interactive user application (Shiny app) > The configuration page > General*), with

-   general email alerts either following this detect span (e.g. if the detect span was four hours, the email alerts will be sent every four hours)

-   email alerts in real-time. In these email alerts, previously generated signals will be omitted.

The different types of email alerts for each user can be specified in the configuration page (see section *The interactive user application (Shiny app) > The configuration page > General*).

#### The alpha parameter: the false positive rate of the signal detection

A key attribute of signal detection is the ability of an algorithm to detect true threats or events without overloading the investigators with too many false positives. In this way, the alpha parameter determines the threshold of the detection interval. If the alpha is high, then more potential signals are generated and if the alpha is low fewer potential signals are generated (but potential threats or events could be missed). The setting of the alpha is often done empirically, and depends also on the resources of those investigating the signals and the importance of missing a potential threat or event.

There is a global alpha, that can be set/changed in the `epitweetr` configuration page under "Signal false positive rate" (see section *The interactive user application (Shiny app) > The configuration page > General*). Additionally, the default alpha can be overridden in the topics list. Here, if you like, you can associate each topic with a specific alpha, depending on the estimated public health importance of the topic or potential associated event or threat.

#### Bonferroni correction

To account for multiple testing, for country-specific signal detection, as a default, the alpha is divided by the number of countries. For continent-specific signal detection, the alpha is divided by the number continents. This is a Bonferroni correction for multiple testing.

To override this, you can uncheck "Bonferroni correction" in the "Signal detection" part of the configuration page in the Shiny app.


#### Using same weekdays as baseline

It is possible that there is a "day of the week effect", where more tweets may be tweeted on a given day of the week (e.g. Monday) than on other days. To avoid this, you can also choose to calculate the baseline not on consecutive days, but on the past N days that correspond to the same 24 hour window N days back. This way if N = 7, the baseline is calculated using the “days” from -7, -14, -21, -28, -35, -42, -49 and -56 (if the current "day" is zero).

This option is on the configuration page of the Shiny app "Default same weekday baseline".

#### Sending email alerts

Emails containing a list of signals detected are sent automatically by `epitweetr` according to the detect span and the subscribers list. Due to the time necessary to collect, geolocate and aggregate the tweets, email alerts will miss the most recent tweets that have not yet gone to these processes. The lag between tweets and alerts is expected to be less than (2 * ( collect\_span ) + detect\_span) which should be 3h30 using default values.

The email alerts will include the following information on the signals for each topic:

-   The date and hour the signal was detected

-   The geographical location(s) where the signal was detected

-   The most frequent elements (top words, urls, hashtags, contexts and entities) in the tweets

-   The number of tweets and the threshold

-   The percentage of tweets from important users

-   Information on the settings, such as: was the Bonferroni correction used, was the same weekday baseline used, were retweets included, etc.

-   The alert category estimated by epitweetr based on user annotations

This information is also available in the alerts page of the Shiny app. 

The subscribers can receive real-time alerts (i.e. as soon as the detection loop is finalised) or scheduled alerts (e.g. once or twice a day). The subscribers list can be changed in the configuration page by downloading the Excel spreadsheet. This file has the following variables:

  - “User”: name of the subscriber (e.g. Jane Doe).

  - “Email”: email of the subscriber (e.g. jane.doe@email.com).

  - “Topics”: list of topics for which the subscriber will receive scheduled alerts. The names used must match the column “Topic” in the list of topics.

  - “Excluded”: topic for which the subscribers will not receive scheduled alerts.

  - “Real time Topics”: list of topics for which the subscriber will receive real-time alerts.

  - “Regions”: list of regions for which the subscriber will receive scheduled alerts.

  - “Real time Regions”: list of regions for which the subscriber will receive real-time alerts.

  - “Alert Slots”: these are the detection loop slots after which the subscriber will receive the scheduled alert. Available slots can be taken from “Launch slots” in the “General” section of the configuration page. If no value is included, the subscriber will receive real-time alerts for all topics and regions, even if there are real-time topics or regions specified in the Excel spreadsheet.

  - “One tweet alerts” (yes/no): Whether the user wants to receive alerts containing just one tweet
  
  - “Topics ignoring one tweet alerts”: Topics that will be ignored for one-tweet alerts 
  
  - “Regions ignoring one tweet alerts”: Regions that will eb ignored for one-tweet alerts
  
When including more than one topic and/or region in the subscribers list, these should be separated by semi-colon (;) with no spaces (e.g. Ebola;infectious diseases;dengue). The names must match the column “Topics” in the list of topics and the column “Name” in the country/region list from the configuration page.

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/subscribers.png")
```

###  Folder structure

`epitweetr` stores and aggregates tweets. When launching the application, you have to designate the  and the configuration in the “data folder” you have to designate when launching the application.

Within the **data** folder there are three JSON files:

-   properties.json, generated from the information from the General properties of the Shiny app

-   topics.json managed by the search loop: it keeps a track of tweet collection plans and progress

-   tasks.json managed by the detect loop: it keeps information and status of the different tasks done by this process.

There are also the following subfolders:

-   "fs", which contains the Lucene indexes for storing tweets and aggregated time series

-   "geo", which contains the GeoNames data as text, index files, tweets waiting to be geolocated and settings about the geolocation algorithm.

-   "alert-ml", which contains the machine learning models used for alert classification

-   "hadoop", which contains Spark dependencies for Windows operating systems

-   "jars", which contains collections of Java dependencies needed in the geolocation and aggregate processes

-   "languages", which contains fasttext files indexes and models used to perform geolocation in the tweet text

-   "stats", which contains JSON files with statistics used to optimise the aggregate process by linking tweet files and posted dates of tweets

-   "alerts", which contains JSON files of alerts detected by the requirements & alerts pipeline

-   "collections", which contains the aggregations produced by `epitweetr`


#### Fs folder &gt; tweets
In the fs folder, the subfolder “tweets” contains Lucene indexes storing the content of the collected tweets and the geolocation information.

The *tweets* contains subfolders for each week.

The *geolocated folder* contains compressed JSON files with geolocated information produced by the geolocation algorithm.

#### Fs folder &gt; country_counts, contexts, entities, geolocated, hashtags, topwords, urls

In the folders *country_counts*, *contexts*,  *entities*, *geolocated*, *hashtags*, *topwords*, *urls*, `epitweetr` stores the aggregated data of the geolocated tweets as well as the top elements identified.

Each folder is named matching the respective time series and emitted by ISO week of date of publication. Each weekly folder contains a Lucene index.

This is the aggregate information as described in the section “How does it work? General architecture of epitweetr > Aggregation”.

## The interactive user application (Shiny app)

You can launch the `epitweetr` interactive user application (Shiny app) from the R session by typing in the R console (replace “data\_dir” with the desired data directory):

```{r, eval=FALSE, echo=TRUE}
epitweetr_app("data_dir")
```


Alternatively, you can use a launcher: In an executable bat or sh file  put the following content, (replacing “data\_dir” with the expected data directory)

R --vanilla -e epitweetr::epitweetr\_app('data\_dir')


The `epitweetr` interactive user application has six pages:

-   The dashboard, where a user can visualise and explore tweets

-   The alerts page, where a user can view the current alerts and train machine learning algorithms for alert classification on user's defined categories

-   The geotag evaluation page, where a user can evaluate the geolocation algorithm and provide annotations for improving its performance

-   The data protection page, where a user can search, anonymise and delete tweets from the epitweetr database to support data deletion requests 

-   The configuration page, where a user can change settings and check the status of the underlying processes

-   The troubleshoot page, with with automatic checks and hints for using `epitweetr` with all its functionalities


### Dashboard: The interactive user interface for visualisation

The dashboard is where you can interactively explore visualisations of tweets. It includes a line graph (trend line) with alerts, a map and top words, URLs, hashtags, entities and contexts of tweets for a given topic. After selecting the parameters, you have to click on the 'Run' button in order to see or refresh the report.

In order to interactively explore the data, you can select from several filters, such as topics, countries and regions, time period, time unit, signal confidence and days in baseline.

Note that whatever options/settings you select on the dashboard, will have no effect on the alert detection. The alert detection settings are all selected in the configuration page of the Shiny app.

#### Filters

**Topics**

You can select one item from the drop-down list of topics, which is populated by what is specified in the topics on the configuration page. You can also start typing in the text field and select the topics from the filtered dropdown list.

**Countries & regions**

```{r echo=FALSE, fig.cap="", out.width = '40%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/topics_countries.png")
```

If you select **World (all)**, all tweets are displayed regardless of their geolocation. You can select an individual country, you can select regions and subregions, and you can select several items at the same time. You can also start typing in the text field and select the geographical item from the drop-down list.

**Period**

```{r echo=FALSE, fig.cap="", out.width = '40%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/period.png")
```

You can select from the past 7 (the default), 30, 60 or 180 days. You can also select "custom" and a calendar option to select study period will appear. These periods will be the time period for inclusion in the visualisations. When selecting custom period, please ensure that the first date is at least one day before the second date.

**Time unit**

```{r echo=FALSE, fig.cap="", out.width = '30%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/time_unit.png")
```

You can display the timeline for the number of tweets with weeks or days as units of time. The default is days.

**Include Retweets/quotes**

```{r echo=FALSE, fig.cap="", out.width = '30%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/include_retweets.png")
```

By default, retweets are not included in any of the visualisations. If “include retweets/quotes” is checked, the visualisations display results of tweets and retweets/quotes. Otherwise, the visualisations display only tweets (without retweets/quotes).

**Location type**

```{r echo=FALSE, fig.cap="", out.width = '30%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/location_type.png")
```

Tweets are geolocated into regions, subregions and countries. “Location type” indicates what should be used for geolocation:

-   Tweet: this comprises geographical information contained within the tweet text or if not available, geographical information contained within the retweet/quoted text, if applicable.

-   User: this is geographical information obtained from the user location. In order of priority, this is the user’s location at time of the tweet, the user’s API location or the “home” in the public profile if none of these is available.

-   Both: the geographical information used for a tweet will be, in order of priority, the location within the tweet text, but if not available the user location.

**Signal detection false positive rate**

```{r echo=FALSE, fig.cap="", out.width = '30%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/signal_detection_fpr.png")
```

Using the slider, you can explore the differences in the signals generated when changing the alpha parameter for the false positive rate. Note that this will not change the signal false positive rate for the alert emails. This is just a tool for the user to explore this parameter. The default is 0.025. A higher false positive rate will increase the sensitivity and possibly the number of signals detected, and vice versa.

**Outlier false positive rate and outlier downweight strength**

```{r echo=FALSE, fig.cap="", out.width = '40%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/outlier_dashboard.png")
```

The outlier false positive rate relates to the false positive rate for determining what an outlier is when downweighting previous outliers/signals. The lower the value, the fewer previous outliers will potentially be included. A higher value will potentially include more previous outliers.

The outlier downweight strength determines how much an outlier will be downweighted by. The higher the value the greater the downweighting. For more information please see [Annex I](#annexI).

**Bonferroni correction**

```{r echo=FALSE, fig.cap="", out.width = '30%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/Bonferroni_correction.png")
```

The Bonferroni correction is selected by default. It accounts for false positive signal detection through multiple testing. For country-specific signal detection, the alpha is divided by the number of countries. For continent-specific signal detection, the alpha is divided by the number continents.

If you do not wish to use this correction, you can uncheck it.

**Days in baseline**

```{r echo=FALSE, fig.cap="", out.width = '30%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/days_in_baseline.png")
```

The default days in baseline is 7. The user can explore the effect of having different days in the baseline. This is only for the visualisation, any changes made for the email alerts have to be made in the configuration page.

**Same weekday baseline**

```{r echo=FALSE, fig.cap="", out.width = '30%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/same_weekday_baseline.png")
```

It is possible that there is a "day of the week effect", where more tweets may be tweeted on a given day of the week (e.g. Monday) than on other days. You can also select to calculate the baseline not on consecutive days, but on the past N days that correspond to the same 24 hour window N days back. This way if N = 7, the baseline is calculated using the “days” from -7, -14, -21, -28, -35, -42, -49 and -56 (if the current "day" is zero).


#### The timeline

The timeline graph is a time series, where you can see the number of tweets for a given topic, geographical unit and study period. Signals are indicated as triangles on the graph, with the alpha and baseline days as specified in the filters. The area under the threshold is indicated in the shaded green colour. Note that the signals are related to the choice of alpha and days in baseline in the filters on the dashboard, rather than what is used for the alert emails. This way you can explore the effect of changing these parameters and adapt the settings for the alert emails if needed.

If you hover over the graph, you obtain extra information on country, date, number of tweets and the number of tweets from the list of known users, the ratio of known users to unknown users, whether the number of tweets was associated with a signal and what the threshold and the alpha was.

```{r echo=FALSE, fig.cap="", out.width = '30%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/hover_trendline.png")
```


#### Map
The map shows a proportional symbol map of the tweets by country and by topic for the study period. The larger the circle, the greater the number of tweets.

The geographical information for the map is based on the choice in the filters: the country/region/subregion and the location type (tweet, user or both).

```{r echo=FALSE, fig.cap="", out.width = '60%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/hover_map.png")
```

When you hover over the map, you can get information on number of tweets and names of the geographical units underlying the circles on the map.

When selecting one country, the symbols show the geographical distribution of tweets at subnational level. When selecting two or more countries or other geographical entity (e.g. regions or continents), the symbols show the geographical distribution of tweets at national level. Note that if a tweet has a geotag at country level (e.g. France), it will not be displayed when selecting only that country since no subnational geotagging is available.

#### Most frequent words, hashtags, URLs, context and entities found in or extracted from tweets

```{r echo=FALSE, fig.cap="", out.width = '50%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/top_words.png")
```

These graphs display the top elements of tweets by topic for the study period for the geographical units chosen, and according to the filter on tweet/retweet.

Note that, differently to other visualisation figures, the most frequent words found in tweets are always related to “tweet location” and are not influenced by the filter option for choice of location (user or tweet location).

One figure displays the most frequent words and hashtags, depending on the selection. A second figure displays the entities and contexts extracted from tweets, depending on the selection. A third figure displays the most frequent URLs as a table allowing users to click directly on the links to access these in the browser.

### The alerts page

The alerts page summarises the signals detected within the specified study period and provides functionality for adding annotations for training machine learning algorithms to classify alerts on user's defined topics. This page is splitted on two sections

#### Find alerts

The user can choose to filter alerts based on the following filters: date, topics, countries/regions. These three filters will define the scope of alerts to search. Also two modifiers are available for searching:

-   Display: Choose the columns to display on the search results

    -    Tweets: Focused on showing relevant information of the alert. Including tweets that are more similar to top words of the alert on the associated period and countries. The following columns are displayed for alerts: Date, Hour, Topic, Region, Category, Tops, Tweets, Top tweets.
    -    Parameters: Focused on showing the parameters of the alerts that were produced. The following columns are displayed for alerts: Date, Hour, Topic Region, Category, Tops, Tweets, % from important user, Threshold Baseline, Bonf. corr., Same weekday baseline Day, rank,  With retweets Location,  Alert FPR (alpha), Outlier FPR (alpha). 

-   Limit: Maximum number of alerts to be displayed.

-   Hide Alerts, Search Alerts: Buttons to hide or display the alerts associated with the filters selected by the user.

-   Add alerts to annotation: The alerts returned by the search are added to the annotations database so the user can annotate and classify them.

#### Alerts annotations

`epitweetr` classify alerts using top words and top tweets using annotations provided by end users through the alert annotation spreadsheet. The user can define different algorithms and its parameters, and `epitweetr` will find the best algorithm to perform the classification. `epitweetr` will randomly split the annotations database assigning 75% of the alerts for training and 25% of alerts for evaluation. All the algorithms provided by the user and/or the three `epitweetr` predefined algorithms will be tested and the best in terms of F1 score will be selected to classify new alerts. From that moment, the category of new alerts will be predicted using the best algorithm. The algorithm is not used to predict the category retrospectively (historical alerts), only prospectively (new alerts).

The user can show, hide, download and upload the alert annotation spreadsheet using the buttons available.

The alert annotation spreadsheet has two sheets that will be used as explained below:

-   Alerts sheet:

    -    Date: Date when the alert was identified
    -    Topic: Topics associated with the alert
    -    Region: Region where the alert was detected
    -    Top words: Top words identified for the alert
    -    Tweets: Number of tweets onserved on the last 24 hours on the alert
    -    Top Tweets: The tweets that contains the most of the topwords associated to the alerts on the alert period and geographical location
    -    Given Category: The category provided by the user. *This is the only column that needs to be updated by the end user to classify alerts*
    -    Epitweetr Category: The category associated by epitweetr using the last version of the alert classification algorithms. Please note that this is only a reference value that include overfitting since the final algoritms is trained with all annotated alerts.

-   Runs sheet: A registry of algorithms to test including performance metrics on the last evaluation.

    -    Ranking: The ranking of the algorithm in terms of f1score
    -    Models: The name of the model, referenced by the Apache Spark class. It has no inherit from the classifier class [*https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/ml/classification/Classifier.html*]
    -    Alerts: The number of alerts used for the training.
    -    Runs: The number of runs to perform to evaluate the algorithm. Each run will make a different random split.
    -    F1Score: The F1score of all classes as provided by Apache Spark [*MulticlassClassificationEvaluator*](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.html)
    -    Accuracy: The Accuracy of all classes as provided by Apache Spark [*MulticlassClassificationEvaluator*](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.html)
    -    Precision by Class: The Precision by Class of all classes as provided by Apache Spark [*MulticlassClassificationEvaluator*](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.html)
    -    Sensitivity by Class: The Sensitivity by Class of all classes as provided by Apache Spark [*MulticlassClassificationEvaluator*](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.html)
    -    FScore by Class: The FScore by class of all classes as provided by Apache Spark [*MulticlassClassificationEvaluator*](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.html)
    -    Last run: The last time and date where this run was evaluated
    -    Active: Whether the run is active (1) or not (0). Only active runs will be tryed to choose the best algorithm and parameters.
    -    Documentation: A link to the algorithm documentation
    -    Custom Parameters: A json object providing values for parameters for the given algorithm. The possibility to perform grid search strategies can be achieved by adding the same algoritm on several lines with different parameters.


```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/alerts.png")
```

### The geotag page
This page supports the user in improving the geolocation algorithm. Epitweetr will allow the users to download the data that was used to build the classifiers for location identificaion. The predefined annotations are based on location vs non location words. Locations are extracted from geonames.org data base. Non location words are obtained from common words on the downloaded models that are not present on geonames. You will be able to add tweets to the annotation datanbase and to manually annotate them until you reach the expected level of performance.

To help you on the evaluation process epitweetr will calculate standard machine learning metrics for evaluating its capacity to identify the right location in texts. These metrics are separately caculated by diferent type of texts: Tweet location, Tweet text, Tweet user description and total.

The following controls are available:

-   Tweets to add: Choose the number of tweets to add to the geolocation spreadsheet
-   Geolocate annotations: Pressing this button will add the selected number of annotations to the epitweetr geolocation spreaddsheet and evaluate the geoloation algorithm on the current annotations
-   Download annotations: Download the current version of annotations
-   Upload & evaluate annotations: Upload a new version of the geolocation annotations to improve annotations performance. The algorithms for location identification will be trained using the provided data. Also performance metrics will be calculated using 75% of the data for training and 25% for test. The final model is trained using all data.

The geolocation spreadsheet has the following columns:

-   Type: This column is a reference for the user to understand the kind of annotation. It can be any of the following.
    -   Text: A text that can contains locations inside it can be a tweet or a user description.
    -   Location: A text containing a location.
    -   Person: A known person e.g. presidents that can be used by epitweetr to detect countries. This data is not used for training, just for over-writing the algorithm behavior.
    -   Demonym: A known word to associate to a country e.g. Chilean. This data is not used for training, just for over-writing the algorithm behavior.
-   Text: The text associated to this annotation.
-   Location in text: The subpart of the text that contains a location. If many locations are present, the same line can be duplicated with different values on this column. If this column is empty epitweetr assumes that the annotation is associated with all the text.
-   Location yes/no: whether the current text contains a location (demonym and people are not considered as location).
-   Associate country code: Whether epitweetr should memorize the annotated text and bypass the machine learning algorithm to always associate the given text to the country (or geonames city) indicated by the provided code.
-   Associate with: Whether epitweetr should memorize the annotated text and bypass the machine learning algorithm to always associate the given text to the country (or geonames city) indicated by the provided name.
-   Source: The source of the annotation, it can be any of: Epitweeter database (provided by epitweetr team), Epitweetr model (obtained automatically by epitweetr) or Tweet (obtained from a tweet).
-   Tweet Id: The tweet id if the text is associated to a tweet.
-   Lang: The language associated to the annotation.
-   Tweet part: The tweet part if associated to a tweet. It can be any of: "text", "user desctiption" or "user location".
-   Epitweetr match: The location found by epitweetr.
-   Epitweetr country match: The country of the location found by epitweer.
-   Epitweetr country code match: The country code of the location found by epitweetr.


```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/geotag_evaluation.png")
```

### The data protection page
In the data protection page, the user can search, anonymise and delete tweets from the epitweetr database to support data deletion requests.

The following search filters are available: Topic, period, country & regions, mentioning (for selecting only mentions of the provided users), from users (for selecting only tweets from the provided users) and both (for getting tweets either mentioning or from the provided user)

The following controls are available:
-   Limit: Limit the perimeter of the search to the first "limit" tweets
-   Search: Perform the search and show the results on the screen.
-   Anonymised search: Perform the search and show anonymised results on the screen. User mentions and authors are replaced by a mention USER.
-   Anonymise: Permanently replace all matching tweets user mentions and authors  by USER.
-   Delete: Permanently delete all the matching tweets.

### The configuration page

In the configuration page, you can change settings of the tool, you can check the status of the various processes/pipelines of the tool and you can add, delete and modify topics and their associated requests, languages for geolocation and the list of the “important users” and email alert subscribers. When changing anything in the “Signal detection” or “General” sections, do not forget to click on the “Update Properties” button at the end of the “General” section. The following sections describe the configuration page in more detail.

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/config.png")
```

**Status**

The status section enables you to quickly assess the latest time point and/or status of the processes for tweet collection (Tweet Search), and geolocation, aggregation and signal detection (Detection pipeline).

```{r echo=FALSE, fig.cap="", out.width = '50%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/status.png")
```

In the status section, you can tell if the embedded database, the search pipeline and the detect pipeline processes are running. If you are running Windows, you can click on “activate” to register these processes as scheduled tasks and run them manually from the Windows task scheduler.


**Requirements & alerts pipeline**

You need to run manually the tasks of dependencies, geonames and languages by clicking in the buttons “Run dependencies”, “Run geonames” and “Run languages” the first time you use `epitweetr`, and then only if you are downloading new versions. 

Geonames and languages relate to the geolocation and language models used by epitweetr. If you would like to update them (this is not something that needs to be done regularly, more on a yearly basis or so), then you can click on “Run”. When running the 'download dependencies' task, epitweetr will request you to stop the 'epitweetr database' in order to perform the update. 

The “Run alerts” button can be used to force the start of this task in case there is any error or issue. You can check their status in the “Requirements & alerts pipeline” table.

The 'Requirements & alerts pipeline' table gives more information about the status of the processes of `epitweetr`. This is useful for troubleshooting any issues arising and for monitoring the progress. It contains the five tasks that are running in the background. GeoNames and languages are tasks that will download and update the local copies of these. This will only be triggered if we add a language or update GeoNames. The start and end dates will generally be much older than those of geotag, aggregate and alerts.

Alerts dates should be more recent if the 'data collection & processing' and 'requirements & alerts' pipelines are active and running. These are scheduled according to the detect span. The status can include running, scheduled, pending, failed or aborted (if it has failed more than three times).

```{r echo=FALSE, fig.cap="", out.width = '70%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/detection_pipeline.png")
```


**Signal detection**

In the signal detection section in the configuration page, you can set the signal false positive rate alpha parameter, which increases (if larger) the the detection interval (more signals are detected), or decreases (if smaller) the detection interval (fewer signals are detected).

```{r echo=FALSE, fig.cap="", out.width = '50%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/signal_detection_config.png")
```

The outlier false positive rate relates to the false positive rate for determining what an outlier is when downweighting previous outliers/signals. The lower the value, the fewer previous outliers will potentially be included. A higher value will potentially include more previous outliers.

The outlier downweight strength determines how much an outlier will be downweighted by. The higher the value the greater the downweighting. For more information please see [Annex I](#annexI).

`epitweetr` calculates a threshold to determine if the current number of tweets for a given 24-hour window exceeds what is expected (see section *"How does it work? General architecture behind epitweetr > Signal detection"*). This threshold is based on a default of the previous 7 days. In the “default days in baseline” field, you can change the number of days.

You can also change the default of using the previous 7 days for calculating a baseline to the previous 7 same days of the week, in order to avoid a "day of the week effect" (it may be that there are always more tweets about this topic on a Monday, for example, which could affect the signal detection).

You can also specify if the signal detection is carried out just with tweet text, or includes retweets/quotes (check the box "Default with retweets/quotes").

The last checkbox "Default with Bonferroni correction", take multiple testing into account, which can result in false positives. If this box is checked then the signal detection alpha parameter is divided by the number of geographical locations in which signal detection is carried out. For example, at country level, the alpha parameter is divided by the total number of countries. At continent level, the alpha parameter is divided by the total number of continents.

When changing anything in the “Signal detection” section, do not forget to click on the “Update Properties” button at the end of the “General” section.


**General**

```{r echo=FALSE, fig.cap="", out.width = '40%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/general.png")
```

-   In **Data directory** you can view the directory that `epitweetr` uses to store the tweet and associated data collected. This is also the directory that the dashboard uses to obtain the datasets for displaying the visualisations. You need to set this folder when you launch `epitweetr` or set the environment variable 'EPI\_HOME'.

-   The **Search span** relates to how long a search plan is carried out. The default is 60 minutes. This value controls the size of the search window of tweets. If you reduce this value you will get tweets sooner, but you may 'waste' requests on topics with very few tweets. If you increase its value you will take more time to get the tweets but you will get more requests for popular tweets increasing the chances of being exhaustive. You can see when you are not able to collect the tweets on the Shiny configuration page if you have more than one active plan for some topics.

-   The **Detect span** relates to how frequently the processes of the detection pipeline (geotagging, aggregation and alert detection) are carried out. The default is 90 minutes. Email alerts are sent at the end of the detect loop. This value is treated as a lower bound, the detect loop could take more time to finish depending on the volume of tweeets and your system specifications. 

-   The **Launch slots** for the detection pipeline processes will be spaced out according to the "Detect span", with the first one starting at midnight. These values can be used in the subscribers file of the configuration page.

-   To avoid storing Twitter credentials in plain files, `epitweetr` uses a system dependent password store functionality, which is stored in the **Password store**. Depending on your system you can choose the mechanism that suits the environment where `epitweetr` is running. For details on each implementation see [*https://CRAN.R-project.org/package=keyring*]

     -   wincred: (Windows only) uses the windows credential manager.

     -   macos: (MAC only) uses the Mac OS keychain services

     -   file: Uses password protected encrypted files

     -   secret service: (Linux only) uses Linux secret service

     -   environment: Uses environment variables (extra setup needed, see [*https://CRAN.R-project.org/package=keyring*])

-   **Spark cores and spark memory:** The memory allocation for `epitweetr` in terms of CPU (Spark cores) and RAM (Spark Memory) is also defined in the “general” section. The default is 6 cores and 6 GB of RAM. This will depend on the CPU and RAM capacity of your machine and it has to be equal or less than that.

-   **Geolocation threshold:** During the geolocation process, sets of words are processed and potential matches to existing locations are determined and given a score. The higher the score, the greater the probability that the geolocation is correct. A threshold is set in `epitweetr`, under which any matches are not considered good enough for geolocation. The scale goes from 1 to 10, and the default is set at 5.

-   **Geonames URL:** The URL used to download the GeoNames database (used for generating locations) is in the general section. Should this URL ever change, you can make the amendment here.

-   **Simplified geonames:** As GeoNames is a very large file, a simplified version of it is used by default, including only existing geographical locations where the population is known. You can uncheck this option if you wish to use the whole GeoNames database.

-   **Stream database results:** Whether to store files on disk instead of performing streaming evaluation of aggregated data frames coming from the epitweetr database. If loading the dashboard takes too long, try selecting/unselecting this.

-   **Maven repository:** This is the URL of the maven repository that will be used to download the JAR dependencies for the detect loop, mainly Spark and Lucene.

-   **Winutils URL:** This is the URL that will be used to download winutils.exe. This is a Windows binary necessary for running Spark locally on Windows. If you do not want to use this version you can produce it yourself by downloading Hadoop 2.8.4 or higher and compiling it on a Windows machine.

-   **Region disclaimer:** If you would like to add a disclaimer to the map you are using. This disclaimer is added to the image export of the dashboard map, and also the PDF export of the dashboard.

**Twitter authentication**

You have two options for authentication to collect tweets, using a Twitter account (utilising the rtweet package) and using a Twitter developer application. You can select which option you will use in the **Twitter authentication** section. User can choose the Twitter apis to use.

The admin email will receive epitweetr notifications of regular sanity checks of the application indicating, for example, if any of the tasks/pipelines has failed.

See section *“How does it work? General architecture behind epitweetr > Collection of tweets > Twitter authentication”* for more details on how to do the Twitter authentication.

```{r echo=FALSE, fig.cap="", out.width = '30%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/twitter_authentication.png")
```

**Email authentication (SMTP)**

In this section, you need to specify the email authentication (SMTP) details for the email that will send the alerts.

If **Unsafe certificates** is checked, then `epitweetr` will use your SMTP server even if the server sends an invalid certificate.

When changing anything in the “general” section, do not forget to click on the “Update Properties” button.

**Topics**

Topics are what determines what tweets `epitweetr` collects. This is done via an Excel spreadsheet that contains the topics and the associated requests that `epitweetr` uses to query Twitter API with.

A query consists of keywords and operators that are used to match on tweet attributes. See section *"How does it work? General architecture behind epitweetr > Collection of tweets > Topics of tweets to collect and queries"* for more details about queries.

`epitweetr` comes with a default list of topics as used by the ECDC Epidemic Intelligence team at the date of package generation (*1^st^ of September, 2020*). You can download this list of topics and upload your own in the “Available Topics” section in the configuration page. See section *"How does it work? General architecture behind epitweetr > Collection of tweets > Topics of tweets to collect and queries"* for more details on how to structure the topics list.

In the topics section on the configuration page, you can view the topic, the associated query, the query length and how many active search plans are associated with the query. If more than one search plan is active, this means that `epitweetr` did not manage to collect all possible tweets in the last session. Additionally, you can see the progress and the number of requests from the last search plan.

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/topics.png")
```

**Languages**

In the languages section, you can determine which language models are used to identify text during the geolocation process. The default languages are French, English, Portuguese and Spanish. You can download and upload the language models in the “Available Languages” section and add and delete languages used by `epitweetr` in the “Active Languages” section. Please consider the computational cost of adding too many languages, depending on the capacity of your machine.

**All default languages (English, French, Spanish and Portuguese) must be downloaded before adding new languages or deleting any of the default languages.**

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/languages.png")
```

### The troubleshoot page
The troubleshoot page has a list of automatic check and hint for using `epitweetr` with all its functionalities. Click on "Run diagnostics" to see the list of checks, whether it passed the check ("true") or not ("false"), and hints in case it did not pass the check. More detailed information can be found in [Annex II](#annexII) of this user documentation.

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/troubleshoot.png")
```

### Downloading outputs from the interactive user interface (Shiny app)

Each visualisation on the Shiny app dashboard can be downloaded as an image, using the "image button". A png is a portable network graphic file and is a versatile file format for images that do not need to be of a very high resolution (e.g. professional print graphics).

Note that the png format is not supported in the Internet Explorer browser (but you can download a svg file instead).

You can also download the data of each visualisation by clicking on the data button. This will give you a csv file containing the underlying data that you can use for further analysis or to create your own graphs.

Alternatively, you can use the PDF or the Md button at the bottom of the filters to download a PDF or an HTML file of the dashboard Note that for this you will need to have MiKTeX or TinyTeX installed.

## Annex I: Downweighting the previous signals {#annexI}

## Introduction

In this annex we propose a downweighting approach built as part of the `eears` algorithm used in the `epitweetr` package and which was described above.

Let the $\mathbf{y}$ denote the vector of historic values which is of length $n$. Part of the computation of the prediction interval at time 0 is the computation of the mean and standard deviation of these historic values, i.e.
$$
\overline{y}_0 = \frac{1}{n}\sum_{t=-n}^{-1} y_{t} \quad\text{and}\quad s_0^2 = \frac{1}{n-1} \sum_{t=-n}^{-1} (y_{t} - \overline{y}_0  )^2
$$
The upper limit of the one-sided $(1-\alpha)\times 100\%$ prediction interval for the observation $y_0$ under an $y_t \stackrel{\text{iid}}{\sim} N(\mu, \sigma^2), t=-n, \ldots, 0$ model is then computed as 
$$
U_0 = \overline{y}_0 + t_{1-\alpha}(n-1) \times s_0 \times \sqrt{1+\frac{1}{n}},
$$
where $t_{1−\alpha}(n − 1)$ denotes the $1 − \alpha$ quantile of the t-distribution with $n − 1$ degrees of freedom. This computation of the threshold corresponds to a statistical sound computation of the threshold [@allevius_prospective_2017].

A desired extension of the above algorithm is the handling of previous signals in the historic values. This problem was already addressed in the quasi-Poisson framework of @farrington_statistical_1996 by first performing a GLM fit and then re-fit the GLM with weights  based on the Anscombe residuals. We follow the same general idea, but adapt it to the Gaussian response used in the EARS algorithm and corresponding residuals from the linear model.

### EARS as a Linear Model

We first observe that the above estimation of $\mu$ and $\sigma^2$ through $\overline{y}_0$ and $s_0^2$ at time 0 can be embedded within a linear regression model, i.e. for $i=1, \ldots, n$ we model
$$
y_i = \mu + \epsilon_i, \quad\text{where}\quad \epsilon_t \stackrel{\text{iid}}{\sim} N(0, \sigma^2).
$$
Note that we, for compatibility with the standard exposition in linear model theory, have indexed the $y$ values s.t. $y_{-n}$ corresponds to $y_1$ and $y_{-1}$ corresponds to $y_n$. In matrix terms let $\mathbf{y} = (y_{1},\ldots,y_n)'$ and for the intercept-only model the design matrix is $\mathbf{X} = (1,\ldots,1)'$, which has rank $k=1$. Thus from standard OLS theory:
$$
\begin{align*}
\hat{\mu} &=  (\mathbf{X}'\mathbf{X})^{-1} \mathbf{X}' \mathbf{y} = \frac{1}{n}\sum_{i=1}^n y_i,
\end{align*}
$$
which corresponds to $\overline{y}_0$. Furthermore, let the raw residuals be defined as $e_i = y_i - \hat{\mu}$ for $i=1,\ldots, n$ and denote by $\mathbf{e}=(e_1,\ldots,e_n)'$ the corresponding vector of residuals. Then
$$
\mathbf{e} = \mathbf{y} - \hat{\mathbf{y}} = \mathbf{y} - \mathbf{P} \mathbf{y} = (\mathbf{I}-\mathbf{P}) \mathbf{y}
$$
where $\mathbf{P} = \mathbf{X} (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'$ is the so called hat-matrix known from linear modelling. With this notation we can write up the estimate for $\sigma^2$ as in @chatterjee_sensitivity_1988:

$$
\hat{\sigma}^2 = \frac{\mathbf{e}' \mathbf{e}}{n-k} = \frac{\mathbf{y}'(\mathbf{I}-\mathbf{P})\mathbf{y}}{n-k} = \frac{1}{n-1} \sum_{t=-7}^1 (y_t - \hat{\mu})^2,
$$
which corresponds to the above used expression for $s_0^2$.

### Downweighting

We now compute the so called **externally Studentized residuals** [@chatterjee_sensitivity_1988]
$$
r_i^* = \frac{e_i}{\hat{\sigma}_{(i)}\sqrt{1-p_{ii}}}, \quad i=1, \ldots, n,
$$
where $p_{ii}$ is i'th diagonal element of the hat-matrix $\mathbf{P}$ from the corresponding linear model used above. Furthermore,
$$
\hat{\sigma}_{(i)}^2 = \frac{\mathbf{y}_{(i)}' (\mathbf{I}-\mathbf{P}_{(i)}) \mathbf{y}_{(i)}}{n-k-1}
$$
is the variance estimate obtained from a linear regression, where the i'th observation is removed. Linear modelling theory [@chatterjee_sensitivity_1988] now states that 
$$
r_i^* \stackrel{\text{identical}}{\sim} t(n-k-1).
$$
Note that the residuals are only identically distributed, because they are not independent (see Section 4.2.1. of @chatterjee_sensitivity_1988 for details). However, the above distributional form allows us to assess for each historic value, if it can be considered as an outlier. 
For this purpose define $r_{\text{threshold}}$ as the $1-\alpha_{\text{outlier}}$ quantile of the t-distribution with $n-k-1$ degrees of freedom.
A historic value is an outlier (for which one possible explanation is that it originates from a true increase in tweets, e.g. an outbreak situation), if
$r_i^* > r_{\text{threshold}}$. We shall use this to formulate a weighting schemes for the historic values:

Downweight-Outliers:
$$
\begin{align}
w^{(\text{dw})}_i &= \left\{
\begin{array}{ll}
1 & \text{if } r_i^* < r_{\text{threshold}}\\
\left(\frac{r_{\text{threshold}}}{(r_i^*)}\right)^k & \text{otherwise}
\end{array}
\right. \\
&= \min\left\{1,\left(\frac{r_{\text{threshold}}}{r_i^*}\right)^k\right\},
\end{align}
$$
where the decay parameter $k>0$ is a known quantity. In the original @farrington_statistical_1996 algorithm, $k=2$ was used. Furthermore, a threshold value of 1 was used. In the later @noufaily_improved_2013 paper, however, a threshold value of 2.58 was recommended. Note: both values are for the standardized Anscombe residuals, which follow a standard normal distribution. If we take corresponding quantiles for the t-distribution with 6 degrees of freedom the values would be `r sprintf("%.2f",qt(pnorm(1), df=6))` and `r sprintf("%.2f",qt(pnorm(2.58), df=6))`.
Note also that the term $(r_{\text{threshold}}/r_i^*)^k$ is a slight adaptation of @farrington_statistical_1996, which instead uses $1/(r_i^*)^2$. The advantage of our proposal is that it ensures a smooth handling of values around the threshold if the threshold is not 1. It might be worth considering a higher power than 2 to ensure an even larger down-weighting for gross outliers. The current default value for the decay parameter in `epitweetr` is 4.

Finally, as in @farrington_statistical_1996, we normalise the weights such that they yield a sum of $n$ by
$$
w_i^* = n \times \frac{w_i}{\sum_{i=1}^n w_i}
$$
and then re-fit the linear model with these weights. For this purpose define the weight matrix as $\mathbf{W} = \operatorname{diag}(w_1^*,\ldots,w_n^*)$. We can use a subsequent weighted least squares approach to find
$$
\begin{align*}
\hat{\mu}_W &=  (\mathbf{X}' \mathbf{W} \mathbf{X})^{-1} \mathbf{X}' \mathbf{W} \mathbf{y} = \frac{1}{n}\sum_{i=1}^n w_i^* y_i,
\end{align*}
$$
where the 2nd equal sign is because $(\mathbf{X}' \mathbf{W} \mathbf{X})=\sum_{i=1}^n w_i=n$
and $\mathbf{X}' \mathbf{W} \mathbf{y} = \sum_{i=1}^n w_i^* y_i$.
Furthermore,
$$
s_W^2 = 
\frac{\mathbf{y}'(\mathbf{I}-\mathbf{P}_W)\mathbf{y}}{n-k} =
\frac{\sum_{i=1}^n w_i^*(y_i - \mu_W)^2}{n-1},
$$
where $P_W = \mathbf{X} (\mathbf{X}'\mathbf{W} \mathbf{X})^{-1}\mathbf{X}\mathbf{W}$ is the hat-matrix of the weighted least squares.

The downweighted procedure thus operates with $\mu_{W}$ and $s_W^2$ instead of $\overline{y}_0$ and $s_0^2$, respectively, when computing the upper limit $U_0$ using the above mentioned formula.

### Example of the downweighting approach using Ebola data

Figure 5 below shows the upper limit of the signal detection threshold for `epitweetr` Ebola data, both the original (in red) with no downweighting and the downweighted upper threshold (in blue) after taking previous signals in the historic values into account. Note that the downweighted upper threshold detects three additional signals, compared to the original threshold. 

<center>Fig 5: Upper limit with and without downweighting for the epitweetr Ebola data</center>

```{r echo=FALSE, fig.cap="", out.width = '100%'}
knitr::include_graphics("https://github.com/EU-ECDC/epitweetr/raw/master/img/ebola.png")
```

## Annex II: Troubleshooting and tips {#annexII}

This annex contains some tips and common solutions to errors or issues that `epitweetr` users may encounter, including an explanation on the checks included in the troubleshoot page.

### The troubleshoot page
After running the diagnostics in the troubleshoot page, you can see the checks and status on the following aspects:

-   **scheduler**: R package `taskscheduleR` is installed. Only applicable for Windows machines
-   **twitter_auth**: Twitter token has been created after authenticating either with Twitter account or Twitter developer application
-   **search_running**: search task is running
-   **database_running**: embedded database is running
-   **tweets**: tweets have been collected
-   **os64**: R is 64bits
-   **java**: Java is installed and accessible to epitweetr
-   **java64**: Java 64bits is installed and accessible to epitweetr
-   **java_version**: the Java version installed is compatible with `epitweetr`
-   **winmsvc**: Microsoft Visual C++ 2010 SP1 Redistributable Package is installed. Only applicable for Windows machines
-   **detect_activation**: detect loop has been activated
-   **detection_running**: detection task is running
-   **winutils**: winutils is installed. If false, it can be downloaded by running the update dependencies task. Only applicable for Windows machines
-   **java_deps**: java dependencies are installed 
-   **move_from_temp**: `epitweetr` can atomically move files from temporary folder to data directory
-   **geonames**: Geonames.org database is downloaded and indexed
-   **languages**: languages vectors are downloaded and indexed
-   **geotag**: geotag task has successfully run
-   **aggregate**: aggregate task has successfully run
-   **alerts**: alerts have been created
-   **pandoc**: Pandoc is installed and accessible to epitweetr. Necessary for PDF creation.
-   **tex**: a tex distribution is installed and accessible to epitweetr. Necessary for PDF creation.

### Management of 'epitweetr database' and 'Data collection and alerts' pipelines (Windows)
After activating the database, search and detection pipeline from the configuration page of `epitweetr` (Windows), three tasks will be created in the task scheduler and three terminal windows will be prompted. Please note that if the computer is logged/turned off or the terminal windows are closed, the search and detection pipeline will stop.

If you activate these tasks from the configuration page of `epitweetr` again, the system will overwrite the tasks created in the task scheduler. Instead, after the first successful activation of these tasks from `epitweetr`, you can easily manage these from the task scheduler. You can stop these tasks by ending and disabling the tasks in the task scheduler, and you can restart these tasks by enabling and running these in the task scheduler.

In the task scheduler, you can establish that the search and detection tasks “run whether the user is logged on or not” to avoid that these tasks stop when you log off or restart the computer. 

### Management of 'epitweetr database' and 'Data collection and alerts' pipelines (Linux and Mac)
Since the search and detection pipeline in Linux or Mac have to be run manually, if the computer is logged/turned off or the terminal windows are closed, the search and detection pipeline will stop. Please remember to follow the steps in the section *Setting up tweet collection and the alert detection loop* to run these tasks again.

### Running 'epitweetr database', 'Data collection and processing' and 'Requirements and alerts' pipelines

#### "Cannot execute task \#####: the task is already running"
The detect and the search loop create two files containing their process IDs located in the `epitwitter` data folder: search.PID and detect.PID. This error arises if `epitweetr` finds another R process currently running with the same ID. In order to fix this error you should first verify if the search / detect loop are not already running in another R session. If this is the case, you should not try to start the task since `epitweetr` only supports one instance of the same task running in the same data folder. If the running process is not associated with the task then you can manually delete the PID file and try to start it again.

#### "Failed while processing alerts"
The error "failed while processing alerts Error in do_next_alerts(tasks): Cannot determine the last aggregated period for alert detection. Please check that series have been aggregated" appears when there are no aggregated series to calculate the alerts. This can happen in the following cases:
* No tweets have been collected in the past days so there were no tweets available to produce the aggregated series
* Geonames and/or languages are not downloaded so the geolocation cannot be extracted from the tweets and, consequently, tweets cannot be aggregated.

If it is the first installation, you should wait until geonames and/or languages tasks are completed and collected tweets are geotagged and aggregated. Depending on the machine, these steps may take some hours.

You should also checked that tweets are being collected.

### Change the user of the Twitter authentication when using a Twitter account
1. End and disable the search loop/task in the task scheduler (Windows) or close the R/terminal window with the search loop/task (Linux and Mac)

2. Search for a file called “.rtweet_token” in hidden files. It is usually saved in the Documents folder.

3. Delete that file.

4. Click on “Update properties” in the configuration page of `epitweetr`.

5. Enable and run the search loop/task in the task scheduler (Windows) or run the command in a new R/terminal window with the search loop/task (Linux and Mac). More details are available in the section *“Setting up tweet collection and the alert detection loop”*

### Downloading GeoNames and/or languages

#### Languages to be added or deleted 
All default languages (English, French, Spanish and Portuguese) must be downloaded before adding new languages or deleting any of the default languages.

#### "The specified size exceeds the maximum representable size. Error: Could not create the Java Virtual Machine"

If this error appears when running GeoNames, it means that the machine has Java 32bits. You need to install Java 64bits. And make it accessible to epitwitter either by setting "JAVA\_HOME" environment variable or by setting the right java binary on the system PATH. 

### The "Launch slots" in the configuration page show NAs instead of the time slots

If it is the first time that you install and launch `epitweetr`, the geotag task of the detection pipeline has to be run at least once in order to see time slots in the “Launch slots” in the configuration page.

### Downloading PDF of the dashboard

#### "Error in: LaTeX failed to compile C:&#92;Users&#92;name~1&#92;...&#92;file######.tex."

This error appears in Windows when clicking on “PDF” in the dashboard and no PDF is saved. The reason is that the path to TEMP and TMP environment variables of the user are too long, Windows shortens the path and epitweetr cannot find this new path. Please follow the next steps to fix this:

1. Open the “environment variable for your account”

2. Change the path for TEMP and TMP to a shorter path (e.g. “C:&#92;Temp”). The same path should be used for both environment variables.

3. Log off and log on

4. You can now download and save the PDF from the dashboard

#### "Error: pandoc document conversion failed with error 6"

1. Downloading this script (https://raw.githubusercontent.com/jgm/pandoc/master/macos/uninstall-pandoc.pl)

2. Uninstall pandoc (https://pandoc.org/installing.html) by running perl uninstall-pandoc.pl

### Different totals in dashboard outputs
When counting the total tweets in the dashboard of the Shiny app or in the downloadable data, you might get differences in the total numbers of tweets between the three outputs. This might be due to the following reasons:

1. World (all) versus World (geolocated)

    - The default option for the regions in World (all), this means that also non-geolocated tweets are included in the trendline, but only geolocated tweets can be visualized in the maps and the most frequent words figure, therefore the overall total of tweets can differ between these outputs wen selecting World (all) or the empty default.

2. Country specific analysis

    - If you select only one country in the filters, the trendline will show all tweets for this country, but the map will show the tweets on a subnational level in the map. It could be that some tweets might have been geolocated to a certain country, but without further subnational data. These tweets will then be visible in the trendline total, but not in the subnational bubbles in the map.

3. Most frequent words

    - In contrast to the other outputs in the dashboard, the most frequent words figure is always based on tweet location regardless of the filter (due to memory capacity). Therefore, if user location or both locations are selected in the location filter, this figure might have a different total then the other two outputs.
    
### Receiving only real-time alerts
This relates to users who have selected topics and/or regions for receiving related alerts in real-time or have selected topics and/or regions for receiving related alerts on a scheduled time span. If, in these cases, you only receive real-time alerts with all topics and regions, it may be that no time slots have been included in the subscribers file from the configuration page. These time slots are used for the scheduled alerts and if no slots are included in the file, alerts from all topics and regions are sent as real-time alerts.

### Not receiving email alerts
If you do not receive email alerts and you see an error in `epitweetr` referring to denied login, it means that `epitweetr` could not login to the email account provided in the configuration page. Some of the reasons for that are:

-   The server or port included in the configuration page are incorrect
-   The attempt of `epitweetr` to login to the email account is being blocked by the server. This can happen with some organisational email accounts. In that case, please contact the IT department of your organisation
-   If using Gmail account, you need to allow less secure apps in the settings of your account

## References

<div id="refs"></div>
  [*https://www.java.com/download/*]: https://www.java.com/download/
  [*https://www.microsoft.com/en-us/download/details.aspx?id=14632*]: https://www.microsoft.com/en-us/download/details.aspx?id=14632
  [*https://pandoc.org/installing.html*]: https://pandoc.org/installing.html
  [*https://yihui.org/tinytex/*]: https://yihui.org/tinytex/
  [*https://miktex.org/download*]: https://miktex.org/download
  [*https://www.openblas.net/*]: https://www.openblas.net/
  [*https://github.com/fommil/netlib-Java*]: https://github.com/fommil/netlib-Java
  [*https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onemkl.html*]: https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onemkl.html
  [*https://git-scm.com/downloads*]: https://git-scm.com/downloads
  [*https://www.scala-sbt.org/download.html*]: https://www.scala-sbt.org/download.html
  [*https://cran.r-project.org/bin/windows/Rtools/*]: https://cran.r-project.org/bin/windows/Rtools/
  [*https://github.com/steveloughran/winutils/raw/master/hadoop-3.0.0/bin/winutils.exe*]: https://github.com/steveloughran/winutils/raw/master/hadoop-3.0.0/bin/winutils.exe
  [*https://developer.twitter.com/en/apply-for-access*]: https://developer.twitter.com/en/apply-for-access
  [*https://CRAN.R-project.org/package=keyring*]: https://CRAN.R-project.org/package=keyring
  [*Monitoring Count Time Series in R: Aberration Detection in Public Health Surveillance*]: https://www.jstatsoft.org/article/view/v070i10
