Houston, we have a problem — Anomaly Detection Methods

Screenshot from Oddity demo

The previous article in my Houston, we have a problem anomaly detection series was a light introduction to the realm of time series anomaly detection in general. In this article, we will elaborate on some of the common methodologies and algorithms used to actually solve time series anomaly detection problems. Finally, we will see how we can use Oddity — a small tool I developed — to conduct easy anomaly detection on our own time-series data.

The following (and final) post in this series will go in-depth on the math behind Oddity and how it works.

Anomaly Detection Methodologies

There is quite a large landscape of anomaly detection methods, seeing as it is such a large and important domain. The following are arguably some of the most common models/techniques:

State Space Models

ARIMA, SARIMAX, SARIMA, Holt-Winters, etc.

These models are statistical models that are generally used for time series forecasting, however, they can be utilized for anomaly detection as well. By checking points in a time series against a model’s predictions (for that time series), any significant difference from the prediction could indicate that point as being anomalous.

Deep Learning (Supervised)

RNNs (Recurrent Neural Networks), etc.

Recurrent neural networks are a type of neural network tailored for time series data and thus can be applied to the topic of anomaly detection. Generally, RNNs will attempt to learn sequential dependencies within the series and use that to make predictions for future time steps. If there is a large deviation from the prediction, that point may be anomalous. This is typically one of the more popular methodologies.

Deep Learning (Unsupervised)

AEs, VAEs, etc.

Different forms of autoencoders have been used for anomaly detection. Interestingly, CNNs (Convolutional Neural Networks), which are traditionally used for spatial applications, have also been used for unsupervised time series anomaly detection with a good level of success (DeepAnT).

Decomposition

STL Decomposition, etc.

Decomposition is a statistical technique wherein a set of time series data is broken into its constituents: trend, seasonality, and residual. The trend and seasonality of the given time series are identified and then subtracted from the series. This leaves nothing but a “residual”, which is anything that doesn’t completely follow the trend and seasonality perfectly. More often than not, time-series data isn’t perfect, and will commonly have some level of residual. If the residual for a point is remarkably high, however, that might indicate a serious deviation from typical trend and seasonal behavior, thus making it anomalous. For more information see ritvikmath.

Other Unsupervised Approaches

Isolation Forest, Clustering, etc.

Other methods of somewhat simple unsupervised machine learning algorithms are also popular choices for anomaly detection. Isolation forests and DBSCAN for example, work great in identifying outliers in data. It is important to note that clustering alone is not usually the best suited for specifically, time series anomaly detection.

Oddity

Oddity is a small open-source tool I developed that uses additive Gaussian processes and time series decomposition to help identify anomalies in time series data. We will first cover some basic information, and how to get started.

Installing Oddity

First and foremost, the main Oddity engine is written in Rust. Rust has incredible speed, which by extension makes Oddity capable of doing very quick computations on even a few thousand-time steps in seconds. We will therefore need to have the Rust compiler installed:

Install Rust

Next, we will need the setuptools-rust library, which will allow Oddity to compile on our system. It can be installed via pip with:

pip install setuptools-rust

Finally, we can install Oddity itself, also via pip:

pip install oddity

Test Data

Once we have Oddity installed, we will need to have some data. We will use the following time series data as our example:

Credit

This is Google Trends data for the search term “ice cream” from 2004. It is a perfect candidate for our anomaly detection experiment because it contains both seasonal and trend components with some contextual anomalies. You can download this data yourself from this link.

Using Oddity

Now that we have some data, we will proceed with a complete example of using Oddity for anomaly detection.

Oddity can be imported as:

import odditylib as od

Importing helper libraries:

import numpy as np
import pandas as pd

Reading data:

# Reading our CSV data
df = pd.read_csv('ICECREAM-DATA-PATH')
#  Selecting only the raw values per time step
timeseries = df['ice cream: (United States)']
# Oddity requires an extra dimension per time step
timeseries = timeseries.values.reshape(-1, 1)

Fitting an Oddity detector:

detector = od.Oddity()  # Creating a default Oddity detector
detector.fit(timeseries)  # Fitting the detector on our data

As a shallow description, an Oddity detector is the sum of Gaussian process regression fittings on the trend and seasonal components of a given time series. We can access the fit, mu, and the covariance matrix, cov:

mu, cov = detector.mu, detector.cov

The fit, mu, is used to generally model the behavior of the time series. Plotting mu alone will look like this:

We can already see that Oddity has been able to capture both the seasonal and trend component automatically. The Oddity detector parameters can be fully customized if needed. To get some visualization, we will plot the Oddity fit against the original time series data.

Already, there are obvious deviations from expected normal behavior. However, we need to add some leeway in the form of intervals.

In this case, we will calculate the intervals using the error of the fit. We will consider an anomaly to be anything 3 standard deviations away from the error mean. This can be implemented as:

error = timeseries - detector.mu  # Calulating error
mean, std = error.mean(), error.std()
interval = mean + (3.0 * std)  # Calculating the interval
# Creating the bounds
upper_bound = detector.mu + interval
lower_bound = detector.mu - interval

Naturally, this can be changed to whatever sensitivities or requirements are needed for a specific use case.

The intervals can be plotted with:

import matplotlib.pyplot as plt

plt.figure(figsize=(16, 10)) # For aesthetic purposes
plt.plot(timeseries, label='Original')
plt.plot(detector.mu, label='Oddity fit')
error = timeseries - detector.mu
mean, std = error.mean(), error.std()
interval = mean + (3.0 * std)
upper_bound = timeseries + interval
lower_bound = timeseries - interval
plt.fill_between(
range(len(timeseries)),
upper_bound,
lower_bound,
alpha=0.2
)
plt.legend()
plt.show()

Finally, we can check which data points are out of bounds — those are the anomalies.

anomalies = list(
filter(lambda value:
value[1] > upper_bound[[value[0]]] or
value[1] < lower_bound[value[0]], enumerate(timeseries))
)

Finally, we can add the anomalies to the plot with:

for anomaly in anomalies:
date, value = anomaly

plt.plot(date, value, 'ro')

By the looks of it, we have successfully managed to identify the major anomalies within this time series data.

Conclusion

Oddity is a fully customizable tool for detecting anomalies in time series data using additive Gaussian processes. Though it is still pretty barebone, it can definitely be of use in many circumstances.

Additional Ressources

The GitHub repository for Oddity:

Lleyton-Ariton/oddity

A working demo using Oddity and streamlit:

Lleyton-Ariton/oddity-demo

The pure Rust engine:

Lleyton-Ariton/oddity-engine

Further Reading


Houston, we have a problem — Anomaly Detection Methods was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.