Evaluating Anomaly Detection Algorithms

Anomaly detection mechanisms are crucial components of machine learning systems that are deployed in safety critical applications, where failures might inflict physical, psychological or economic damage to some party. In such settings, it is important to identify observations or events that diverge so much from the data that has been used to determine the parameters of the machine learning model that the model can not be expected to generalize to the new input.

As anomaly detection methods are usually taken as unsupervised learning problems, estimating their performance under realistic settings turns our to be rather difficult; current evaluation protocols might underestimate the probability of failure and do sometimes not account for randomness in algorithms. Deep models dealing with high dimensional data suffer from this problem in particular. The goal of this project is to develop methods that are able to reliably evaluate unsupervised anomaly detection algorithms.