Basic anomaly detection finds data points that are unusual relative to an expected distribution and there are good methods for defining expected distributions and quantifying deviation from them. However, using anomaly detectors in operational systems has proven to be a challenge. Naive flagging of anomalous points can lead to an overwhelming number of false positives and result in detection thresholds being lowered to the point of effectively turning off the anomaly detector.

I will discuss two improvements for anomaly detection systems. The first is to observe that we are often not interested in individually anomalous data points. Instead, there are underlying phenomena we want to identify and those phenomena affect large groups of data points. Anomaly detection over sets of data can produce more significant results. Second, anomaly detection should be in a closed loop process that generates additional labeled data, updated models, and ever fewer detections, thus reducing load on the user. We will illustrate these ideas with some scientific and commercial use cases.