通用时间序列在线异常值检测的简单算法
我正在处理大量的时间序列。 这些时间序列基本上是每 10 分钟进行一次的网络测量，其中一些是周期性的（即带宽），而另一些则不是周期性的（即路由流量）。
我想要一个简单的算法来进行在线“异常值检测”。基本上，我想将每个时间序列的整个历史数据保留在内存中（或磁盘上），并且我想检测实时场景中的任何异常值（每次捕获新样本时）。 实现这些结果的最佳方法是什么？
我目前正在使用移动平均线来消除一些噪音，但接下来该怎么办？像标准差、疯狂之类的简单的东西对整个数据集来说效果不佳（我不能假设时间序列是静止的），我想要一些更“准确”的东西，最好是一个黑匣子，比如：
double outlier_detection(double* vector, double value);
其中 vector 是包含历史数据的 double 数组，返回值是新样本 "value" 的异常分数。
I am working with a large amount of time series.
These time series are basically network measurements coming every 10 minutes, and some of them are periodic (i.e. the bandwidth), while some other aren't (i.e. the amount of routing traffic).
I would like a simple algorithm for doing an online "outlier detection". Basically, I want to keep in memory (or on disk) the whole historical data for each time series, and I want to detect any outlier in a live scenario (each time a new sample is captured).
What is the best way to achieve these results?
I'm currently using a moving average in order to remove some noise, but then what next? Simple things like standard deviation, mad, ... against the whole data set doesn't work well (I can't assume the time series are stationary), and I would like something more "accurate", ideally a black box like:
double outlier_detection(double* vector, double value);
where vector is the array of double containing the historical data, and the return value is the anomaly score for the new sample "value" .
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
这是一个庞大而复杂的主题，答案取决于 (a) 您想在这方面投入多少精力，以及 (b) 您希望异常值检测有多有效。一种可能的方法是自适应过滤，通常用于降噪耳机等应用。有一个不断适应输入信号的滤波器，有效地将其滤波器系数与信号源的假设短期模型相匹配，从而减少均方误差输出。然后，这将为您提供低电平输出信号（残余误差）除非当您获得异常值时，这将导致易于检测的尖峰（阈值）。阅读自适应过滤、LMS 过滤器 等，如果您认真对待这种技术。
This is a big and complex subject, and the answer will depend on (a) how much effort you want to invest in this and (b) how effective you want your outlier detection to be. One possible approach is adaptive filtering, which is typically used for applications like noise cancelling headphones, etc. You have a filter which constantly adapts to the input signal, effectively matching its filter coefficients to a hypothetical short term model of the signal source, thereby reducing mean square error output. This then gives you a low level output signal (the residual error) except for when you get an outlier, which will result in a spike, which will be easy to detect (threshold). Read up on adaptive filtering, LMS filters, etc, if you're serious about this kind of technique.
我建议采用下面的方案，该方案应该在一天左右的时间内即可实现：
训练
计算“异常值”：
对于您想知道其“异常值”：
这将是您的异常值：100% 是极端异常值。
PS. In calculating the Mahalanobis distance, use the correlation matrix, not the covariance matrix. This is more robust if the sample measurements vary in unit and number.
I suggest the scheme below, which should be implementable in a day or so:
Training
Calculating "outlierness":
For the single sample of which you want to know its "outlierness":
That will be your outlier score: 100% is an extreme outlier.
PS. In calculating the Mahalanobis distance, use the correlation matrix, not the covariance matrix. This is more robust if the sample measurements vary in unit and number.