C 中的滚动中值算法
我目前正在研究一种用 C 语言实现滚动中值滤波器(类似于滚动均值滤波器)的算法。从我对文献的搜索来看,似乎有两种相当有效的方法可以实现。第一种是对初始值窗口进行排序,然后执行二分搜索以插入新值并在每次迭代时删除现有值。
第二个(来自 Hardle 和 Steiger,1995,JRSS-C,算法 296)构建了一个双端堆结构,一端是 maxheap,另一端是 minheap,中间是中位数。这产生了一种线性时间算法,而不是 O(n log n) 的算法。
这是我的问题:实现前者是可行的,但我需要在数百万个时间序列上运行它,因此效率非常重要。事实证明后者非常难以实施。我在 R 的 stats 包代码的 Trunmed.c 文件中找到了代码,但它相当难以解读。
有谁知道线性时间滚动中值算法的编写良好的 C 实现吗?
编辑:链接到 Trunmed.c 代码 http://google.com/codesearch/p?hl=en&sa=N&cd=1&ct=rc#mYw3h_Lb_e0/R-2.2 .0/src/library/stats/src/Trunmed.c
I am currently working on an algorithm to implement a rolling median filter (analogous to a rolling mean filter) in C. From my search of the literature, there appear to be two reasonably efficient ways to do it. The first is to sort the initial window of values, then perform a binary search to insert the new value and remove the existing one at each iteration.
The second (from Hardle and Steiger, 1995, JRSS-C, Algorithm 296) builds a double-ended heap structure, with a maxheap on one end, a minheap on the other, and the median in the middle. This yields a linear-time algorithm instead of one that is O(n log n).
Here is my problem: implementing the former is doable, but I need to run this on millions of time series, so efficiency matters a lot. The latter is proving very difficult to implement. I found code in the Trunmed.c file of the code for the stats package of R, but it is rather indecipherable.
Does anyone know of a well-written C implementation for the linear time rolling median algorithm?
Edit: Link to Trunmed.c code http://google.com/codesearch/p?hl=en&sa=N&cd=1&ct=rc#mYw3h_Lb_e0/R-2.2.0/src/library/stats/src/Trunmed.c
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(13)
我已经看过 R 的 src/library/stats/src/Trunmed.c 几次了,因为我想在独立的 C++ 类/C 子例程中也有类似的东西。请注意,这实际上是两个实现合二为一,请参阅 src/library/stats/man/runmed.Rd(帮助文件的源代码),其中表示
It would be happy to see this re-used in更加独立的时尚。你是自愿的吗?我可以帮助解决一些 R 位问题。
编辑 1:除了上面旧版本 Trunmed.c 的链接之外,这里还有
Srunmed.c
(对于 Stuetzle 版本)Trunmed.c
(对于 Turlach 版本)runmed.R
R 函数对于调用这些编辑 2 的 :Ryan Tibshirani 在 快速中值分箱,这可能是窗口方法的合适起点。
I have looked at R's
src/library/stats/src/Trunmed.c
a few times as I wanted something similar too in a standalone C++ class / C subroutine. Note that this are actually two implementations in one, seesrc/library/stats/man/runmed.Rd
(the source of the help file) which saysIt would be nice to see this re-used in a more standalone fashion. Are you volunteering? I can help with some of the R bits.
Edit 1: Besides the link to the older version of Trunmed.c above, here are current SVN copies of
Srunmed.c
(for the Stuetzle version)Trunmed.c
(for the Turlach version)runmed.R
for the R function calling theseEdit 2: Ryan Tibshirani has some C and Fortran code on fast median binning which may be a suitable starting point for a windowed approach.
我找不到具有顺序统计的 C++ 数据结构的现代实现,因此最终在 MAK 建议的顶级编码器链接中实现了这两种想法( 比赛编辑:向下滚动到 FloatingMedian)。
两个多重集
第一个想法将数据划分为两个数据结构(堆、多重集等),每次插入/删除的时间复杂度为 O(ln N),不允许在没有很大成本的情况下动态更改分位数。即我们可以有滚动中位数,或滚动 75%,但不能同时两者。
线段树
第二种想法使用线段树,它的插入/删除/查询时间复杂度为 O(ln N),但更灵活。最重要的是“N”是数据范围的大小。因此,如果您的滚动中位数有一个包含 100 万个项目的窗口,但您的数据从 1..65536 变化,那么 100 万个滚动窗口的每次移动只需要 16 次操作!
C++ 代码类似于 Denis 上面发布的内容(“这是一个用于量化数据的简单算法”)
GNU 阶数统计树
就在放弃之前,我发现 stdlibc++ 包含阶数统计树!
它们有两个关键操作:
请参阅libstdc++手册policy_based_data_structs_test(搜索“split and join”)。
我已将树包装在一个方便的标头中,供支持 c++0x/c++11 样式部分 typedef 的编译器使用:
I couldn't find a modern implementation of a c++ data structure with order-statistic so ended up implementing both ideas in top coders link suggested by MAK ( Match Editorial: scroll down to FloatingMedian).
Two multisets
The first idea partitions the data into two data structures (heaps, multisets etc) with O(ln N) per insert/delete does not allow the quantile to be changed dynamically without a large cost. I.e. we can have a rolling median, or a rolling 75% but not both at the same time.
Segment tree
The second idea uses a segment tree which is O(ln N) for insert/deletions/queries but is more flexible. Best of all the "N" is the size of your data range. So if your rolling median has a window of a million items, but your data varies from 1..65536, then only 16 operations are required per movement of the rolling window of 1 million!!
The c++ code is similar to what Denis posted above ("Here's a simple algorithm for quantized data")
GNU Order Statistic Trees
Just before giving up, I found that stdlibc++ contains order statistic trees!!!
These have two critical operations:
See libstdc++ manual policy_based_data_structures_test (search for "split and join").
I have wrapped the tree for use in a convenience header for compilers supporting c++0x/c++11 style partial typedefs:
我已经完成了 C 实现此处。这个问题中有更多细节:C 中的滚动中位数 - Turlach 实现 。
使用示例:
I've done a C implementation here. A few more details are in this question: Rolling median in C - Turlach implementation.
Sample usage:
我使用这个增量中值估计器:
它与更常见的均值估计器具有相同的形式:
这里eta是一个小的学习率参数(例如
0.001
),并且sgn()
是返回{-1, 0, 1}
之一的符号函数。 (如果数据是非平稳的并且您想要跟踪随时间的变化,请使用这样的常量eta
;否则,对于固定源,请使用类似eta = 1 / n
收敛,其中n
是迄今为止看到的样本数。)此外,我修改了中值估计器以使其适用于任意分位数。一般来说,分位数函数告诉您将数据分为两个分数的值:
p
和1 - p
。下面逐步估计该值:值
p
应在[0, 1]
范围内。这本质上将sgn()
函数的对称输出{-1, 0, 1}
转向一侧,将数据样本划分为两个大小不等的容器(分数)数据的p
和1 - p
分别小于/大于分位数估计)。请注意,对于p = 0.5
,这会简化为中值估计量。I use this incremental median estimator:
which has the same form as the more common mean estimator:
Here eta is a small learning rate parameter (e.g.
0.001
), andsgn()
is the signum function which returns one of{-1, 0, 1}
. (Use a constanteta
like this if the data is non-stationary and you want to track changes over time; otherwise, for stationary sources use something likeeta = 1 / n
to converge, wheren
is the number of samples seen so far.)Also, I modified the median estimator to make it work for arbitrary quantiles. In general, a quantile function tells you the value that divides the data into two fractions:
p
and1 - p
. The following estimates this value incrementally:The value
p
should be within[0, 1]
. This essentially shifts thesgn()
function's symmetrical output{-1, 0, 1}
to lean toward one side, partitioning the data samples into two unequally-sized bins (fractionsp
and1 - p
of the data are less than/greater than the quantile estimate, respectively). Note that forp = 0.5
, this reduces to the median estimator.这是量化数据的简单算法(几个月后):
Here's a simple algorithm for quantized data (months later):
滚动中位数可以通过维护两个数字分区来找到。
为了维护分区,请使用最小堆和最大堆。
最大堆将包含小于等于中位数的数字。
最小堆将包含大于等于中位数的数字。
平衡约束:
如果元素总数是偶数,那么两个堆应该具有相同的元素。
如果元素总数为奇数,则最大堆将比最小堆多一个元素。
中值元素:如果两个分区具有相同数量的元素,则中值将是第一个分区中的最大元素与第二个分区中的最小元素之和的一半。
否则中位数将是第一个分区中的最大元素。
Rolling median can be found by maintaining two partitions of numbers.
For maintaining partitions use Min Heap and Max Heap.
Max Heap will contain numbers smaller than equal to median.
Min Heap will contain numbers greater than equal to median.
Balancing Constraint:
if total number of elements are even then both heap should have equal elements.
if total number of elements are odd then Max Heap will have one more element than Min Heap.
Median Element: If Both partitions has equal number of elements then median will be half of sum of max element from first partition and min element from second partition.
Otherwise median will be max element from first partition.
也许值得指出的是,有一种特殊情况有一个简单的精确解决方案:当流中的所有值都是(相对)小的定义范围内的整数时。例如,假设它们必须全部位于 0 到 1023 之间。在这种情况下,只需定义一个包含 1024 个元素的数组和一个计数,并清除所有这些值。对于流中的每个值,增加相应的 bin 和计数。流结束后,找到包含 count/2 最高值的 bin - 通过添加从 0 开始的连续 bin 即可轻松完成。使用相同的方法,可以找到任意排名顺序的值。 (如果需要在运行期间检测存储箱饱和度并将存储箱的大小“升级”为更大的类型,则会出现一个小复杂情况。)
这种特殊情况可能看起来很人为,但实际上很常见。如果实数位于某个范围内并且已知“足够好”的精度水平,它也可以用作实数的近似值。这几乎适用于对一组“现实世界”对象进行的任何测量。例如,一群人的身高或体重。套装不够大?它对于地球上所有(单个)细菌的长度或重量也同样适用 - 假设有人可以提供数据!
看起来我误读了原文——它似乎需要一个滑动窗口中位数,而不是一个很长的流的中位数。这种方法仍然有效。加载初始窗口的前 N 个流值,然后对于第 N+1 个流值递增相应的 bin,同时递减与第 0 个流值对应的 bin。在这种情况下,有必要保留最后 N 个值以允许递减,这可以通过循环寻址大小为 N 的数组来有效地完成。由于中位数的位置只能改变 -2,-1,0,1 ,2 在滑动窗口的每个步骤中,无需将每个步骤中的所有 bin 求和至中值,只需根据修改了哪一侧 bin 来调整“中值指针”即可。例如,如果新值和被删除的值都低于当前中值,则它不会改变(偏移量 = 0)。当 N 变得太大而无法方便地保存在内存中时,该方法就会失败。
It is maybe worth pointing out that there is a special case which has a simple exact solution: when all the values in the stream are integers within a (relatively) small defined range. For instance, assume they must all lie between 0 and 1023. In this case just define an array of 1024 elements and a count, and clear all of these values. For each value in the stream increment the corresponding bin and the count. After the stream ends find the bin which contains the count/2 highest value - easily accomplished by adding successive bins starting from 0. Using the same method the value of an arbitrary rank order may be found. (There is a minor complication if detecting bin saturation and "upgrading" the size of the storage bins to a larger type during a run will be needed.)
This special case may seem artificial, but in practice it is very common. It can also be applied as an approximation for real numbers if they lie within a range and a "good enough" level of precision is known. This would hold for pretty much any set of measurements on a group of "real world" objects. For instance, the heights or weights of a group of people. Not a big enough set? It would work just as well for the lengths or weights of all the (individual) bacteria on the planet - assuming somebody could supply the data!
It looks like I misread the original - which seems like it wants a sliding window median instead of the just the median of a very long stream. This approach still works for that. Load the first N stream values for the initial window, then for the N+1th stream value increment the corresponding bin while decrementing the bin corresponding to the 0th stream value. It is necessary in this case to retain the last N values to allow the decrement, which can be done efficiently by cyclically addressing an array of size N. Since the position of the median can only change by -2,-1,0,1,2 on each step of the sliding window, it isn't necessary to sum all the bins up to the median on each step, just adjust the "median pointer" depending upon which side(s) bins were modified. For instance, if both the new value and the one being removed fall below the current median then it doesn't change (offset = 0). The method breaks down when N becomes too large to hold conveniently in memory.
如果您能够将值作为时间点的函数进行参考,则可以应用 引导 以生成置信区间内的引导中值。与不断地将传入值排序到数据结构中相比,这可以让您更有效地计算近似中位数。
If you have the ability to reference values as a function of points in time, you could sample values with replacement, applying bootstrapping to generate a bootstrapped median value within confidence intervals. This may let you calculate an approximated median with greater efficiency than constantly sorting incoming values into a data structure.
对于那些需要 Java 运行中位数的人来说...PriorityQueue 是您的朋友。 O(log N) 插入,O(1) 当前中值,O(N) 删除。如果您知道数据的分布,您可以做得更好。
For those who need a running median in Java...PriorityQueue is your friend. O(log N) insert, O(1) current median, and O(N) remove. If you know the distribution of your data you can do a lot better than this.
这是当精确输出不重要时可以使用的一个(用于显示目的等)
您需要总计数和最后中位数,加上新值。
为 page_display_time 等内容生成非常精确的结果。
规则:输入流需要在页面显示时间的顺序上平滑,数量大(> 30等),并且具有非零中值。
例子:
页面加载时间,800个项目,10ms...3000ms,平均90ms,真实中位数:11ms
经过30次输入,中位数误差一般<=20%(9ms..12ms),并且越来越小。
800 个输入后,误差为 +-2%。
另一位具有类似解决方案的思想家在这里:中值过滤器超高效实现
Here is one that can be used when exact output is not important (for display purposes etc.)
You need totalcount and lastmedian, plus the newvalue.
Produces quite exact results for things like page_display_time.
Rules: the input stream needs to be smooth on the order of page display time, big in count (>30 etc), and have a non zero median.
Example:
page load time, 800 items, 10ms...3000ms, average 90ms, real median:11ms
After 30 inputs, medians error is generally <=20% (9ms..12ms), and gets less and less.
After 800 inputs, the error is +-2%.
Another thinker with a similar solution is here: Median Filter Super efficient implementation
这是java实现
Here is the java implementation
基于 @mathog 的想法,这是一个 C# 实现,用于在具有已知值范围的字节数组上运行中位数。可以扩展到其他整数类型。
根据运行中位数进行测试,并计时:
Based on @mathog thoughts, this is a C# implementation for a running median over an array of bytes with known range of values. Can be extended to other integer types.
Testing against a running median, with timing:
如果您只需要平滑平均值,一种快速/简单的方法是将最新值乘以 x,将平均值乘以 (1-x),然后将它们相加。这将成为新的平均值。
编辑:不是用户所要求的,并且在统计上不有效,但对于很多用途来说已经足够好了。
我会把它留在这里(尽管有反对票)以供搜索!
If you just require a smoothed average a quick/easy way is to multiply the latest value by x and the average value by (1-x) then add them. This then becomes the new average.
edit: Not what the user asked for and not as statistically valid but good enough for a lot of uses.
I'll leave it in here (in spite of the downvotes) for search!