寻找“最干净”的方法数据子集，即变异性最低的子集

发布于 2024-07-16 23:10:02 字数 1197 浏览 10 评论 0原文

我试图在几个数据集中找到趋势。趋势涉及寻找最佳拟合线，但如果我想象该过程对于任何其他模型都不会太大不同（只是可能更耗时）。

有 3 种可能的情况：

所有好的数据，其中所有数据都符合具有低变异性的单一趋势
所有坏数据，其中所有或大部分数据表现出巨大的变异性，并且整个数据集必须被丢弃。
部分好的数据，其中一些数据可能是好的，而其余的则需要丢弃。

如果具有极端变异性的数据的净百分比太高，则必须丢弃整个数据集。这意味着基本上只有这种类型的数据，并且不良数据的百分比各不相同：
0% 不良 = 情况 1
100% 不好 = 情况 2
我只寻找低变异性的连续部分；也就是说，我不在乎是否有一些单独的点符合趋势

我正在寻找的是一种智能方法来对数据集进行分段并搜索指定的趋势。由于问题的本质，我并不是在寻找最适合总体趋势的部分。据我所知，具有“更干净”数据的小节最终的趋势线属性将与整体（其中包含异常值）略有不同。这正是我想要的，因为这部分数据最能反映实际趋势。

我精通 C++，但是，由于我试图使代码开源和跨平台，所以我坚持 ISO C++ 标准。这意味着没有 .NET，但如果您有 .NET 示例，如果您也能帮我将其转换为 ISO C++，我将不胜感激。我还了解JAVA，一些汇编和fortran。

数据集本身并不大，但约有 1.5 亿个，因此暴力可能不是最好的方法。

提前致谢

，我知道我已经把一些事情悬而未决，所以让我澄清一下：

每个数据集可以而且可能会有不同的趋势；即我并不是在所有数据集中寻找相同的趋势。
程序用户将定义他们想要的拟合程度
程序用户将定义在考虑趋势拟合之前子集必须有多连续
如果程序扩展为允许任何类型的拟合（不仅仅是线性），则用户将定义要拟合的模型 - 这不是优先事项，如果解决了上述查询，那么我确信这种扩展将相对微不足道
异常值是由于实验的性质和数据采集技术而产生的即使已知这些区域会产生异常值，仍然必须收集来自“不良”区域的数据。丢弃这些异常值并不意味着数据被操纵以适应任何趋势（统计免责声明，呵呵）。

原文

I am trying to find a trend in several datasets. The trends involve finding the best fit line, but if i imagine the procedure would not be too different for any other model (just possibly more time consuming).

There are 3 conceivable scenarios:

All good data where all the data fits a single trend with a low variability
All bad data where all or most of the data exhibits tremendous variability and the entire dataset must be discarded.
Partial good data where some of the data may be good while the rest needs to be discarded.

If the net percentage of data with extreme variability is too high then the entire set must be discarded. This implies that there is essentially only this type of data and the percentage of bad data varies:
0% bad = Case 1
100% bad = Case 2
I am only looking for contiguous sections with low variablity; i.e. I don't care if there are some individual points that fit the trend

What I am looking for is a smart way to subsection section the dataset and search for the specified trend. As is the nature of the problem, I am not looking for sections that best fit the overall trend. I understand that the subsection with "cleaner" data will end up having slightly different trendline properties than the overall (which would contain the outliers). This is exactly what i want since this part of the data would best best reflect the actual trend.

I am fluent in C++ but, since I am trying to make the code open source and cross-platform, I am stick to ISO C++ standards. This implies no .NET but if you have a .NET example I would appreciate if you could also help me convert it to ISO C++. I also have knowledge of JAVA, some assembly and fortran.

The datasets themselves are not huge but there are about 150 million of them and so brute force may not be the best way.

Thanks in advance

I understand that I have left some things up in the air and so let me clarify:

Each dataset can, and probably will, have different trends; i.e. I am not looking for the same trend throughout all datasets.
The program user will define how close a fit they want
The program user will define how contiguous the subset must be before it its considered for trend fitting
In case the program is extended to allow for any type of fit (not simply linear), the user will define what model is to be fit -- THIS IS NOT A PRIORITY and if the above query is solved then I am sure this expansion would be relatively trivial
The outliers come about as a result of the nature of the experiment and the data acquisition technique whereby data from "bad" sections must still be collected even though these areas are known to give outliers. The discarding of these outliers DOES NOT imply that the data is being manipulated to fit any trend (statistics disclaimer, hehe).

分享到QQ

分享到微博