在计算快速流程运行的时间时，应使用多少次运行？
假设我要运行进程 X 并看看它需要多长时间。 我将把运行此过程的日期以及所花费的时间保存到数据库中。我想知道要放入数据库中的内容。
进程 X 几乎总是运行在 1500 毫秒以下，因此这是一个很短的进程。它通常运行在 500 到 1500 毫秒之间，这是一个相当大的范围（3 倍的差异）。
我的问题是，一次运行应将多少“运行”保存到数据库中？
每次运行都保存到数据库中 自己的行？
5 次运行，取平均值，然后保存 时间？
平均 10 次运行？
运行 20 次，删除超过 2 的任何内容 标准偏差消失，并保存 该范围内的所有内容？
有人有任何好的信息支持他们吗？
Lets say I am going to run process X and see how long it takes.
I am going to save into a database a date I ran this process, and the time it took. I want to know what to put into the DB.
Process X almost always runs under 1500ms, so this is a short process. It usually runs between 500 and 1500ms, quite a range (3x difference).
My question is, how many "runs" should be saved into the DB as a single run?
Every run saved into the DB as its
own row?5 Runs, averaged, then save that
time?10 Runs averaged?
20 Runs, remove anything more than 2
std deviations away, and save
everything inside that range?
Does anyone have any good info backing them up on this?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（6）
取决于你想要什么样的数据。我想说最初每次运行一行，然后分析数据，从那里开始。如果您想合并，可以存储 X 次运行的最小/最大/平均值。
Depends what kind of data you want. I'd say one line per run initially, then analyze the data, go from there. Maybe store a min/max/average of X runs if you want to consolidate it.
http://en.wikipedia.org/wiki/Sample_size
布莱恩是对的 - 你需要调查更多的。如果您的代码即使在“大多数”时间也有很大的差异，那么由于其他进程、操作系统分页或其他因素，您的测试环境可能会出现很大的波动。如果不是，那么似乎您的代码路径正在执行大量不同的工作，并且提出单个数字/运行数据来描述这种多模式系统的性能不会告诉您太多信息。因此，我想说尽可能隔离您的设置，运行至少 30 次试验，并了解您的性能曲线是什么样的。一旦有了这个，您就可以使用该维基百科页面得出一个数字，该数字将告诉您每次代码更改需要运行多少次试验，以查看性能是否以某种程度的统计显着性增加/减少。
http://en.wikipedia.org/wiki/Sample_size
Bryan is right - you need to investigate more. if your code has that much variance even "most" of the time then you might have a lot of fluctuation in your test environment because of other processes, os paging or other factors. If not it seems that you have code paths doing wildly varying amount of work and coming up with a single number/run data to describe the performance of such a multi-modal system is not going to tell you much. So i'd say isolate your setup as much as possible, run at least 30 trials and get a feel for what your performance curve looks like. Once you have that, you can use that wikipedia page to come up with a number that will tell you how many trials you need to run per code-change to see if the performance has increased/decreased with some level of statistical significance.
虽然说“保存每次运行”很好，但在您的情况下可能不切实际。然而，我确实认为仅存储平均值会消除太多数据。我喜欢存储十次运行的平均值，但我不仅仅存储平均值，还存储最大值和最小值，这样我就可以了解数据除了中心值之外的分布情况。
特别是最大和最小信息将告诉您极端情况出现的频率。 1500 毫秒的情况是千分之一的异常值吗？或者它是定期重复发生的事情吗？
While saying, "Save every run," is nice, it might not be practical in your case. However, I do think that storing only the average eliminates too much data. I like storing the average of ten runs, but instead of storing just the average, I'd also store the max and min values, so that I can get a feel for the spread of the data in addition to its center.
The max and min information in particular will tell you how often corner cases arise. Is the 1500ms case a one-in-1000 outlier? Or is it something that recurs on a regular basis?
将每次运行的数据保存到其自己的行中。然后，您可以随心所欲地使用和分析数据……即，您列出的所有其他选项都可以在事后执行。在不了解更多情况的情况下，其他人实际上不可能就如何平均/分析数据得出有意义的结论。
Save the data for every run into its own row. Then later you can use and analyze the data however you like... ie, all you the other options you listed can be performed after the fact. It's not really possible for someone else to draw meaningful conclusions about how to average/analyze the data without knowing more about what's going on.
最快的运行是仅对您的代码进行最准确计时的运行。
由于操作系统调度程序引入的噪音，所有较慢的运行速度都会变慢。
您所体验到的差异因机器而异，即使在相同的机器上，可运行进程集也会引入噪声。
The fastest run is the one that most accurately times only your code.
All slower runs are slower because of noise introduced by the operating system scheduler.
The variance you experience is going to differ from machine to machine, and even on identical machines, the set of runnable processes will introduce noise.
以上都不是。布兰很接近。您应该保存每个测量结果。但不要平均它们。在这种类型的分析中，平均值（算术平均值）可能非常具有误导性。原因是您的某些测量值会比其他测量值长得多。发生这种情况是因为事情可能会干扰您的过程 - 即使在“干净”的测试系统上也是如此。它也可能发生，因为您的过程可能不像您想象的那样具有确定性。
有些人认为，简单地采集更多样本（运行更多迭代）并对测量值取平均值将为他们提供更好的数据。事实并非如此。你跑得越多，遇到扰动事件的可能性就越大，从而使平均值过高。
更好的方法是运行尽可能多的测量（如果时间允许）。 100 不是一个坏数字，但 30 左右就足够了。
然后，按大小对它们进行排序并绘制图表。请注意，这不是标准分布。计算 计算一些简单的统计数据：平均值、中值、最小值、最大值、下四分位数、上四分位数。
与某些指导相反，不要“丢弃”外部值或“异常值”。这些通常是最有趣的测量结果。例如，您可以建立一个良好的基线，然后寻找偏离。了解这些偏差将帮助您充分了解流程如何工作、系统如何影响您的流程以及什么会干扰您的流程。它通常很容易暴露错误。
None of the above. Bran is close though. You should save every measurment. But don't average them. The average (arithmetic mean) can be very misleading in this type of analysis. The reason is that some of your measurments will be much longer than the others. This will happen becuse things can interfere with your process - even on 'clean' test systems. It can also happen becuse your process may not be as deterministic as you might thing.
Some people think that simply taking more samples (running more iterations) and averaging the measurmetns will give them better data. It doesn't. The more you run, the more likelty it is that you will encounter a perturbing event, thus making the average overly high.
A better way to do this is to run as many measurments as you can (time permitting). 100 is not a bad number, but 30-ish can be enough.
Then, sort these by magnitude and graph them. Note that this is not a standard distribution. Compute compute some simple statistics: mean, median, min, max, lower quaertile, upper quartile.
Contrary to some guidance, do not 'throw away' outside vaulues or 'outliers'. These are often the most intersting measurments. For example, you may establish a nice baseline, then look for departures. Understanding these departures will help you fully understand how your process works, how the sytsem affecdts your process, and what can interfere with your process. It will often readily expose bugs.