如何测量我的 MPI 程序传输的数据量？
我正在 24 台计算机上试验我的分布式集群算法（用 MPI 实现），我使用 BCCD（可启动集群 CD）将这些计算机设置为集群，可以在 http://bccd.net/。
我编写了一个批处理程序来运行我的实验，其中包括多次运行我的算法，改变节点数量和输入数据的大小。
我想知道每次运行算法时 MPI 通信中使用的数据量，以便我可以了解在改变前面提到的参数时数据量如何变化。我想使用批处理程序自动完成这一切。
有人告诉我使用 tcpdump，但我发现这种方法有些困难。
首先，在每次运行我的算法之前，我不知道如何在我的批处理程序（使用 C++ 编写的用于进行调用的命令系统）中调用 tcpdump，因为 tcpdump 需要另一个终端与我的应用程序并行运行。由于网络使用交换机，我无法在另一台计算机上运行 tcpdump。所以我需要在主节点上运行它。
其次，我在实验进行时看到了 tcpdump 的流量，但我无法弄清楚 MPI 使用的端口是什么。似乎使用了很多端口。我想知道如何过滤包裹。
第三，我尝试捕获整个包并使用 tcpdump 将其保存到一个文件中，几秒钟后该文件就变成了 3.5MB。但我整个实验需要2天时间。因此，如果我遵循这种方法，最终的日志文件将会很大。
理想的方法是仅捕获包标头中的大小字段，并将其相加以获得传输的数据总量。这样，日志文件会比捕获整个包小得多。但我不知道该怎么做。
另一个限制是我无法访问计算机光盘。所以我只有 RAM 和 4GB USB 闪存驱动器。所以我不能有巨大的日志文件。
我已经考虑过使用一些 MPI 跟踪或分析工具，例如 http:// /www.open-mpi.org/faq/?category=perftools。到目前为止我只测试过Sun Performance Analyzer。问题是我猜想在 BCCD 上安装这些工具会很困难，甚至可能是不可能的。除此之外，这个工具将使我的实验需要更长的时间才能结束，因为它增加了开销。但如果有人熟悉 BCCD 并认为使用其中一种工具是不错的选择，请告诉我。
希望有人能提供解决方案。
I'm experimenting my distributed clustering algorithm (implemented with MPI) on 24 computers that I set up as a cluster using BCCD (Bootable Cluster CD) that can be downloaded at http://bccd.net/.
I've written a batch program to run my experiment that consists in running my algorithm several times varying the number of nodes and the size of the input data.
I want to know the amount of data used in the MPI communications for each run of my algorithm so I can see how the amount of data changes when varying the previous mentioned parameters. And I want to do all this automatically using a batch program.
Someone told me to use tcpdump, but I found some difficulties in this approach.
First, I don't know how to call tcpdump in my batch program (which is written in C++ using the command system for making calls) before each run of my algorithm, since tcpdump requires another terminal to run in parallel with my application. And I can't run tcpdump in another computer since the network uses a switch. So I need to run it on the master node.
Second, I saw the traffic with tcpdump while my experiment was going on and I couldn't figure out what was the port used by MPI. It seems to use many ports. I wanted to know that for filtering the packages.
Third, I tried capturing whole packages and saving it to a file using tcpdump and in a few seconds the file was 3,5MB. But my whole experiment takes 2 days. So the final log file will be huge if I follow this approach.
The ideal approach would be to capture just the size field in the header of the packages and sum this up to obtain the total amount of data transmitted. In that way the logfile would be much smaller than if I were capturing the whole package. But I don't know how to do it.
Another restriction is that I don't have access to the computer disc. So I only have the RAM and my 4GB USB Flash drive. So I can't have huge logfiles.
I have already thought about using some MPI tracing or profiling tool such as those mentioned at http://www.open-mpi.org/faq/?category=perftools. I have only tested Sun Performance Analyzer until now. The problem is that I guess it will be difficult to install those tools on BCCD and maybe even impossible. In addtion to that, this tool will make my experiment take longer to end, sice it adds overhead. But if someone is familiar with BCCD and think it is a good choice to use one of those tools, so please let me know.
Hope someone have a solution.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
无论如何，如果存在使用分片内存进行通信的多核节点，则像 tcpdump 这样的实现将无法工作。
使用诸如 MPE 之类的东西几乎肯定是可行的方法。这些工具增加的开销非常小，如果您想对消息进行计数，那么一些开销总是必要的。您可以使用 mpitrace 写出每个 MPI 调用，并自行解析生成的文本文件。顺便说一句，请注意 bccd 网站 上明确讨论了 MPE。 MPICH2 内置了 MPE，但可以针对任何实现进行编译。我只发现 MPE 的开销非常小。
IPM 是另一个不错的工具，可以对邮件和大小进行计数；您应该能够解析 XML 输出，或者使用后处理工具并手动集成图表（按等级表示 bytes_rx/bytes_tx，或者消息缓冲区大小/计数图）。 IPM 的开销甚至比 MPE 还要少，并且大部分发生在程序完成运行以执行文件 I/O 之后。
如果您真的非常担心这些方法的开销，您始终可以使用 分析接口，包装了 MPI_Send、MPI_Recv 等，仅计算每个进程发送和接收的字节数，并仅输出总数在最后。
Implementations like tcpdump won't work if there are multi-core nodes which use shard memory to communicate, anyway.
Using something like MPE is almost certainly the way to go. Those tools add very little overhead, and some overhead is always going to be necessary if you want to count messages. You can use mpitrace to write out every MPI call, and parse the resulting text file yourself. By the way, note that MPE is explicitly discussed on the bccd website. MPICH2 comes with MPE built in, but it can be compiled for any implementation. I've only found a very modest overhead for MPE.
IPM is another nice tool that does counting of messages and sizes; you should be able either parse the XML output, or use the postprocessing tools and just manually integrate the graphs (say either bytes_rx/bytes_tx by rank, or the message buffer size/count graph). The overhead for IPM is even less than for MPE, and mostly comes after the program's finished running to do the file I/O.
If you were really super worried about the overhead with either of these approaches, you could always write your own MPI wrappers using the profiling interface that wrapped MPI_Send, MPI_Recv, etc, and just counted # of bytes sent and recieved for each process, and output only that total at the end.