MSVC10 /MP 在项目中跨文件夹构建非多核
我希望有人指出我们所遇到的错误或解决方法。
使用 /MP 编译项目时，似乎仅同时编译同一文件夹中的文件。我使用进程资源管理器来滑动命令行并确认行为。
项目过滤器似乎对同时编译的内容没有影响。
磁盘上的项目结构：
Folder\
project.vcxproj
source\
foo.cpp
foo1.cpp
other_folder\
bar.cpp
bar1.cpp
bar3.cpp
初始进程树：
MSBuild.exe
cl.exe ( passed: source\foo.cpp source\foo1.cpp )
cl.exe ( passed: source\foo.cpp )
cl.exe ( passed: source\foo1.cpp )
cl.exe 的 2 个子实例完成后，父进程关闭，并出现以下进程树：
MSBuild.exe
cl.exe ( passed: other_folder\bar.cpp other_folder\bar1.cpp other_folder\bar2.cpp )
cl.exe ( passed: other_folder\bar.cpp )
cl.exe ( passed: other_folder\bar1.cpp )
cl.exe ( passed: other_folder\bar2.cpp )
我们的源代码在与磁盘上标头的布局 - 我不想放弃它来利用 /MP。
I'm hoping someone points out something wrong or a workaround for what we are experiencing.
When compiling a project with /MP it appears that only files in the same folder are compiled concurrently. I used process explorer to swipe the command line and confirm the behavior.
Project filters seem to have all no impact on the what gets compiled concurrently.
Project structure on disk:
Folder\
project.vcxproj
source\
foo.cpp
foo1.cpp
other_folder\
bar.cpp
bar1.cpp
bar3.cpp
Initial process tree:
MSBuild.exe
cl.exe ( passed: source\foo.cpp source\foo1.cpp )
cl.exe ( passed: source\foo.cpp )
cl.exe ( passed: source\foo1.cpp )
After the 2 child instances of cl.exe complete the parent closes and the following process tree appears:
MSBuild.exe
cl.exe ( passed: other_folder\bar.cpp other_folder\bar1.cpp other_folder\bar2.cpp )
cl.exe ( passed: other_folder\bar.cpp )
cl.exe ( passed: other_folder\bar1.cpp )
cl.exe ( passed: other_folder\bar2.cpp )
Our source is nicely organized in many levels of nested folders that match the layout of the headers on disk - I would hate to have to give that up to take advantage of /MP.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（4）
在“对象文件名”（在 vcxproj XML 中，CL.exe 命令行上的 /Fo）项目中使用 %(RelativeDir) 会导致 msbuild 基于每个目录将 cpp 文件批处理到 cl.exe。这会对使用 /MP 获得的好处产生重大影响。
请注意，如果您的项目对目标文件使用 %(RelativeDir)，则配置可能会尝试避免来自不同文件夹中具有相同名称的 cpp 文件的 .obj 文件发生冲突。
/Fo 命令行参数通常是编译器将 obj 文件转储到的文件夹 - 仅传递了一个，因此给定目录的所有 cpp 文件一次只能传递给 CL.exe。
这是一个痛苦 - 但我很高兴有一个原因和一个解决方案。希望有帮助。
更新
一位队友发现，每当将 MSBuild 参数发送到 CL.exe 时，它似乎就会破坏或严重限制 /MP。这很可能是因为 /MP 要想正常工作，顶层 CL.exe 需要有一组 cpp 文件。
我们的解决方案是不使用任何 msbuild 参数（我认为它是 %params% ）作为“对象文件名”。这需要我们重命名一些 cpp 文件，这样它们就不会发生冲突。
希望这在 VS2012 或 VS2013 中有所改变。
The use of %(RelativeDir) in "Object File Name" ( in the vcxproj XML, /Fo on the CL.exe command line ) project causes msbuild to batch the cpp files to cl.exe on a per directory basis. This can have a significant impact on the benefits gained from using /MP.
Note that if your project uses %(RelativeDir) for object files its likely that the configuration is trying to avoid colliding .obj files that from cpp files with the same name in different folders.
The /Fo command line parameter is typically a folder that the compiler dumps the obj files into - there is only ONE passed, so all of the cpp files for a given directory can only be passed to CL.exe at a time.
That was a pain - but I'm glad there is a reason and a solution. Hope it helps.
Update
A team mate found that anytime an MSBuild parameter is sent to CL.exe it seems to break or severely limit /MP. This is most likely because for /MP to work well the top level CL.exe needs to have a bundle of cpp files.
Our solution was to not use any msbuild params ( I think its the %params% ) for 'Object File Name'. This required that we rename some cpp files so they did not collide.
Hope this has changed in VS2012 or VS2013.
根据 MSDN，这些文件应该当有线程来处理它们时进行编译，同时它不保证文件的编译顺序：
它还指出创建的进程数量将受到命令行中线程和文件数量的限制：
将这些与事物结合起来，我们可以看到编译器增量地处理编译（文件方面），以便它可以正确地将工作分派给子级，它这样做逐个文件夹。
如果您生成了一个自定义 make 文件，您也许能够解决这个问题，您应该能够同时处理多个文件夹（或者尝试使用
MSBUILD.exe工具） 。According it MSDN, the files should be compiled when ever there is a thread to handle them, it at the same time makes no guarantee's about the order in which files are to be compiled:
It also states that the number of created process will be bound by the number of threads and files in the command line:
Combining these to things, we can see that the compiler handles the compilation incrementally(file wise) so that it can dispatch work to children correctly, it dose this folder by folder.
You might be able to get around this if you generated a custom make file, where you should be able to get more than one folder being processed at the same time (or try using the
MSBUILD.exetool).我在这里用一种新方法解决了这个问题，以避免 .obj 破坏冲突
https://stackoverflow.com/a/26935613/4253427 考虑到上述内容
，我可以详细说明“每当 MSBuild 参数发送到 CL.exe 时，它似乎会破坏或严重限制 /MP”。 /MP 一次运行一个 CL.exe 调用。当您“发送 msbuild 参数”时，您实际上所做的是使用针对每个源文件定制的命令行创建多个 CL.exe 调用。这些不能批量处理。上面链接的解决方案尝试通过将命令行定制为最小的输出目录集来解决该问题。
这应该可以通过防止破坏同时保持高度批处理来解决您的问题，只要您的项目不包含 100 个在不同目录中全部名为 x.cpp 的文件...通常根据我的经验，只有几个冲突的 .obj在一个更大的项目中。
I've addressed this issue here with a new approach for avoiding .obj clobbering conflicts
https://stackoverflow.com/a/26935613/4253427
With the above in mind, I can elaborate on "anytime an MSBuild parameter is sent to CL.exe it seems to break or severely limit /MP". /MP works one CL.exe invocation at a time. When you "send an msbuild parameter" what you actually do is create multiple CL.exe invocations with commandlines tailored to each source file. These can't be batched. The above linked solution attempts to work around that by tailoring a commandline to a minimal set of output directories.
This should solve your problem by preventing clobbers while maintaining a high degree of batching, so long as your project doesn't contain 100 files all named x.cpp in different directories... usually in my experience there are only a few colliding .obj in a much larger project.
可以确认问题，而且原因并不容易找到，所以让我们添加更多关键字，以供其他用户遇到这个问题时使用。
我注意到我最新的 MSVC 项目的重建花费了太长的时间。特别是，CPU 核心几乎没有被利用，任务管理器中 cl.exe 进程的数量变化很大，并且输出窗口显示正在批量编译的源文件。一次编译 1 到 16 个文件，稍作停顿，然后编译另一组文件，依此类推。相比之下，在我的旧项目中，CPU 几乎得到充分利用，并且“输出”窗口显示正在编译的源文件的连续流。
现在，我的新项目的最大区别是更好地利用具有匹配目录结构的命名空间，这意味着最终会出现一些具有相同名称的类，并由于不同的 .obj 文件指向同一目录而导致冲突，从而导致更改 C/C++ 中的对象文件名 ->输出文件。
对输出窗口的更新也与目录结构匹配。如果存在一个名称空间/目录，其中包含一个源文件，则 VS 一次仅显示正在编译的一个文件。下一个目录有 10 个源文件，VS 显示所有 10 个源文件正在同时编译。
解决方案并不多。要么避免使用相同名称的类并且不要更改对象文件名，要么使用 zeromus 发布的解决方法，它的功能很棒。我的重建时间从 03:15 到 01:20，这是一个很大的差异，并且与大多数编译期间 CPU 利用率从 ~35% 到平坦的 100% 相匹配。
VS 2015、2017和2019都是这样，所以改变的希望不大。
Can confirm the issue, and the cause wasn't easy to find, so let's get some more keywords in for other users stumbling on it.
I've been noticing that rebuilds on my latest MSVC project take way too long. In particular, the CPU cores are barely utilized, the number on cl.exe processes in Task Manager varies wildly and the Output window shows the source files being compiled in what seem like batches. Anywhere from one to 16 are being compiled at a time, a small pause, then another set of files, and so on. In comparison, on my older projects the CPU is pretty much fully used and the Output window shows a continuous stream of source files being compiled.
Now, the big difference in my new project was making a better use of namespaces with a matching directory structure, which meant ending up with some classes having identical names and causing conflicts due to the different .obj files heading for the same directory, which lead to changing the Object File Name in C/C++ -> Output Files.
Updates to the Output window also match the directory structure. If there's a namespace/directory with one source file inside it, then VS shows just that one file being compiled at a time. The next directory has 10 source files and VS shows all 10 being compiled simultaneously.
There aren't many solutions. Either avoid classes with the same name and don't change the Object File Name, or use the the workaround posted by zeromus, it functions great. My rebuild time went from 03:15 to 01:20, which is quite a difference and matches the CPU utilization going from ~35% to a flatline 100% during most the compilation.
VS 2015, 2017 and 2019 all behave in this way, so not much hope for a change.