失败的测试是否会导致持续构建失败？
如果一个项目的测试作为构建计算机上构建过程的一部分执行，那么如果一组测试失败，整个构建是否会失败？
回答这个问题时应该考虑哪些因素？ 哪些测试失败重要吗？
引发此问题的背景信息：
目前我正在开发一个项目，该项目具有 NUnit 测试，这些测试是作为构建过程并在我们的 cruise control .net 构建机器上执行。
该项目过去的设置是，如果任何测试失败，构建就会失败。 原因是如果测试失败，则意味着产品无法工作/不完整/项目失败，因此构建应该失败。
我们添加了一些测试，尽管它们失败了，但它们对项目并不重要（更多详细信息请参见下文）。 因此，如果这些测试失败，该项目并不是完全失败，我们仍然希望它能够构建。
通过的测试之一是验证不正确的参数是否会导致异常，但未通过的测试是检查所有允许的参数是否不会导致异常。 因此该类拒绝所有无效的情况，但也拒绝一些有效的情况。 这对于项目来说不是问题，因为被拒绝的有效参数是边缘案例，应用程序不会依赖这些案例。
If one has a project that has tests that are executed as part of the build procedure on a build machine, if a set tests fail, should the entire build fail?
What are the things one should consider when answering that question? Does it matter which tests are failing?
Background information that prompted this question:
Currently I am working on a project that has NUnit tests that are done as part of the build procedure and are executed on our cruise control .net build machine.
The project used to be setup so that if any tests fail, the build fails. The reasoning being if the tests fail, that means the product is not working/not complete/it is a failure of a project, and hence the build should fail.
We have added some tests that although they fail, they are not crucial to the project (see below for more details). So if those tests fail, the project is not a complete failure, and we would still want it to build.
One of the tests that passes verifies that incorrect arguments result in an exception, but the test does not pass is the one that checks that all the allowed arguments do not result in an exception. So the class rejects all invalid cases, but also some valid ones. This is not a problem for the project, since the rejected valid arguments are fringe cases, on which the application will not rely.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（7）
如果它以任何方式可行，那就去做吧。 它大大减少了破碎窗口问题：
在系统中在没有（可见的）缺陷的情况下，引入一个小缺陷通常被视为一个非常糟糕的主意。 因此，如果您有一个处于绿色状态的项目（没有单元测试失败）并且您引入了第一个失败的测试，那么您（和/或您的同事）将有动力去解决问题。
另一方面，如果存在已知的失败测试，则添加另一个失败的测试将被视为保持现状。
因此，您应该始终努力保持所有测试运行（而不仅仅是“大多数”）。 将每一个失败的测试都视为构建失败的原因对于实现这一目标大有帮助。
If it's in any way doable, then do it. It greatly reduces the broken-window-problem:
In a system with no (visible) flaws, introducing a small flaw is usually seen as a very bad idea. So if you've got a project with a green status (no unit test fails) and you introduce the first failing test, then you (and/or your peers) will be motivated to fix the problem.
If, on the other side, there are known-failing tests, then adding just another broken test is seen as keeping the status-quo.
Therefore you should always strive to keep all tests running (and not just "most of them"). And treating every single failing test as reason for failing the build goes a long way towards that goal.
如果单元测试失败，则某些代码的行为不符合预期。 所以代码不应该被发布。
尽管您可以出于测试/错误修复目的进行构建。
If a unit test fails, some code is not behaving as expected. So the code shouldn't be released.
Although you can make the build for testing/bugfixing purposes.
如果您认为某个案例足够重要，需要为其编写测试，那么如果该测试失败，则该软件就会失败。 仅基于此，是的，它应该认为构建失败并且不再继续。 如果您不使用这种推理，那么谁来决定哪些测试不重要？ “如果失败了就没关系，但如果失败了就不行了”之间的界线在哪里？ 失败就是失败。
If you felt that a case was important enough to write a test for, then if that test is failing, the software is failing. Based on that alone, yes, it should consider the build a failure and not continue. If you don't use that reasoning, then who decides what tests are not important? Where is the line between "if this fails it's ok, but if that fails it's not"? Failure is failure.
我认为像您这样的良好设置应该始终成功构建，包括通过所有单元测试。
正如 Gamecat 所说，构建本身已成功，但此代码永远不应该投入生产。
想象一下，您的一位团队成员在代码中引入了一个单元测试（总是失败）所涵盖的错误。 它不会被测试发现，因为您允许该测试总是失败。
在我们的团队中，我们有一个简单的政策：如果所有测试均未通过，我们不会将代码投入生产。 这对于我们项目经理来说也是非常简单易懂的。
I think a nice setup like yours should always build successfully, including all unit tests passed.
Like Gamecat said, the build itself is succeeded, but this code should never go to production.
Imagine one of your team members introducing a bug in the code which that one unit test (which always fails) covers. It won't be discovered by the test since you allow that one test to always fail.
In our team we have a simple policy: if all tests don't pass, we don't go to production with the code. This is also a very simple to understand for our project manager.
在我看来，这实际上取决于您的单元测试，...
如果您的单元测试确实是单元测试（就像它们应该=>“参考无尽的书籍;）”）
那么构建应该失败，因为某些东西的行为不应该......
但最常见的是（不幸的是经常看到），在很多项目中，这些单元测试只覆盖一些“边缘”和/或集成测试，那么构建应该继续
（是的，这是一个主观答案；）
简而言之：
你知道单元测试没问题吗：失败；
其他：建立在
In my opinion it really depends on your Unit Tests,...
if your Unit tests are really UNIT tests (like they should be => "reference to endless books ;)" )
then the build should fail, because something is not behaving as should...
but most often (unfortunately to often seen), in so many projects these unit tests only cover some 'edges' and/or are integration tests, then the build should go on
(yes, this is a subjective answer ;)
in short:
do you know the unit tests to be fine: fail;
else: build on
真正的问题是你失败的测试。 您不应该进行可以失败的单元测试，因为这是一种边缘情况。 决定边缘情况是否重要 - 如果不重要，则删除单元测试； 如果是，则修复代码。
就像其他一些答案所暗示的那样，当单元测试失败时，这绝对是一种代码味道。 如果你忍受这种气味，那么你就不太可能发现下一个问题
The real problem is with your failing tests. You should not have a unit test where it's OK to fail because it's an edge case. Decide whether the edge case is important or not - if not then delete the unit test; if yes then fix the code.
Like some of the other answers implied, it's definitely a code smell when unit tests fail. If you live with the smell, then you're less likely to spot the next problem
所有的答案都很好，这就是我决定要做的：
让 NUnit 忽略不重要的测试（或者如果需要拆分失败的测试）（我在提出问题后记得这个功能）。 这允许：
我认为这是最好的妥协，迫使人们修复测试，但不一定立即（但他们必须捍卫现在不修复它的决定，因为每个人都知道他们做了什么）。
我实际上做了什么：修复了损坏的测试。
All the answers have been great, here is what I decided to do:
Make the tests (or if need be split a failing test) that are not crucial be ignored by NUnit (I remembered this feature after asking the question). This allows:
I think that is the best compromise, forcing people to fix the tests, but not necessarily right away (but they have to defend their decision of not fixing it now since everyone knows what they did).
What I actually did: fixed the broken tests.