您需要从测试工具中得到什么？
我是 Test Anything Protocol (TAP) IETF 小组的成员之一（如果有兴趣，请随时加入邮件列表）。 许多编程语言开始采用 TAP 作为他们的主要测试协议，他们希望从中获得比我们目前提供的更多的东西。 因此，我们希望从具有 xUnit、TestNG 或任何其他测试框架/方法背景的人员那里获得反馈。
基本上，除了简单的通过/失败之外，您还需要从测试工具中获得哪些信息？ 仅举一些示例：
- 文件名和行号（如果适用）
- 开始和结束时间
- 诊断输出，例如您得到的内容与您期望的内容之间的差异。
等等 ...
I'm one of the people involved in the Test Anything Protocol (TAP) IETF group (if interested, feel free to join the mailing list). Many programming languages are starting to adopt TAP as their primary testing protocol and they want more from it than what we currently offer. As a result, we'd like to get feedback from people who have a background in xUnit, TestNG or any other testing framework/methodology.
Basically, aside from a simple pass/fail, what information do you need from a test harness? Just to give you some examples:
- Filename and line number (if applicable)
- Start and end time
- Diagnostic output such as the difference between what you got and what you expected.
And so on ...
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（10）
最肯定的是每个单独项目的列表中的所有内容：
你得到的和你得到的之间的区别
你所期望的。
从我的头脑中没有太多其他的，但对于测试组，我想知道
Most definitely all things from your list for each individual item:
difference between what you got and
what you expected.
From the top of my head not much else but for the group of tests I would like to know
编写测试必须非常非常容易，运行它们也同样容易。 对我来说，这是测试工具最重要的一个功能。 如果有人必须启动 GUI 或跳过一堆麻烦才能编写测试，他们就不会使用它。
It must be very, very easy to write a test, and equally easy to run them. That, to me, is the single most important feature of a testing harness. If someone has to fire up a GUI or jump through a bunch of hoops to write a test, they won't use it.
任意一组标签 - 因此我可以将测试标记为“集成、UI、管理”。
（你知道我会问这个不是吗:-)
An arbitrary set of tags - so I can mark a test as, for example "integration, UI, admin".
(you knew I was going to ask for this didn't you :-)
对于你所说的，我要添加：
To what you said I'd add:
任何类型的诊断输出 - 尤其故障输出都是至关重要的。 如果测试失败，您不希望总是在调试器下重新运行测试以查看发生了什么 - 输出中应该包含一些内容。
我还喜欢查看关键系统变量（例如可用内存或硬盘空间）的前后快照，因为这些也可以提供很好的线索。
最后，如果您在任何测试中使用随机种子，请将种子写入日志文件，以便在必要时可以重现测试。
Any sort of diagnostic output - especially on failure is critical. If a test fails, you don't want to always have to rerun the test under a debugger to see what happened - there should be some cludes in the output.
I also like to see a before and after snapshot of critical system variables like memory or hard disk space available as those can provide great clues as well.
Finally, if you're using random seeds for any of the tests, write the seed out to the logfile so that the test can be reproduced if necessary.
我想要连接和嵌套 TAP 流的能力。
I'd like the ability to concatenate and nest TAP streams.
能够识别单个测试的唯一 ID（uuid、md5sum）——例如，在将测试结果插入数据库时使用，或者在错误跟踪器中识别它们以使 QA 能够重新运行单个测试。
这也使得在产品多个修订的整个生命周期中跟踪单个测试的行为成为可能。 这最终可能允许“历史”事件（新员工、产品发布、硬件升级）与因此类事件而失败的测试概况之间存在更大规模的关联。
我还认为 TAP 应该通过专用的侧通道发出，而不是与标准输出混合。 我不确定这是否属于协议定义的范围。
A unique id (uuid, md5sum) to be able to identify an individual test -- say, for use when inserting test results in a database, or identifying them in a bug tracker to make it possible for QA to rerun an individual test.
This would also make it possible to trace an individual test's behavior from build-to-build through the entire lifecycle of multiple revisions of a product. This could eventually allow larger-scale correlations between 'historic' events (new hire, product release, hardware upgrades) and the profile(s) of tests that fail as a result of such events.
I'm also thinking that TAP should be emitted through a dedicated side-channel rather than mixed in with stdout. I'm not sure this is under the scope of the protocol definition.
我使用TAP作为一组简单的C++测试方法的输出协议，并且发现了以下缺点：
最后，测试输出必须适合作为轻松生成 HTML 报告文件的基础，其中列出了成功的测试简洁地给出失败测试的详细输出，并且可以快速跳转到 IDE 到失败的测试行。
I use TAP as output protocol for a set of simple C++ test methods, and have seen the following shortcomings:
In the end, the test output must be suitable as basis for easily generating an HTML report file which lists succeeded tests very concisely, gives detailed output for failed tests, and makes it possible to quickly jump into the IDE to the failing test line.
可选的ascii彩色输出，绿色表示良好，黄色表示待处理，红色表示错误
待处理事物的想法
将运行各个测试的命令的测试报告末尾的摘要 其中
列出项目
出了点问题
测试中的某些内容正在等待
optional ascii coloured output, green for good, yellow for pending, red for errors
the idea of things being pending
a summary at the end of the test report of commands that will run the individual tests where
List item
something went wrong
something in the test was pending
TAP 的扩展想法：
能够附加 #json 评论...
- 可以被现有代码安全地忽略
- 定义明确的标签可以在 testanything.org 上轻松保留
- 易于生成、解析和读取复杂类型
- yaml 很痛苦
Extension idea for TAP:
Ability to attach a #json comment...
- can be safely ignored by existing code
- well-defined tags can be easily reserved at testanything.org
- easy to produce, parse and read complex types
- yaml is a pain