我的经验是，自动化的成本是相当直观的。然而，经常需要修改现有程序以使其更易于测试。例如，在上面的示例中，如果您有一个 API 允许您向画布添加线条，那就太好了。如果您必须编写一个将逻辑坐标转换为屏幕的鼠标机器人等等，那么它就会更长。
In general, automated testing makes sense under the condition that the total investment in setting up and maintaining the automated test for the life of the product is lower than the total investment in doing the manual test whenever needed for the life of the product.
Therefore, automated testing becomes more attractive if it is very cheap to create the test compared to running them manually (e.g., fit testing where you essentially create spreadsheets), or if there is going to be a large enough number of executions of the test to offset the cost difference.
Unit testing works well because there is a large enough number of executions - with every minor change or refactoring you want to make sure that there is no new failure.
Acceptance testing is dicier because the likelihood of repeated execution changes significantly from project to project, and business needs affect long-term support and maintenance issues.
If a project is delivered once it passes acceptance testing and there is no further support or changes, then automated acceptance testing might not make sense unless it is very cheap to produce the automated tests.
If, on the other hand, one expects future releases (even to same customer) or a long term support contract, then the initial investment may become worth it over time.
Our company produces trading systems and algorithms for a variety of customers, and new features and products are built on top of old ones. The cost of running all our tests manually is so high, that we are working on adding automated acceptance testing for many scenarios that have previously been tested manually.
Follow up:
IMHO it is important to first have very clear and complete step-by-step test plans (A good QA person or team is critical here). In the case of acceptance testing, the customers may provide their own or QA would have to provide with their customers. I believe that a test plan should often be part of the contract.
Once you have that, it is easier to estimate how long it would take a human to run it (and you can experiment on a human to find out). It is important to pay attention to what steps require "grunt work", and what steps require real human input or discretion.
This is an important distinction. For example, suppose you are building a drawing program. if you tell a human to draw lines in specific coordinates, this can be automated easily. However, if your step says: "Verify that the shape is a flower", a human can do it easily but automating it is almost impossible. Many test cases can be semi-automated. You leave "hooks" for user input and have a tester focus on these. A button starts a series of operations, and the user is presented with "did the correct output come out?".
My experience is that the cost of automation is fairly intuitive. However, there is often a need to modify the existing program to make it more testable. For example, in the above example, if you have an API to allow you to add lines to the canvas, things are great. If you have to write a mouse robot that translates logical coordinates to screen and all that, it is much longer.
Writing automated acceptance tests can reap large benefits. In particular, it will:
Improve customer confidence in a product which is proven to be stable
Lower maintenance costs
Reduce the risk and fear of change
Allow for faster and more cheaper changes to the system
While you may be 100% familiar with the application now, try putting it down for a few months or years and come back to it to make changes. Believe me, it is much easier being able to run a suite of tests to prove that your new changes haven't broken anything.
Finally, consider writing your acceptance tests before you write the code, ala Test Driven Development fashion. Not only will this give you better code but you'll have full test suite coverage when your finished.
Don't kid yourself about the costs. Yes, it takes time, commitment and energy to write and maintain an automated acceptance test suite. But this is cheaper than alternative which is no tests and no idea what happens if you change something and what knockon effect it will have. What is your customer confidence worth? What is the real cost of fixing a bug after its been shipped?
发布评论
评论（3）
一般来说，如果在产品生命周期内设置和维护自动化测试的总投资低于在产品生命周期内需要时进行手动测试的总投资，那么自动化测试是有意义的。
因此，如果与手动运行测试相比，创建测试非常便宜（例如，在本质上创建电子表格的情况下进行测试），或者如果有足够多的测试执行次数，那么自动化测试就变得更具吸引力。抵消成本差异。
单元测试效果很好，因为有足够多的执行次数 - 对于每一个微小的更改或重构，您都希望确保没有新的失败。
验收测试更加危险，因为重复执行的可能性因项目而异，并且业务需求会影响长期支持和维护问题。
如果项目在通过验收测试后交付，并且没有进一步的支持或更改，那么自动化验收测试可能没有意义，除非生成自动化测试非常便宜。
另一方面，如果人们期望未来的版本（甚至是针对同一客户）或长期支持合同，那么随着时间的推移，最初的投资可能会变得值得。
我们公司为各种客户生产交易系统和算法，新功能和产品建立在旧功能和产品之上。手动运行所有测试的成本非常高，因此我们正在努力为许多以前手动测试的场景添加自动化验收测试。
跟进：
恕我直言，首先要有非常清晰和完整的逐步测试计划（一个好的质量保证人员或团队在这里至关重要）。在验收测试的情况下，客户可以提供自己的测试，或者 QA 必须与客户一起提供。我认为测试计划通常应该成为合同的一部分。
一旦掌握了这一点，就可以更容易地估计人类运行它需要多长时间（并且您可以在人类身上进行实验来找出答案）。重要的是要注意哪些步骤需要“繁重的工作”，以及哪些步骤需要真正的人工输入或判断力。
这是一个重要的区别。例如，假设您正在构建一个绘图程序。如果你告诉人们在特定坐标上画线，这可以很容易地实现自动化。然而，如果你的步骤是：“验证形状是一朵花”，人类可以轻松做到，但自动化几乎是不可能的。 许多测试用例可以半自动化。您为用户输入留下“钩子”，并让测试人员关注这些。一个按钮启动一系列操作，并向用户呈现“正确的输出出来了吗？”。
我的经验是，自动化的成本是相当直观的。然而，经常需要修改现有程序以使其更易于测试。例如，在上面的示例中，如果您有一个 API 允许您向画布添加线条，那就太好了。如果您必须编写一个将逻辑坐标转换为屏幕的鼠标机器人等等，那么它就会更长。
In general, automated testing makes sense under the condition that the total investment in setting up and maintaining the automated test for the life of the product is lower than the total investment in doing the manual test whenever needed for the life of the product.
Therefore, automated testing becomes more attractive if it is very cheap to create the test compared to running them manually (e.g., fit testing where you essentially create spreadsheets), or if there is going to be a large enough number of executions of the test to offset the cost difference.
Unit testing works well because there is a large enough number of executions - with every minor change or refactoring you want to make sure that there is no new failure.
Acceptance testing is dicier because the likelihood of repeated execution changes significantly from project to project, and business needs affect long-term support and maintenance issues.
If a project is delivered once it passes acceptance testing and there is no further support or changes, then automated acceptance testing might not make sense unless it is very cheap to produce the automated tests.
If, on the other hand, one expects future releases (even to same customer) or a long term support contract, then the initial investment may become worth it over time.
Our company produces trading systems and algorithms for a variety of customers, and new features and products are built on top of old ones. The cost of running all our tests manually is so high, that we are working on adding automated acceptance testing for many scenarios that have previously been tested manually.
Follow up:
IMHO it is important to first have very clear and complete step-by-step test plans (A good QA person or team is critical here). In the case of acceptance testing, the customers may provide their own or QA would have to provide with their customers. I believe that a test plan should often be part of the contract.
Once you have that, it is easier to estimate how long it would take a human to run it (and you can experiment on a human to find out). It is important to pay attention to what steps require "grunt work", and what steps require real human input or discretion.
This is an important distinction. For example, suppose you are building a drawing program. if you tell a human to draw lines in specific coordinates, this can be automated easily. However, if your step says: "Verify that the shape is a flower", a human can do it easily but automating it is almost impossible. Many test cases can be semi-automated. You leave "hooks" for user input and have a tester focus on these. A button starts a series of operations, and the user is presented with "did the correct output come out?".
My experience is that the cost of automation is fairly intuitive. However, there is often a need to modify the existing program to make it more testable. For example, in the above example, if you have an API to allow you to add lines to the canvas, things are great. If you have to write a mouse robot that translates logical coordinates to screen and all that, it is much longer.
我经常是一个人做生意，所以从我的角度来看，主要的好处是让机器代替我来做这件事。
I am often a one man shop, so from my perspective the primary benefit is having machines do it instead of me.
编写自动化验收测试可以获得巨大的好处。特别是，它将：
虽然您现在可能 100% 熟悉该应用程序，尝试把它放下几个月或几年，然后再回来做出改变。相信我，能够运行一套测试来证明您的新更改没有破坏任何内容要容易得多。
最后，考虑在编写代码之前编写验收测试，就像测试驱动开发的方式一样。这不仅会为您提供更好的代码，而且在完成后您将获得完整的测试套件覆盖。
不要在成本上欺骗自己。是的，编写和维护自动化验收测试套件需要时间、承诺和精力。但这比没有测试的替代方案便宜，也不知道如果你改变某些东西会发生什么以及它会产生什么连锁反应。您的客户信心值多少钱？发布后修复错误的实际成本是多少？
Writing automated acceptance tests can reap large benefits. In particular, it will:
While you may be 100% familiar with the application now, try putting it down for a few months or years and come back to it to make changes. Believe me, it is much easier being able to run a suite of tests to prove that your new changes haven't broken anything.
Finally, consider writing your acceptance tests before you write the code, ala Test Driven Development fashion. Not only will this give you better code but you'll have full test suite coverage when your finished.
Don't kid yourself about the costs. Yes, it takes time, commitment and energy to write and maintain an automated acceptance test suite. But this is cheaper than alternative which is no tests and no idea what happens if you change something and what knockon effect it will have. What is your customer confidence worth? What is the real cost of fixing a bug after its been shipped?