使用嵌入式硬件进行测试自动化
有人成功地直接在嵌入式硬件上进行了自动化测试吗？
具体来说，我正在考虑对硬件层模块进行一系列单元测试的自动化。 我们需要对我们的硬件层代码有更大的信心。 我们的许多项目都使用中断驱动的定时器、ADC、串行 io、串行 SPI 设备（闪存）等。
这值得付出努力吗？
我们通常的目标是：
处理器：8 或 16 位微控制器（一些 DSP 的东西）
语言：C（有时是 c++）。
Has anyone had success automating testing directly on embedded hardware?
Specifically, I am thinking of automating a battery of unit tests for hardware layer modules. We need to have greater confidence in our hardware layer code. A lot of our projects use interrupt driven timers, ADCs, serial io, serial SPI devices (flash memory) etc..
Is this even worth the effort?
We typically target:
Processor: 8 or 16 bit microcontrollers (some DSP stuff)
Language: C (sometimes c++).
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（9）
当然。 在汽车行业，我们为每个新产品使用价值 100,000 美元的定制测试仪来验证硬件和软件是否正常运行。
然而，开发人员还构建了一个更便宜（低于 1,000 美元）的测试仪，其中包括一堆 USB I/O、A/D、PWM 输入/输出等，并且可以在工作站上使用脚本，或者专门构建的 HIL/SIL 测试软件例如MxVDev。
硬件在环 (HIL) 测试可能就是您的意思，它只涉及连接到设备 I/O 的一些 USB 硬件 I/O，并使用计算机上的软件对其运行测试。
值不值得就看情况了。
在高可靠性行业（飞机、汽车等），客户指定非常广泛的硬件测试，因此您必须拥有它才能获得投标。
在消费行业，对于不复杂的项目通常不值得。
不过，对于涉及多名程序员的任何项目，每晚在硬件上运行回归测试确实是件好事 - 很难正确地模拟硬件以达到满足自己要求的程度。软件测试就足够了。
当问题进入构建时，测试会立即显示。
通常，您执行黑盒和白盒测试 - 您在设备上运行诊断代码，允许您监视硬件中的信号和内存（这可能只是一个调试器，或者可能是您编写的对消息做出反应的代码）例如，公共汽车）。 这将是白盒测试，您可以在其中看到内部发生的情况（甚至导致某些事情发生，例如严重的内存错误，如果不亲自引入错误就无法进行测试）。
我们还运行了一系列“黑盒”测试，其中诊断路径被忽略，仅 I/O 被激励/读取。
对于更便宜的设置，您可以获得 100 美元的带有 USB 和/或以太网的微控制器板（例如 Atmel UC3 系列），您可以将其连接到您的设备并运行基本测试。
它对于产品维护特别有用 - 当项目完成时，在 CD 上存储一些工作板、测试仪和一整套软件。 当您需要进行修改或调试问题时，可以轻松地将其全部备份并在了解主要功能不受更改影响的情况下（在测试后）进行处理。
-亚当
Sure. In the automotive industry we use $100,000 custom built testers for each new product to verify the hardware and software are operating correctly.
The developers, however, also build a cheaper (sub $1,000) tester that includes a bunch of USB I/O, A/D, PWM in/out, etc and either use scripting on the workstation, or purpose built HIL/SIL test software such as MxVDev.
Hardware in the Loop (HIL) testing is probably what you mean, and it simply involves some USB hardware I/O connected to the I/O of your device, with software on the computer running tests against it.
Whether it's worth it depends.
In the high reliability industry (airplane, automotive, etc) the customer specifies very extensive hardware testing, so you have to have it just to get the bid.
In the consumer industry, with non complex projects it's usually not worth it.
With any project where there's more than a few programmers involved, though, it's really nice to have a nightly regression test run on the hardware - it's hard to correctly simulate the hardware to the degree needed to satisfy yourself that the software testing is enough.
The testing then shows immediately when a problem has entered the build.
Generally you perform both black box and white box testing - you have diagnostic code running on the device that allows you to spy on signals and memory in the hardware (which might just be a debugger, or might be code you wrote that reacts to messages on a bus, for instance). This would be white box testing where you can see what's happening internally (and even cause some things to happen, such as critical memory errors which can't be tested without introducing the error yourself).
We also run a bunch of 'black box' tests where the diagnostic path is ignored and only the I/O is stimulated/read.
For a much cheaper setup, you can get $100 microcontroller boards with USB and/or ethernet (such as the Atmel UC3 family) which you can connect to your device and run basic testing.
It's especially useful for product maintenance - when the project is done, store a few working boards, the tester, and a complete set of software on CD. When you need to make a modification or debug a problem, it's easy to set it all back up and work on it with some knowledge (after testing) that the major functionality was not affected by your changes.
-Adam
是的。
难度取决于您要测试的硬件类型。 正如其他人之前所说，问题在于您需要应用的外部刺激的复杂性。 外部刺激可能最好通过一些外部测试装置来实现（正如亚当·戴维斯所描述的）。
不过，需要考虑的一件事是您要验证的确切是什么。
人们很容易认为，要验证硬件和固件的交互，那么您实际上别无选择，只能直接应用外部激励（即将 DAC 应用于所有 ADC 输入等）。 但是，在这些情况下，您真正想要测试的极端情况通常会受到时序问题的影响（例如，当您执行函数 foo() 时到达中断），这将非常难以测试一种有意义的方式 - 甚至更难从中获得有意义的结果。 （即，我们运行此测试的前 100K 次一切正常。最后一次运行它失败。为什么？！？）
但是硬件的验证应该单独完成。 完成此操作后，除非它定期更改（通过可下载的 fpga 图像等），否则您应该能够假设硬件正常工作并纯粹测试您的固件。
因此，在这种情况下，您可以集中精力验证用于处理外部刺激的算法。 例如，使用固定值调用 ADC 转换例程，就好像它们直接来自 ADC 一样。 这些测试是可重复的，因此是有益的。 不过，他们将需要特殊的测试版本。
测试设备的通信路径将相对简单，不需要构建特殊的代码。
Yes.
The difficulty depends on the type of hardware that you're trying to test. As others have said earlier the issue is going to be the complexity of the external stimulus that you need to apply. External stimulus is probably best achieved with some external test rig (as Adam Davis has described).
One thing to consider, though, is exactly what it is that you're trying to verify.
It's tempting to assume that to verify the interaction of the hardware and the firmware then you've really no option but to directly apply external stimulus (ie. applying DACs to all of your ADC inputs, etc.). In these cases, though, the corner cases that you really want to test are often going to be subject to issues of timing (eg. interrupts arriving when you're executing function foo()) which are going to be incredibly difficult to test in a meaningful way - and even harder to get meaningful results from. (ie. The first 100K times we ran this test it was fine. The last time we ran it it failed. Why?!?)
But the verification of the hardware should be done separately. Once this is done, unless it's changing regularly (through downloadable fpga images or the like) then you should be able to assume that the hardware works and purely test your firmware.
So in this case you can concentrate on verifying the algorithms that are used for processing your external stimulii. For example, calling your ADC conversion routines with a fixed value as if they came from your ADC directly. These tests are repeatable and therefore of benefit. They will require special test builds though.
Testing the communications paths of your device is going to be relatively straightforward and shouldn't require special code builds.
是的。 我已经取得了成功，但这不是一个需要解决的直接问题。 简而言之，我的团队所做的就是：
使用自制的 C 单元测试框架定义了各种单元测试。 基本上就很多宏，大部分被命名为
TEST_EQUAL、TEST_BITSET、TEST_BITVLR等，
编写了一个启动代码生成器将这些已编译的测试编排到执行环境中。 它只是一个执行我们正常启动例程的小驱动程序 - 但它不是进入控制循环，而是执行一个测试套件。 完成后，它将最后运行的套件存储在闪存中，然后重置 CPU。 然后它将运行下一个套件。 这是为了在套件死亡时提供隔离。 （但是，您可能希望禁用此功能以确保您的模块配合。但这是集成测试，而不是单元测试。）
各个测试将使用串行端口记录其输出。 这对于我们的设计来说没问题，因为串行端口是免费的。 如果消耗了所有 IO，您将必须找到一种方法来存储结果。
有效！ 拥有它真是太好了。 使用我们的自定义数据记录器，您可以点击“测试”按钮，几分钟后，您将获得所有结果。 我强烈推荐它。
更新以阐明测试驱动程序的工作原理。
Yes. I have had success, but it is not a stragiht-forward problem to solve. In a nutshell here is what my team did:
Defined a variety of unit tests using a home-built C unit-testing framework. Basically, just a lot of macros, most of which were named
TEST_EQUAL,TEST_BITSET,TEST_BITVLR, etc.Wrote a boot code generator that took these compiled tests and orchestrated them into an execution environment. It's just a small driver that executes our normal startup routine - but instead of going into the control loop, it executes a test suite. When done, it stores the last suite to run in flash memory, then it resets the CPU. It will then run then next suite. This is to provide isolation incase a suite dies. (However, you may want to disable this to make sure your modules cooperate. But that's an integration test, not a unit test.)
Individual tests would log their output using the serial port. This was OK for our design because the serial port was free. You will have to find a way to store your results if all your IO is consumed.
It worked! And it was great to have. Using our custom datalogger, you could hit the "Test" button, and a couple minutes later, you would have all the results. I highly recommend it.
Updated to clarify how the test driver works.
我们在嵌入式系统上进行自动化测试，取得了良好的结果。 我们用高级（易于编程和调试）语言编写的测试在专用测试机器上运行。 这些测试通常会进行健全性检查或生成设备的随机输入，然后检查行为是否正确。 生成和维护这些测试需要做很多工作。 我们设计了一个框架，然后让实习生自己进行测试。
这不是一个完美的解决方案，测试肯定容易出错，但最重要的部分是改进现有的覆盖漏洞。 找到最大的洞并设计一些东西以自动方式覆盖它，即使它并不完美或无法覆盖整个功能。 稍后，当您的所有内容都得到一定程度的覆盖时，您可以回来解决最差的覆盖范围或最关键的功能。
需要考虑的一些事情：
We have had good results with automated testing on our embedded systems. We have test written in high level (easy to program and debug) languages that run on dedicated test machines. These test generally do sanity checking or generate random inputs into the devices, then check for correct behavior. There is a lot of work to generate and maintain these tests. We designed a framework and then let interns work on the tests themselves.
It's not a perfect solution, and the tests are certainly prone to errors, but the most important part is to improve on your existing coverage holes. Find the biggest hole and design something to cover it in an automated fashion, even if it isn't perfect or won't cover the entire feature. Later when all of your stuff is covered somewhat, you can come back and address the worst coverage or the most critical features.
Some things to consider:
如果您的目标是测试低级驱动程序代码，您可能需要创建某种测试装置，使用环回电缆或多个互连单元来测试每个驱动程序。 将具有已知良好软件的板与运行开发版本的板配对将允许您测试通信协议等方面的回归。
具体测试策略取决于您想要测试的硬件。 例如，可以通过呈现已知波形并转换一系列样本，然后检查适当的范围、频率、平均值等来测试 ADC。
我发现这种类型的测试在过去非常有价值，使我能够自信地修改和改进驱动程序代码，而不必担心破坏现有应用程序。
If your goal is to test your low-level driver code you will likely need to create some sort of test fixture, using loopback cables or multiple interconnected units to allow you to exercise each driver. Pairing a board with known-good software with a board running a development build will allow you to test for regressions in communication protocols, etc.
Specific test strategies depend on the hardware you wish to test. For example, ADCs can be tested by presenting a known waveform and converting a series of samples, then checking for the proper range, frequency, average value, etc.
I have found this type of testing to be very valuable in the past, allowing me to confidently modify and improve driver code without fear of breaking existing applications.
是的，我这样做了，尽管我一直有一个串行端口可用于测试 I/O。
通常很难让设备完全保持不变。 有些测试需要注释掉一行或添加一个调用，例如处理看门狗。
恕我直言，这比根本没有单元测试要好。 当然，您还需要进行完整的集成/系统测试。
Yes, I do this, although I've always had a serial port available for test I/O.
It is frequently difficult to leave the unit totally unmodified. Some tests require a line commented out or a call added e.g. to deal with a watchdog.
IMHO, this is better than no unit testing at all. And of course you need to be doing complete integration/system testing, too.
对嵌入式项目进行单元测试非常困难，因为它通常需要外部刺激和外部测量。
我们已经成功开发了一个外部串行协议（rs232或udp或tcpip消息），其基本命令用于在低级驱动程序中通过调试日志记录来执行硬件，以查找错误条件甚至轻微异常条件（特别是用于限制检查）
但是一旦开发完成，我们就可以根据需要在每次构建后运行测试。 它肯定会让您提供质量更好的产品。
Unit testing embedded projects is quite diffucult, as it usually requires a external stimulus and external measurment.
We have been successful in developing a external serial protocol (either rs232 or udp or tcpip messages) with basic commands for exercising the hw with debug logging in the low level drivers looking for erroneous conditions or even slightly abnormal conditions(espcially for limit checking)
But once developed we then can run the testing after every build if required. It will definitly allow you to deliver a better quality product.
如果您的目标是制造测试（确保模块正确组装，没有意外短路/开路/等），您应该首先关注测试电缆和连接器，然后是插座和焊接连接，然后是 PCB 本身。 这些物品都可以进行短裤和短裤测试。 通过查找将每条线驱动为高电平而其相邻线为低电平（反之亦然）的访问模式来打开，然后读回线的值。
如果不了解硬件的更多细节，就很难更具体，但大多数嵌入式处理器可以将 I/O 引脚设置为 GPIO 模式，从而简化此类测试。
如果您没有对 PCA 进行针床测试，则此测试应被视为新制造的主板的强制性第一步。
If your goal is manufacturing test (ensuring that the modules are properly assembled, no inadvertent shorts/opens/etc), you should focus first on testing cables and connectors, followed by socketed and soldered connections, then the PCB itself. These items can all be tested for shorts & opens by finding access patterns that drive each individual line high while its neighbors are low and vice-versa, then reading back the lines' values.
Without knowing more details of your hardware it's difficult to be more specific, but most embedded processors can set I/O pins to a GPIO mode that simplifies this sort of testing.
If you are not performing bed-of-nails testing on your PCAs, this testing should be considered a mandatory first step for newly manufactured boards.
我知道这已经过时了，但也许会有帮助。 是的，您可以做到，但这取决于您想在所需的解决方案上投资多少。 两年多来，我一直致力于 AUTOSAR 的 MCAL 层的测试和验证。 这是软件测试中可以获得的最低水平。 这是一种组件级测试。 有些人可能将其称为单元级别，但它略高于此水平，因为我们正在测试 MCAL 组件的 API。 例如：ADC、SPI、ICU、DIO 等。
使用的解决方案包括：
- 在目标微机上运行的测试框架
- dSPACE 盒，用于在需要时向目标提供信号并从目标读取信号
- 通过Vector CANape访问XCP以触发测试执行和结果收集
- 用于执行测试控制和结果验证的 python 框架
测试用例是用 C 编写的，它们与被测软件一起闪烁在目标上。 这是一个黑盒测试，因为我们没有以任何方式改变 MCAL 的实现。 我认为甚至连启动顺序都没有被触及。 空闲任务用于连续检查标志的状态，该标志是开始执行测试的信号。 使用 10 毫秒的任务来实际运行测试。 测试用例实际上是一个开关用例。 此开关中的每个案例都是一个测试步骤。 Python 在测试步骤级别触发测试执行。 这种方法的一个好处是可以重用具有不同参数的步骤。 该测试控制（执行什么以及如何执行）是由 Python 通过测试控制数据结构完成的，该数据结构充当测试实现与测试触发和评估机制之间的 API。 这就是 CANape 的用途。 设置要执行的测试并读取测试结果。 测试步骤获得的每个值都存储在数据结构的数组部分中。 测试步骤本身不参与任何验证，因为目标被视为测试环境的不可信组件。 验证由Python根据测试规范完成。 Python 正在解析这些规范，并能够自动创建测试触发脚本，包括每个测试步骤的验证标准。 每个测试用例的规范是一系列测试步骤描述及其验证标准。 其中一些步骤是 dSPACE 相关步骤。 例如，第一步是初始化某些内容，并要求在已配置的通道上捕获一些边缘，下一步是通过命令 dSPACE 设备在该通道上应用信号。
更便宜的解决方案是使用内部电路板而不是 dSPACE 设备。 在某种程度上，甚至可以使用可编程信号发生器，但如果您需要验证目标输出的信号，这将无济于事。
I know this is old now, but maybe it will help. Yes, you can do it but it depends on how much you want to invest in the solution you want. More than two years I have worked on test and validation for the MCAL layer of AUTOSAR. This is kind of the lowest you can get when it comes to software testing. It was a sort of component level testing. Some may call it unit level but it was slightly higher than that because we were testing the APIs of the MCAL components. Things like: ADC, SPI, ICU, DIO and so on.
The solution used involved:
- a test framework that was running on the target micro
- a dSPACE box to provide and read signals to and from the target when required
- XCP access through Vector CANape to trigger the test execution and results collection
- a python framework to perform the test control and validation of the results
The test cases were written in C and they were flashed on the target along with the software under test. It was a black box test cause we didn't alter in any way the implementation of the MCAL. And I think not even the startup sequence was touched. An Idle task was used to continuously check the state of a flag that was the signal to start executing a test. A 10 ms task was used to actually run the test. A test case was in fact a switch case. Every case in this switch was a test step. Python was triggering the test execution at the test step level. A good thing with this approach was the reusing of steps with different parameters. This test control, what to execute and how, was done by Python through a test control data structure acting as an API in between the test implementation and the test triggering and evaluation mechanism. This is what CANape was used for. To set the test to be executed and to read the results of the test. Every value obtained by a test step was stored in an array part of the data structure. The test step itself wasn't involved in any validation because the target was considered a non trust-able component of the test environment. The validation was done by Python based on the test specifications. Python was parsing these specifications and was able to automatically create test triggering scripts including the validation criteria for every test step. The specification of every test case was a series of test steps descriptions together with their validation criteria. Some of these steps were dSPACE related steps. As an example, one step was initializing something and was calling for some capturing some edges on an already configured channel, and the next step was applying the signal on that channel by commanding the dSPACE equipment.
A cheaper solution would involve using an in-house board instead of the dSPACE equipment. To some extent, even a programmable signal generator can be used, but that would not help if you need to validate signals output-ed by the target.