如何在嵌入式系统中执行回归测试
在嵌入式环境中或在自动化测试的可能性非常有限的其他情况下运行回归测试有哪些良好的实践和策略。
根据我的经验，许多测试必须手动执行，即测试人员需要按下一系列按钮并验证机器是否正常运行。 作为一名开发人员，很难保证自己的更改不会破坏其他内容。
如果没有适当的回归测试，在大型重构等过程中情况会变得更糟。
有人认识到这个问题吗？ 您是否找到了处理此类问题的良好解决方案或流程？
What good practices and strategies are there for running regression tests in embedded environments or in other situations where the possibility to automate tests is very limited.
In my experience a lot of the testing has to be performed manually i.e. a tester needs to push a sequence of buttons and verify that the machine behaves correctly. As a developer it is really hard to assure yourself that your changes don't break something else.
Without proper regression tests the situation gets even worse during big refactorings and such.
Does anyone recognize the problem? Did you find a good solution or process to deal with this kind of problem?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（9）
就我个人而言，我非常喜欢在目标硬件和我自己的计算机上编译我的嵌入式代码。 例如，当针对 8086 时，我包含了一个映射到 8086 硬件上重置的入口点和一个 DOS 入口点。 硬件的设计使得所有 IO 都被映射到内存。 然后，我在硬件模拟器中进行有条件编译，并有条件地将硬件内存位置更改为模拟硬件内存。
如果我要在非 x86 平台上工作，我可能会编写一个模拟器。
另一种方法是创建一个测试装置，其中硬件的所有输入和输出均通过软件控制。 我们在工厂测试中经常使用它。
有一次我们在 IO 硬件中内置了一个模拟器。 这样，可以通过 CAN 发送一些命令将硬件置于模拟模式来测试系统的其余部分。 同样，精心设计的软件可以具有“模拟模式”，其中模拟 IO 来响应软件命令。
Personally, I'm a big fan of having my embedded code compile on both the target hardware and my own computer. For example, when targeting an 8086, I included both an entry point that maps to reset on the 8086 hardware and a DOS entry point. The hardware was designed so all IO was memory mapped. I then conditionally compiled in a hardware simulator and conditionally changed the hardware memory locations to simulated hardware memory.
If I were to work on a non-x86 platform, I'd probably write an emulator instead.
Another approach is to create a test rig where all the inputs and outputs for the hardware are controlled through software. We use this a lot in factory testing.
One time we built a simulator into the IO hardware. That way the rest of the system could be tested by sending a few commands over CAN to put the hardware into simulated mode. Similarly, well-factored software could have a "simulated mode" where the IO is simulated in response to software commands.
对于嵌入式测试，我建议您在开发过程的早期就设计出解决方案。 将嵌入式代码沙箱化以在 PC 平台上运行会很有帮助，然后进行模拟:)
这将确保大部分代码的完整性，但稍后您仍然需要手动进行系统和验收测试。
For embedded testing, I would suggest that you design your way out of this very early in the development process. Sandboxing your embedded code to run on a PC platform helps a lot, and then do mocking afterwards :)
This will ensure integrety for the most of it, but you would still need to do system and acceptance testing manually later on.
明确地。
技术组合：
关于在 PC 编译器上进行编译：这对于高级模块以及具有适合测试的工具的低级模块来说肯定有意义。
例如，当涉及到必须处理来自多个源的实时信号的部分代码时，仿真是一个很好的起点，但我认为这还不够。 通常没有什么可以替代在尽可能真实的环境中在实际硬件上测试代码的方法。
Most definitely.
A combination of techniques:
With regard to compiling on a PC compiler: that would certainly make sense for high-level modules, and for low-level modules with a test suitable harness.
When it comes to, for example, parts of the code which have to deal with real-time signals from multiple sources, emulation is a good place to start, but I don't think it is sufficient. There is often no substitute for testing the code on the actual hardware, in as realistic an environment as possible.
与迄今为止的大多数响应者不同，我使用的嵌入式环境根本不像桌面系统，因此无法在桌面上模拟嵌入式系统。
为了编写好的测试系统，您需要测试系统具有前馈和反馈。 JTAG 是控制器件最常见的前馈方式。 您可以设置设备的完整状态（如果幸运的话甚至可以设置整个板），然后设置要运行的测试代码。 此时您会得到反馈。 JTAG 还可以用作反馈设备。 然而，在这种情况下，带有软件 API 的逻辑分析仪是最好的选择。 您可以查找引脚上的某些电平、计数脉冲，甚至解析来自流外设的数据流。
Unlike most responders so far, I work with embedded environments that do not resemble desktop systems at all, and therefore cannot emulate the embedded system on the desktop.
In order to write good testing systems, you need your test system to have feedforward and feedback. JTAG is the most common feed-forward way to control the device. You can set the complete state of the device (perhaps even the entire board if you're lucky) and then set the test code to run. At which point you get your feedback. JTAG can also serve as a feedback device. However, a logic analyzer with an software API is the best in this situation. You can look for certain levels on pins, count pulses and even parse data streams from streaming peripherals.
为各个子系统和整个项目提供模拟目标环境的测试工具/沙箱/模型。
这并没有消除在真实环境中进行测试的需要，但大大减少了测试的数量，因为模拟将捕获大多数问题，因此当它们全部通过并且您执行昂贵的人工驱动测试时，您有相当的信心您将首先通过该测试时间。
Provide test harnesses / sandboxes / mockups for individual subsystems, and for the entire project, that emulate the target environment.
This does not remove the need for tests in the real environment, but greatly reduces their number as the simulation will catch most problems so by the time they all pass and you perform the expensive human-driven test you are reasonably confident you will pass that first time.
除了到目前为止关于确保您的应用程序可以构建的建议之外
并至少在普通 PC 上进行部分测试（这对于
使用像 Valgrind 这样的工具）我会考虑你的软件设计。
我参与的一个项目有一个用于驱动硬件的组件，一个
用于处理管理任务和另一个网络管理任务。
网络管理由SNMP处理，因此很容易编写
远程运行以驱动硬件执行某些操作的脚本。
为了运行低级硬件测试，我编写了一个简单的脚本阅读器
解析测试脚本并将命令注入到我的 IPC 中
司机。 由于输出是基于视频的，因此很难自动化
除了肉眼之外的处理验证，但它确实节省了
我的RSI。 它对于生成强调的脚本也非常有用
测试或模拟已知的故障条件以确保错误不会出现
再次发生。
如果我重来一次，我可能会实现一个共享的
测试工具使用的库和发送核心的真实代码
消息。 然后我会用 python （或者其他东西）包装这个库
类似）所以我的测试可能稍微更“可脚本化”。
Apart from the suggestions so far about ensuring your app can build
and at least partially test on normal PC's (which is also useful for
using tools like Valgrind) I would think about your software design.
One project I worked on had a component for driving the hardware, one
for dealing with management tasks and another of network management.
The network management was handled by SNMP so it was easy to write
scripts that ran remotely to drive the hardware to do something.
To run the low level hardware tests I wrote a simple script reader
that parsed test scripts and injected commands into the IPC of my
driver. As the output was video based it was hard to automate the
verification of the handling other than by eye, but it certainly saved
me RSI. It was also very useful in generating scripts that stress
tested or simulated known failure conditions to ensure bugs didn't
re-occur.
If I where doing it all over again I would probably implement a shared
library used by the test harness and the real code to send the core
messages. I would then wrap up the lib in python (or something
similar) so my testing could be slightly more "scriptable".
我同意每个人都说自动化硬件是必须的 - 我们正在使用这种方法来测试我们的一些单元的嵌入式软件。 我们建立了充满硬件模拟器的大型两机架测试站，并使用 NI TestStand 结合 Labview VI、C# 代码、供应商 DLL 等来管理所有这些。 我们必须测试很多硬件——这就是为什么我们有那么多废话。 如果您只是测试软件，那么您可以将其缩减到最基本的功能。 测试串行接口？ 只需构建一个设备来模拟串行流量并测试所有消息（以及一些无效消息）即可确保软件正确响应。 测试DIO？ 这很简单——有大量的 USB 外设或嵌入式设备可以模拟 DIO。 如果计时很重要，您将必须使用另一个嵌入式设备来获得您正在寻找的严格公差，否则 PC 就可以了。
重要的是始终知道您正在测试什么，而不是测试除此之外的任何内容。 如果是软件，请确保测试尽可能独立于硬件。 如果您正在测试波形生成或使用 D/A 进行测试，请将任务分开 - 使用嵌入式设备上的特殊软件版本来测试 D/A 硬件，该软件除了输出预先安排的序列之外不会执行任何花哨的操作。电压水平。 然后您可以查看参考是否关闭、滤波器是否设置为错误的频率等。然后您应该能够独立于硬件测试软件 - 使用开发板测试软件并验证处理器的行为引脚是正确的。
I agree with everyone who says automated hardware is a must - we're using that approach to test embedded software with some of our units. We have built up large two-rack test stations full of hardware simulators and we use NI TestStand with a mix of Labview VIs, C# code, vendor DLLs, etc to manage all of it. We have to test a lot of hardware - that's why we have all of that crap. If you're just testing software then you can scale it back to the bare essentials. Testing a serial interface? Just build a device to simulate the serial traffic and exercise all of the messages (and a few non-valid messages) to ensure the software responds correctly. Testing DIO? That's easy-there are plenty of USB peripherals or embedded devices to simulate DIO. If timing is important you'll have to use another embedded device to get the tight tolerances you're looking for, otherwise a PC will do just fine.
The important part is to always know what you're testing and not to test anything other than that. If it's software, make sure the test is independent of the hardware to the largest degree possible. If you're testing waveform generation or something with a D/A, separate out the tasks - test the D/A hardware with a special build of software on the embedded device that doesn't do anything fancy except spit out a prearranged sequence of voltage levels. Then you can see if your references are off, if your filters are set to the wrong frequency, etc. Then you should be able to test the software independent of the hardware - use a development board to test the software and verify behavior at the processor pins is correct.
我工作中使用的解决方案是自动化的夜间构建和测试程序。
如果您使用某种通信协议，则测试脚本很容易运行。 这对于内部单元测试很有好处。 让情况变得更有趣（也更彻底）的是制作一个插入电路板的线束来模拟外部 IO。
仿真有利于开发和基本的初始测试，但真实的物理运行时间是系统验证的唯一可靠方法。 物理操作可以找出非代码问题（由编码方法引起），例如电压骤降、噪声、毛刺、去抖问题、竞争条件等。
长时间的系统测试也很重要。 设置自动化测试来连续几天/几周连续滥用系统是消除几个月后才可能出现的问题的好方法。 当事情开始变得奇怪时，告诉客户重新启动电源并不是所有行业都能接受的奢侈行为。
A solution in use where I work is automated nightly build and test procedure.
The test scripts are easy to run if you are using some sort of communication protocol. That's good for internal unit tests. What makes the situation more interesting (and thorough) is to make a wiring harness that plugs into the board to simulate external IO.
Emulating is good for development and basic initial testing, but real physical operating time is the only reliable method for system validation. Physical operation can ferret out non-code issues (caused by coding methods) such as voltage sags, noise, glitches, debounce issues, race conditions, etc.
Prolonged system testing is important as well. Setting up an automated test to abuse a system continuously for days/weeks straight is a good way to force out issues that may not crop up until several months later in the field. Telling a customer to just cycle power whenever things start acting funny is not a luxury that all industries can entertain.
根据我的经验，自动化硬件测试至关重要。 -- 在 PC 和 PC 上投资双编译 target 是一个“很好有”的功能，但如果有选择，我宁愿投资于自动化硬件测试。 最终这将是一个更具成本效益的解决方案，因为制造部门无论如何都希望/需要故障分析的能力。
In my experience, automated hardware testing has been critical. -- Investing in dual compilation in both PC & target is a "nice to have" feature but given the choice, I'd much rather invest in automated hardware testing. It'll be a more cost-effective solution in the end since the manufacturing arms will want/need the capability anyways for failure analysis.