All engineers and developers know they should be testing their code, but it is very easy for extensive testing to be left until the very end of a project.
This often results in finding large issues when there isn’t enough schedule remaining for a good fix. Then code ends up being delivered with quick fixes that will break very easily in the future.
It is often understood that the best way to combat this is to test early and often. However, that can be quite tedious with manual testing, especially for embedded code running on external targets.
We have found our process has improved greatly when utilizing automated testing across many projects.
Unit testing is a useful testing approach for development and performing regressions, but it really does not account for full system functionality and integration. One of the most important approaches to automated testing on embedded systems is HIL testing.
HIL testing involves running your software on the actual target hardware and verifying it is functioning correctly and failing gracefully when expected. This can be difficult to achieve as embedded targets and software can range significantly based on what interfaces are available, however, serial, USB, or ethernet are usually an option.
There are a ton of testing frameworks out there when you want to do unit testing, but there are not many options that support HIL testing. When a new project begins, the question always arises on whether to choose an off-the-shelf framework or to roll your own.
At DornerWorks, we have attempted to roll our own several times to mixed success. One project I was involved in developed a decent tool that we tried to reuse on multiple projects, but, without enough resources for upkeep, it bit rotted and essentially became useless.
After investigating the few options available for open-source test runners, I settled on utilizing Robot Framework. It isn’t specifically targeted for HIL testing, but with a couple of plugins, it works well as a HIL test runner and can support a myriad of interfaces that would be available on embedded targets. Also, python as a language is quite nice for writing tests due to its simplicity.
A continuous integration server will automatically run tests when certain triggers are met. Some examples of triggers are code being pushed to a specific branch, a merge request being made, a scheduled test time (such as nightly), or a manual trigger.
While CI isn’t always a necessity, it can greatly improve your development process and the further automation ensures that the test suite has run with the latest code.
We mostly utilize CI to perform build tests and unit tests on all merge requests. However, we also use it to automatically run the HIL tests on a target connected to the CI hardware before merging our development branch into our main branch.
One of the benefits of automated testing is generating testing repots. Robot Framework does this automatically and provides output that is very easy to navigate and summarize. This was very helpful for a customer of ours that not only wanted automated testing during development to show correct functionality and avoid regressions, but also wanted to be able to really show correct functionality with some robustness testing.
The goal was primarily to disable a driver and show that the system had an expected failure and then re-enable the driver and show the system working flawlessly. This allowed our customer to have test results and visual proof for their tech savvy customers to really show the security of their solution. The automated nature of the tests allows for the customer to easily get the results with newer versions of the codebase so that their customers can rest assured knowing that everything is still functioning as expected.
In the next blog, we walk through Robot Framework, explain some of the standard and custom libraries available, and show how you can write custom libraries of your own.
If you want to reduce the risk involved in developing technical products, DornerWorks engineers can help you achieve that confidence with automated testing. Schedule a meeting with our team today and we will help you plan out a solution that meets your goals.
