Embedded software is everywhere – from ordinary home appliances like dishwashers and microwave ovens to sophisticated medical devices, automotive engines, and flight control guidance systems. The global embedded software market was estimated at USD 10.46 billion in 2015 and is expected to grow by more than 7% annually from 2016 to 2023, according to the Global Market Insights report.
While the market is increasing in size, embedded software is steadily growing in complexity, sometimes requiring millions of lines of code. In many cases, embedded software performs both highly sophisticated and mission-critical functions. Even small software defects can affect the reliability and safety of the overall system and endanger people’s lives. Therefore, it is impossible to overestimate the importance of embedded software testing.
Maintaining our leadership in the field of embedded systems testing, Auriga has recently conducted a survey with the assistance of LTM Research, an independent research company, questioning 55 embedded companies about software testing approach and, then, analyzing their answers. The main takeaways from the survey can be found below. In short: automation, continuous integration, and direct access to the hardware are the keys to effective and efficient software testing.
Software Testing Approach Ratings
33% of companies participating in our survey self-assess their software testing approach as ‘excellent’ – these form the ‘Leaders’ group. The majority of respondents (38%) say their software testing approach is just ‘good’ – we refer to them as ‘Challengers’. The other 29% of companies are ‘Followers’ with the self-rating of ‘average’ or lower.
What are the main components of an excellent software testing approach? According to our respondents, there are three:
-
A well-defined testing strategy
94% of Leaders state they have a well-defined testing strategy, while 63% of Followers cannot say the same: in fact, 13% of Followers have not defined their testing strategy at all.
-
Percentage of test process automated
78% of Leaders have more than half of test process automated, while only 57% of Challengers and 31% of Followers managed to achieve that level of test process automation.
-
Team access to the test hardware
The majority of Leaders (89%) and Challengers (81%) state that their teams always have access to the test hardware. The majority of Followers (38%) have access to test hardware only sometimes.
At the same time, there are many barriers that can hinder an effective software testing approach. The survey shows that the hardest challenge in testing is providing access to exact versions of software and hardware for debugging and testing. Capturing the exact environment (configuration and interconnection of devices), which created the bug, and reproducing the sequence of events that led to the bug seem easier to our respondents.
Software Testing Practices
There is a wide range of software testing practices and techniques; however, it is important to understand which of them work best for the embedded software domain.
The survey reveals that bug replication in the embedded software case typically involves multi-device interactions. Almost 40% of our respondents note that complex scenarios (including interactions between multiple devices) are more common than large datasets and scenarios consisting of long sequences of events.
The most used software testing practices that may turn a Challenger or a Follower into a Leader include common automated testing framework (3rd party or in-house), remotely accessible hardware test lab, and continuous integration with automated tests and reports for each build.
Interestingly, on average, it is precisely these three practices that are least used by the survey respondents.
Test Early, Test Often
It is common knowledge for any IT specialist that the relative cost of bugfixing increases dramatically as we move through the software development life cycle. But have you ever tried to figure out the exact cost of the errors to be fixed at the later project phases? Our survey results on this issue were truly shocking.
That is why continuous testing matters. Not only does it help to deliver quality software solutions, but also improve quality awareness and build quality culture within the company.
Auriga’s Expertise in Embedded Software Testing
Auriga provides customers with a full range of embedded testing services, including test development and optimization, test execution and reporting, test automation, fixing defects, consulting and providing support for end-users.
According to Leonid Migunov, Testing expert at Auriga,
Embedded and system-level software development has always been one of Auriga’s main focuses. We have accumulated extensive knowledge in testing embedded software, implementing automated testing frameworks, and creating remotely accessible test labs. Our remote test labs provide constant and ubiquitous access to the hardware, efficient implementation of a common test environment, and a network of interconnected devices for reproducing hard defects.
With 25+ years in software development and testing, Auriga can help you bring your embedded software testing program to the next level. If you are interested in the subject, please read the paper “Testing of Embedded Software Products” [pdf] created by our engineers or learn more at our web page.
