Embedded Software Development: How to Avoid Stepping on a Rake of WTF Code

Devices today are much more than just a six-pin three-button relay with a couple of LED lights. Users expect functionality as well as efficient interaction to define their experiences with devices. For example, factory workers want their equipment to be as reliable, compatible, safe, and easy to handle as the machines that drivers, pilots, and doctors use to facilitate their routine operations and literally save lives.

In fact, embedded software is the heart of any smart device these days; it ensures the seamless and failure-free operation of various types of medical/industrial equipment, vehicles, and security systems. Auriga’s expertise goes way beyond firsthand experience on the topic. The “embedded software development boutique” our founding fathers originally created evolved into a mature organization with a sophisticated structure and well-developed processes. What makes us really stand out? The fact that old-school embedded programmers have put down roots with us!

Sacred Secret: Diving in and Digging to the Core

Auriga has been engaged in embedded software development since 1990. We have developed software for many international companies, including IBM, HP, and Dräger Medical. Our clients trust us (and for good reason!) to dig into the core meaning of embedded software. Of course, we get our hands on software that makes a boiler LCD display return steam temperature and pressure data, but we also understand the real deal—the heart or, better to say, the brain that controls compressors and pressure relief valves and uses sensors for measurements.

Naturally, Auriga can easily design graphical user interfaces—no device these days functions without them. But our approach to such tasks is defined by the nature of our company—we assume that most GUI developers are not experienced in all the tricks of embedded software, including coding under certain conditions, such as the following:

  • limited memory (not just virtual but often even physical);
  • no operating system (or even if there is one, it’s not Linux but QNX or ThreadX or sometimes even OS-9);
  • no debugger, when successfully following the steps takes real luck (one can think of such scary words as “Flash,” “EEPROM,” or just “ROM” or using a UV light before a launch);
  • processor core suddenly not being x86, when insiders mysteriously whisper something about powerpc, arm, mips, or RISC V;
  • integration with some code that is not even C but some DSP assembler;
  • no processes or threads (not just POSIX-like, but no threads at all);
  • an almost dead processor that strongly disagrees with the brilliant idea of drawing a QWidget object in the upper-right corner of the display;
  • when some other programmers have used up all the memory from HEAP, and they faced off in a royal battle with STACK and secured a glorious victory, which made the accelerator program freeze and blow up;
  • finally, when detecting an incoming data error, it is impossible to throw an exception, because there is nobody to process it, and it can be dangerous and even fatal to process it on production equipment.

This is exactly why Auriga has a group of insiders who take the time to teach newcomers the tricks of embedded software science. When the library seems “ugly,” one needs to look into the source code and understand why the system has failed. If there is no source code disassembler, are you a programmer or what? When the hardware seems “weak,” you need to learn to optimize algorithms. Don’t get too obsessed about an object-oriented approach, and don’t get carried away by functions either. Also note that under limited memory conditions, a go-to operator is the way to go.

Stepping on Rakes of Embedded Software Development

All this is not even the thin end of the wedge. The real deal starts when you have to use live examples to prove to your customers that it costs more to hire experienced specialists who work with nonstandard tasks (that is, of course, if you plan on the smooth, trouble-free operation of the system). Indeed, the main difference between a pro and a regular developer lies in the fact that the former makes the software run anytime and anywhere, while the latter claims the hardware has overheated 15 minutes after launch.

Over the years in the industry, we have come across a variety of cases. The easy cases for us are when clients choose the wrong specialists, get into a deadlock, and pray for salvation. Well, that’s when we come in and save the day. But more often, we face more difficult cases, like “We just got the new pin-to-pin compatible microcontroller, changed the software a bit, and put it back together, but it does not work. Old binary files work on the new microcontroller but not as well as on the old one. Can you spend just a couple of days to look into it and fix it? Also, we would like to assemble the code for various hardware platforms so that it really works.”

In this case, as a CEO, I must react immediately by calling for a couple of inquisitors who exorcise the demons of user-mode programmers and an evangelist who introduces the client to traditional knowledge and increases their spiritual power. We take baby steps and don’t bog the client down with details or terms such as kernel of an operating system, driver, BIOS, or microcode. We start with the ABC’s and talk to the client first about basic things, such as the following:

  • Little-Endian and Big-Endian
  • Some variables sometimes take up six bytes instead of four.
  • For some reason, compact packaging often happens in some network protocols (everybody knows that structure fields need to be aligned with bus data width to get it up to speed, but not everybody remembers compatibility).
  • It is necessary to use a simulator to facilitate development and search for functional errors.
  • Side-by-side field tests using production hardware are a must. No battle plan survives first contact with the enemy, so it is important to get your hands dirty to actually identify bottlenecks.

We team up with the client’s developers to study the microcontroller register settings and assembler, calculate memory, and use macros instead of functions to speed up the processor. We consider using a new compiler or library the last resort, because we know it will kill us, and when it does, we look into source code (and binary files if necessary) and start fixing from there. We explain why it is necessary to use testing, and preferably automated testing, and why system testing is crucial. Our experience says you have to perform several million tests to catch microsecond race conditions, and manual testing will not catch anything except functional bugs in embedded software; only simulators will help here.

As we get through the ABC’s and act as an eye opener, many customers say, “We wish we would’ve known…” Hopefully, now you know how important it is to deal with experienced programmers. If you have issues with embedded software, let real professionals get their hands on it. Yes, the bill will be pricey, but your system will work, guaranteed!