ESC study highlights software checking

A recent study carried out by a leading static analysis tool provider, PRQA, showed that, perhaps not surprisingly, engineers aren't as efficient at identifying code violations as a dedicated tool. However, while the headline results may seem contrived, the underlying premise is that an engineer's time and skills are better used in resolving the subjective issues that arise from automated code inspection.

The study is based on the results of PRQA's Developers' Challenge, held at the Embedded Systems Conference in Silicon Valley earlier this year, which targeted engineers with a "genuine interest in writing high quality code."

While its primary objective is to unashamedly demonstrate the value of automated code inspection and review, it highlighted the significant gap between engineers' ability to (rapidly) identify coding violations in a sample of C/C++ source code provided, which compiled with only a few warnings, yet contravened a number of recognized coding standards.

The inability for compilers to do a good job of reviewing code isn't news, neither is the fact that software can check software quicker than wetware, but the follow-on conclusion is that engineers are still needed to apply discretion to the results. In the challenge, 50 engineers of varying ability spent around 30 minutes (although one particularly diligent participant returned the sample the following day) to identify between none and 33 issues contained within the code.

The automated tool took considerably less time to identify 120. However, once the less subjective issues were identified and addressed (by engineers) there remained a number of violations that could only be assessed in context; something that even PRQA admits isn't possible using an automated approach.

In PRQA's defense, the intention isn't to replace code reviews but to expedite them. It's only fair to point out too that the static analysis tool was configured to mimic the compiler's own settings, as well as modifying the coding standards against which it was analyzed. This was intended to exclude violations that were perhaps not relevant to the code's level of completion, as well as avoiding compiler-centric issues that would otherwise be omitted.