Coverity has published the 2008 edition of its Open Source Report. The report uses static code analysis on C, C++, and Java source code to determine the quality of the code. These reports are funded by the US Department of Homeland Security and supported by Stanford University, and are part of the US government’s Open Source Hardening Project. The report is based on over two years’ worth of data from Coverity Scan.55 Million lines of recurring code from 250 open source projects, which lead to 14238 analysis runs, resulting in almost 10 billion lines of code analysed. The core results are neatly summarised in the executive summary:
- The overall quality and security of open source software is improving – Researchers at the Scan site observed a 16% reduction in static analysis defect density over the past two years
- Prevalence of individual defect types – There is a clear distinction between common and uncommon defect types across open source projects
- Code base size and static analysis defect count – Research found a strong, linear relationship between these two variables
- Function length and static analysis defect density – Research indicates static analysis defect density and function length are statistically uncorrelated
- Cyclomatic complexity and Halstead effort – Research indicates these two measures of code complexity are significantly correlated to codebase size
- False positive results – To date, the rate of false positives identified in the Scan databases averages below 14%
The 2006 benchmark data shows that the participating open source project had roughly one static analysis defect per 3333 lines of code. Based on the most recent results, the participating projects now have roughly one static analysis defect per 4000 lines of code, a decrease of 16%. A few projects have even eliminated all static analysis defects: courier-maildir, curl, libvorbis, and vim. Each of these projects consists of 50000 lines of code or more.