Submitted by Coverity has released the results of a Homeland Security Department-funded bug hunt that ranged across 40 popular open-source programs. The company found less than one-half of one bug per thousand lines of code on average, and found even fewer defects in the most widely used code, such as the Linux kernel and the Apache Web server.
Wow that is amazing! Kudos! Salutations! I dont know what else to say…that is most impressive! I always knew of the benefits of having your code open so that it can be easily scrutinized. It would be great if they could post Microsoft numbers in there as well but Microsoft wont ever allow it. I would like to see Java software in their list as well. AFAIK for mustang there are 670 bugs that are still not fixed.
…they should do the same tests on Windows and Office 😀
HEHEHE… That would be an eye opener I bet.
I can see the report now…
856 bugs per 1000 lines of code (The rest of the lines were comments) 😀
Edited 2006-05-09 18:52
This is great news for the F/OSS camp. There is a lot of FUD being thrown at the free software movement claiming that free software projects are bug-ridden and they are an inferior product. This test shows exactly the opposite.
This is great news for the F/OSS camp. There is a lot of FUD being thrown at the free software movement claiming that free software projects are bug-ridden and they are an inferior product. This test shows exactly the opposite.
Bogus. You cannot draw any sane conclusions about this test when compared to closed-source projects, as, well, they are closed source and therefore not tested. So you cannot compare. For all you know Windows has less bugs per 1000 lines.
———For all you know Windows has less bugs per 1000 lines.———–
Guess who microsoft won’t be submitting it’s coding to any time soon.
From the article:
Generally speaking, it is difficult to determine how well these open-source programs compare with their proprietary counterparts, Chelf said. Coverity has tested only a few commercial products, so direct comparisons cannot be made.
The company has drawn a number of observations from the study, and elaborated upon them in a paper accompanying the results (available on the Coverity site after registration). The chief lesson is that the number of lines of code is not an indicator of quality. Smaller programs can have plenty of bugs while larger projects, such as the Linux kernel, can be tightly controlled. Quality is more accurately reflected by the ratio of developers to the size of the code base and by the number of users who use the software (and provide feedback).
Still, .43 bugs per 1000 lines of code seems pretty good. Leaving out the comparison to closed source, I’d say it is fair to say that open source is able to create pretty bug-free projects.
Who said anything about comparing to closed source products? Any “comparisons” are entirely made up by yourself.
True, you’ll have to look at a previous report ( http://www.coverity.com/news/nf_news_12_14_04_story4.html ), which claims that proprietary software typically has 20-30 bugs/KLOC.
this is “only” static code checking. so while it has found many bugs related to things like usage of recently freed memory or uninitialized variables (which are important and even dangerous if unfixed) it doesn’t find bugs like “i clicked on X and instead of the expected Y, Z happened”.
so for kind of flaws coverity checks for, yes, we’re doing pretty damn well. just remember when quoting these numbers to others that it’s only one part of the overall defect picture.
still … pretty impressive. congrats to everyone involved =)
FTA:
the Linux Kernel had an average of .33 bugs per thousand lines of code
And yet Andrew Morton and Linus Torvaulds still think the kernel needs a bug fixing release. If thats not dedication to quality, then I dont know what is.
Well, any bug in a kernel is *bad*, and if the number of bugs is increasing, a bug-fix-release might be a good idea to remove potential bad stuff that otherwise will go unfixed.
NetBSD had an average of 0.001 — now that’s impressive.
“NetBSD had an average of 0.001 — now that’s impressive.”
Wow, the figures in that page change fast ! 1 or 2 hours ago it had near 1000 bugs fixed, and another 1500 pending or verified. Now it has a bit more than 1000 fixed and just 4 pending???? What happened with the rest???
By the way, the average for NetBSD was 0.33 or so just an hour ago…
Gnome (my desktop of choice) is twice as buggy on a per-line-of-code basis as KDE
This is a self interest study. You can’t trust the results folks.
This is the same kind of thing you see with Microsoft “get the facts” stuff.
This is no different.
Don’t be fooled.
Exactly. The company touting the results is trying to sell a product which does static-analysis. This is nothing more than a press release, advertising.
True, but in this case isn’t it in the company’s best interest to find more bugs? Thereby making open source look bad and Coverity look important. So it is kind of a “get the facts” campaign by Coverity against open source but by a company who wants to be friendly. It certainly isn’t doing this to make open source look good.
Edit: Maybe that’s actually what you were saying and I was just thrown off by the LinuxHater name.
Edited 2006-05-09 20:19
Coverity would be nowhere without open source software. Their business model hinges on providing free access to their static analysis tools for any open source project, so long as they agree to mark the bugs fixed as “identified by Coverity.” Coverity needs open source for its massive amount of freely available test-cases. They stand to improve their tools as much as these open source projects stand to improve their codebases. Obviously, Coverity gets to publicize their success at finding bugs in open source software in hopes of winning commercial contracts, some of which can be worth several million dollars (for their big clients like Sun and Oracle).
When Coverity first started publicizing open source analysis results around a few years ago, they found something like 2000 bugs in the Linux kernel and the PR focused on how Coverity found so many bugs. Now that many of the initally discovered bugs have been fixed and the defect densities are lower, they focus on low defect densities across open source projects. Why? Because such an article is likely to get linked from many online news sites that follow IT-related stuff.
This is a situation where everyone wins. It’s nice to know that open source developers have the opprtunity to use the leading-edge static analysis tool on their project at no charge, and that many projects (including the Linux kernel) have already integrated Coverity (plus sparse and maybe lint) into their development processes. In the spirit of the “more eyes on the code” principle, open source projects should take advantage of every free (gratis) automatic analysis system possible, as time and resources permit. Throw whatever we can at the code and shake out the obscure bugs.
I consider it a lot more trustworthy than the bullshit from Microsoft. Microsoft doesn’t mind lying, while sticking to the truth is right out necessary in the FLOSS world.
They would gain nothing by lying about it. Unlike Microsoft and the hilarious “Get the FUD”-campaign.
> You can’t trust the results folks.
no, the results are accurate. i’ve obviously seen the full reports for kde and they are pretty good. one just needs to be careful not to misinterpret the meaning of these numbers.
saying that there are very few code-level errors would be very accurate. this is a decent indicator of overall quality to an extent and certainly says something (quite positive) about security (though even there it’s not a complete picture of course).
saying that this proves there are next to no bugs of all categories would be incorrect.
but it is an accurate measure of one particular and meaningful metric. it’s also a huge puncture in the recent balloon microsoft has been floating that open source projects don’t use QA tools like static code checkers.
and this is generally where we separate ourselves from people like Microsoft: the community developers tend to try our best not to misrepresent the meaning of facts, but we have no problem standing by the facts for what they actually mean: good or bad.
you’ll find most corporate driven studies take facts of often dubious quality and then render them into rubbish through mischaracterization and poor interpretation (usually purposefully).
don’t lump us in with those clowns, because we (the community) haven’t earned that.
Edited 2006-05-09 20:20
Coverity got a government contract to do this code audit. They were not hired by a company with an interest in making free software look good. Yes, Coverity wants to look good, but they do so by finding bugs.
The results are trustworthy if interpreted correctly. Zero bugs found doesn’t mean zero bugs, of course, but the kinds of bugs Coverity does find are likely to be exploitable (buffer overruns and the like), which is why they control access to the results.
The interesting thing is to see how many bugs have been fixed since the results were first published 2 months ago. Back then, the Linux kernel had 0.33 bugs/Kloc, now just 0.20. AMANDA had 1.2 b/Kloc, now 0.0 (as other projects listed in the upper table).
NetBSD has fixed almost 1000 bugs since then, while FreeBSD didn’t fix even 1 (guess they’ve been busy with 6.1 release, but still….)
It’s a pity it doesn’t say with versions were first scrutinized and which versions have the fixes. Though new versions with bugs fixed may also have new ones…
If you read the footers, you would see that that number doesn’t necessarily mean bugs fixed. They can also mean false positives..
That is odd since I have seen many fixes ( http://groups.google.com/group/mailing.freebsd.cvs/search?q=coverit… ) by FreeBSD developers due to Coverity.
Maybe they (FreeBSD team) did fix bugs and didn’t report it to coverity? Their table shows 0 bugs fixed.
About false positives, yes, some might be. But they have at least to study it to mark it as false positive. So it again shows interest of developers in fixing any possible bug.
Edited 2006-05-09 22:27