Linked by Thom Holwerda on Fri 16th Mar 2007 17:02 UTC, submitted by Shawna McAlearney
Privacy, Security, Encryption "Starting today, I plan on posting a monthly vulnerability scorecard for common server and workstation Operating System products. I'm going to keep these scorecards pretty clean of discussion, but you can review my methodology, sources and assumptions." Note that these results speak only of fixed vulnerabilities; the author aims to include information on non-fixed problems and the time it takes to fix problems as well. You should also read this, by the way.
Thread beginning with comment 221971
To read all comments associated with this story, please click here.
Not enough info provided
by Samhain on Fri 16th Mar 2007 18:20 UTC
Member since:

He does not give you any way (at least that I found) to actually see the vulnerabilities.

I wanted to look at them because that tells you if he falls for the obviously flaw in these kind of graphs.

Which is:

How many pieces of software, or packages are included? For example a typical linux distro includes several mail servers, usually at least two databases, probably a choice of more then one browser, several web programming languages (Perl, PHP, Ruby, Python, etc) and prebuilt apps, etc.

To compare apples vs. apples, this means that with windows you need to include any vulnerabilities fixed in Exchange, MS-SQL, Oracle, Firefox, Adobe products, Cold Fusion, etc.

Very rarely is this done and therefore you are comparing apples to oranges.

I don't know if he is doing this or not though, and there does not seem to be any way to see the actual vulnerabilities he is graphing.

Reply Score: 4

RE: Not enough info provided
by Thom_Holwerda on Fri 16th Mar 2007 18:36 in reply to "Not enough info provided"
Thom_Holwerda Member since:

He does not give you any way (at least that I found) to actually see the vulnerabilities.

How many pieces of software, or packages are included?

Is it that hard to read the teaser? Or the article? Both link to this methodology page with descriptions of which packages are included in the installations used.

Reply Parent Score: 1

dylansmrjones Member since:

The stats are useless. The graph says conveys no information at all, and his explanation on his bias is merely a preemptive strikt against constructive criticism.

Fact is that Jeff Jones is NOT counting fixed vulnerabilities. He is counting the number of binary packages updated as a result of a vulnerability. On most binary distributions in Linux, a single solved vulnerability typical means updating all packages linking against the package with said vulnerability. This gives a high number and a different number for different distributions despite have the same packages and having solved the same vulnerabilities.

His methodology is completely flawed and hilarious and must stem from his lack of knowledge of how to count to 3.

The numbers for Windows XP SP2 fits my experience with Windows 2003 Server (around 24 in that period). OTOH Gentoo has only had 5 or 6 fixes in the same period. And that's because I simply recompile the vulnerable package (or more for that matter).

For Redhat, Ubuntu and possibly Mac OS X Jeff Jones is not counting fixed vulnerabilities but is counting the number of applications directly or indirectly hit by the vulnerabilities. For Windows he is however counting number of fixed vulnerabilities instead of fixed packages.

He is comparing apples with oranges as is often the case with weird graphs.

Reply Parent Score: 5