posted by Tony Bourke on Thu 22nd Jan 2004 21:29 UTC

"A note on Benchmarking in general"
Benchmark Player Haters

Benchmarks are always such a contentious and embattled topic. No matter what benchmark you run, invariably someone has a negative comment to make (often rude and obnoxious). If you think otherwise, then write an article and include benchmarks of some sort and get it published. Then sit back and enjoy the belligerent emails and rude comments.

So why is this? Well, I think it's a combination of a couple of factors.

There are two pervasive emotional factors in benchmark-bashing. One is when a beloved and heroic operating system or beloved application ends up on the losing end to some vile, contemptible waste-of-time operating system or application. Such choices in operating systems, applications, hardware platforms, choice in databases, etc., are very personal, so it's easy for some people to take results of a benchmark as an affront to one's manhood.

The problem is that benchmarks, by their very nature, are narrow in scope and fail to encompass the complexity of an operating system, application, or hardware platform. As a result, someone with even a mediocre knowledge of the technology can easily poke holes, and make themselves seem smart in the process.

But that's not what benchmarks are varying depths of exploration into unknown territory. Sometimes they can be very comprehensive, and other times they can be very simple. They answer only the questions they are asked, and can provide a basis for asking other questions.

For instance, in my review of UnixWare 7.1.3, I ran some OpenSSL tests to see if there'd be any performance hit from the Linux emulation layer, known as the LKP. Why did I do this? I had no idea if there would be any performance penalty. No one had tested it before to my knowledge, so predicting the outcome was impossible.

Even someone intimately familiar with the inner workings of UnixWare, Linux, and the methodology that enables the LKP to work, could not know the outcome without running the test. Any supposition as to the results would be just that: a supposition. Anyone who says they know doesn't know what they're talking about, and it's easy to say so after the fact. Now that those benchmarks have been run, now we know. It was an easy test, and one that I could run.

But still, there were a few he doesn't know what he's talking about comments, including one particularly obnoxious guy who posted the same ignorant chest-beating comment in Slashdot and OSNews, like some sort of cyber-geek-stalker-player-hater. He relies on the virtues of hindsight, looking back and saying of course the results would be such!

If I ever do a review and you want to make a point, drop me a line. If you're polite about it, I'm happy to discuss it and I'll even take suggestions for other benchmarks. If you enjoy making obnoxious remarks about benchmarks done by myself or others, then do your own benchmarks, write it up in an article, and get it published. No one's stopping you, and it's not all that difficult.

So keep that in mind as you read my reviews and benchmarks, and as you read benchmarks from others.

Table of contents
  1. "Benchmark: 64bit vs 32bit, Page 1"
  2. "Benchmark: 64bit vs 32bit, Page 2"
  3. "Benchmark: 64bit vs 32bit, Page 3"
  4. "A note on Benchmarking in general"
e p (0)    70 Comment(s)

Technology White Papers

See More