Linked by Christopher W. Cowell-Shah on Thu 8th Jan 2004 19:33 UTC
General Development This article discusses a small-scale benchmark test run on nine modern computer languages or variants: Java 1.3.1, Java 1.4.2, C compiled with gcc 3.3.1, Python 2.3.2, Python compiled with Psyco 1.1.1, and the four languages supported by Microsoft's Visual Studio .NET 2003 development environment: Visual Basic, Visual C#, Visual C++, and Visual J#. The benchmark tests arithmetic and trigonometric functions using a variety of data types, and also tests simple file I/O. All tests took place on a Pentium 4-based computer running Windows XP. Update: Delphi version of the benchmark here.
Permalink for comment
To read all comments associated with this story, please click here.
Author's Reply, part 1
by Chris Cowell-Shah on Fri 9th Jan 2004 08:59 UTC

I expected plenty of comments and criticism, and you folks didn't disappoint! I do appreciate all of the suggestions on how to improve the benchmark. I'd like to respond to the questions or criticisms that either arose most often or that seem most significant. Some of the comments point to real methodological flaws, while others seem to come from a lack of understanding about what I was trying to achieve (which is probably my fault for not being clearer). Many of the complaints could have been avoided if I had included more detailed explanations of my testing procedures and their justifications, but I didn't want the article to get too long or too dry. So in no particular order, here we go...

Why didn't you include my favorite language? It's [faster|newer|cooler] than any of the languages you picked.
I had to limit the number of languages somehow, so I put together what I hoped was a representative selection of the major languages in use today. Also, I had limited time and limited skills. Sure I could have added Perl or Fortran or whatever, but then I would have had to learn enough Perl or Fortran to code the benchmark. Before starting this project, the only language I knew well enough to code even this simple benchmark in was Java/J#. Besides, if anyone is really interested in seeing how AppleScript, Standard ML of New Jersey, or Visual C# running on Mono compare, I invite you to adapt my code and run it yourself. Porting over the benchmark should be trivial if you already know the language, and I'd love to see more results (particularly if you use Lisp, Forth, Perl, or Ruby).

Why not use other C compilers? There are a ton of them out there.
See above.

Why didn't you test on AMD hardware, or on a Solaris box?
The only machine I had ready access to was my trusty P4 laptop.

The GCC C code is going to run faster in a POSIX environment, linked to glibc instead of Windows libraries. Why didn't you run it on Linux?
Lack of time, lack of space on my hard drive to dual-boot even a minimal Linux distro. I did run the gcc code within Cygwin, linked to the Cygwin libraries (I assume Cygwin uses glibc, but don't know for sure). I didn't post those results since they were nearly identical to the results of the gcc code linked to Windows libraries, but in retrospect I should have included them in my report.

You didn't really test a fully interpreted language. Python gets compiled down to bytecode by the Python interpreter, so it doesn't count. Why not include Perl or PHP?
Good point. I didn't realize that any compilation was going on at all with Python until I read about it here. So yes, it would be instructive to see Perl results (assuming it really is fully interpreted--there seems to be some debate here on that point). But I don't know Perl and am trying my best never to learn it.

All .NET languages should perform the same. Why did you benchmark all four of them?
Because I wanted to see if Microsoft is telling the truth when they say that functionally identical code gets compiled into identical MSIL code. It turns out that, for the most part, it does.

You can't be a serious .NET programmer if you don't even know how to start an unmanaged Visual C++ project!
You're right. I'm not. But now I know how to do it, thanks. I considered using Visual C++ 6 instead, but ultimately decided to just stick with whatever languages Microsoft's throwing their weight behind now, and that's the .NET languages.

It's unfair to test Java 1.4.2 with the -server flag, but Java 1.3.1 with the -client flag. Everyone knows that the -server version of the JVM runs bytecode faster than the -client version (at the expense of slightly longer startup time).
I was astonished to see that the JVM included with the 1.3.1 SDK doesn't have a -server version. The only flag available for setting the JVM version is -hotspot, which is the default JVM for 1.3.1 anyway. Install a 1.3.1 SDK, type "java -help" and see for yourself. Maybe they had the -server option in earlier versions of 1.3.1--I used 1.3.1_09.

Why is it surprising to see Java perform well? The bytecode is compiled by the JVM before (or as) it runs, after all.
It's surprising only because everyone thinks Java is slow. This is probably because early versions of Java really were slow, but I think we're now witnessing a case of perception lagging behind reality.

Java 1.4.2 is slow on the trig component because it's using the StrictMath package, which trades speed for accuracy.
Well, maybe. I called the Math package, which (as stated in the Javadoc) may or may not in turn call the StrictMath package. So I don't really know what's going on behind the scenes. I did randomly compare results out to eight decimal places or so and got the same trig results for all languages.

You're not being fair to VB--you're using its native I/O functions instead of using the standard CLR I/O classes.
You're probably right, but what I did was... hang on. This requires a detour for a second. I'll come back to this after the next comment.

You said the only language you knew before writing these benchmarks was Java. Then what right do you have to call these real benchmarks? There are probably all sorts of optimizations that you didn't know about and didn't use--real programmers understand their languages better and know how to squeeze performance out of them. No one codes production-quality code after spending a single afternoon learning a language!
I beg to differ. For better or worse, tons of people code exactly like that. In my industry (IT consulting), virtually everyone does! It's absolutely routine to be given a programming task, a language, and an unrealistic deadline. You're expected to learn what you can from on-line help, whatever tutorials you can scrounge up on the net, and O'Reilly books, and cobble together code that works. In an ideal world we'd have loads of time to optimize everything and profile the heck out of code before releasing it, but on an actual project that's very rare. At least, that's been my experience. So I treated these benchmarks the same way: pick up a book, learn how to do what you need to do, spend a little time making sure you're being smart about stuff (use buffered I/O streams in Java, for example), but don't expect it to be 100% optimized. Then move on to the next language. My results won't duplicate results derived from laboratory conditions, but they should be close to real world (or at least, IT consulting world) results. This is a point I should have made much, much clearer in the article, and I'm sorry for the confusion I caused by not being making it more explicit.

You never answered the VB I/O question!
Right. I learned how to do VB I/O by using the built-in help in Visual Studio .NET 2003. I did what it told me to do. If it told me to use native VB I/O classes, that's what I did. If I had spent a lot more time poking around I might have been able to figure out how to use more efficient CLR classes, but that route was non-obvious and I had no way of knowing whether its code would have been faster without actually trying it. Again: I was trying to replicate real-world, time-constrained, scenarios with programmers who know the basics but are by no means experts. Having said all that, I appreciate the advice about speeding up VB I/O. Some day I may re-code with that change in mind.

continued in next post...