Linked by ebasconp on Fri 10th Jun 2011 22:22 UTC
Benchmarks "Google has released a research paper that suggests C++ is the best-performing programming language in the market. The internet giant implemented a compact algorithm in four languages - C++, Java, Scala and its own programming language Go - and then benchmarked results to find 'factors of difference'."
Thread beginning with comment 476870
To view parent comment, click here.
To read all comments associated with this story, please click here.
pantheraleo
Member since:
2007-03-07

I agree the paper has multiple problems.

Another issue with the the Java portion is they don't state whether they ran the JVM in client mode or server mode. That makes an enormous difference. Running it in server mode will almost always run circles around client mode.

Also, it's a rather theoretical benchmark that says next to nothing about real world performance. As they admit themselves, they didn't test threading performance or anything. The benchmark is not very indicative of the kinds of operations that most real world applications actually perform.

Edited 2011-06-10 23:46 UTC

Reply Parent Score: 5

Kasi Member since:
2008-07-12

You sir, are a comedic god.

Reply Parent Score: 0

pantheraleo Member since:
2007-03-07

You sir, are a comedic god.


You, sir. Added absolutely nothing of value to this conversation with this stupid comment.

Reply Parent Score: 2

Savior Member since:
2006-09-02

Another issue with the the Java portion is they don't state whether they ran the JVM in client mode or server mode.

They used server mode, as should be obvious from the first paragraph of section V:

"The benchmark consists of a driver ... and performs loop recognition on it for 15.000 times. The purpose is to force Java/Scala compilation, so that
the benchmark itself can be measured on compiled code."


The client-mode VM compiles the code right away and then leaves it at that. It's the server mode that needs a few thousand executions to recompile and optimize critical parts.

Also, it's a rather theoretical benchmark that says next to nothing about real world performance.

While the benchmark is rather theoretical, the issues discovered are real and DO affect real world performance. Everybody uses collections, recursions, etc. Also, I would definitely consider mathematical computations part of the "real world".

Reply Parent Score: 2

pantheraleo Member since:
2007-03-07

The client-mode VM compiles the code right away and then leaves it at that. It's the server mode that needs a few thousand executions to recompile and optimize critical parts.



Not true. Both the client and server mode JVM to JIT compilation. Also, it typically doesn't take a few thousand executions before JIT can be done.

While the benchmark is rather theoretical, the issues discovered are real and DO affect real world performance.


Some of the issues do, sure. But if I use a different test, I can make Java be 5 times faster than C++. For example, try doing a benchmark with infilife in Java vs. C++. The Java version is literally almost 5 times faster than the C++ version. The reason is because infilife is one of those tests where garbage collection provides a major advantage in that it optimizes memory allocation and deallocation. Sure, if you really want to spend the time on it, you could probably optimize the C++ version to be as fast as the Java version. But the code would be many times larger, and much more complex than the Java version.

The point is, using different code and different operations, I can very easily produce very different results. You can't take one sample like this and try to say that C++ is the fastest language.

Google even admitted in the paper that optimizing the C++ version was very difficult, and that most times those types of optimizations would not be done.

Reply Parent Score: 3