Linked by ebasconp on Fri 10th Jun 2011 22:22 UTC
Benchmarks "Google has released a research paper that suggests C++ is the best-performing programming language in the market. The internet giant implemented a compact algorithm in four languages - C++, Java, Scala and its own programming language Go - and then benchmarked results to find 'factors of difference'."
Thread beginning with comment 476868
To read all comments associated with this story, please click here.
Meant as a Scala Demonstration
by Rooki on Fri 10th Jun 2011 23:20 UTC
Rooki
Member since:
2011-05-12

This paper has some pretty serious issues as has been discussed extensively https://groups.google.com/forum/#!topic/golang-nuts/G8L4af-Q9WE and http://www.reddit.com/r/programming/comments/hqkwk/google_paper_com...

For instance nearly all the optimization for the C++ code could have been applied to the Java code. Also according to Ian Lance Taylor:

Despite the name, the "Go Pro" code was never intended to be an example
of idiomatic or efficient Go. Robert asked me to take a look at his
code and I hacked on it for an hour to make a little bit nicer. If I
had realized that he was going to publish it externally I would have put
a lot more time into making it nicer.

I'm told that the program as compiled by 6g is significantly faster now
than it was when Robert did his measurements.


So yeah, in general a fairly floored benchmark, see the threads I linked for more details. I'm sure there's equivalent Java and Scala biased threads floating around as well!

Edited 2011-06-10 23:22 UTC

Reply Score: 3

pantheraleo Member since:
2007-03-07

I agree the paper has multiple problems.

Another issue with the the Java portion is they don't state whether they ran the JVM in client mode or server mode. That makes an enormous difference. Running it in server mode will almost always run circles around client mode.

Also, it's a rather theoretical benchmark that says next to nothing about real world performance. As they admit themselves, they didn't test threading performance or anything. The benchmark is not very indicative of the kinds of operations that most real world applications actually perform.

Edited 2011-06-10 23:46 UTC

Reply Parent Score: 5

Kasi Member since:
2008-07-12

You sir, are a comedic god.

Reply Parent Score: 0

Savior Member since:
2006-09-02

Another issue with the the Java portion is they don't state whether they ran the JVM in client mode or server mode.

They used server mode, as should be obvious from the first paragraph of section V:

"The benchmark consists of a driver ... and performs loop recognition on it for 15.000 times. The purpose is to force Java/Scala compilation, so that
the benchmark itself can be measured on compiled code."


The client-mode VM compiles the code right away and then leaves it at that. It's the server mode that needs a few thousand executions to recompile and optimize critical parts.

Also, it's a rather theoretical benchmark that says next to nothing about real world performance.

While the benchmark is rather theoretical, the issues discovered are real and DO affect real world performance. Everybody uses collections, recursions, etc. Also, I would definitely consider mathematical computations part of the "real world".

Reply Parent Score: 2