“Earlier this month we published an article looking at the Linux versus OpenSolaris performance when using the new AMD Shanghai Opteron CPUs. Ubuntu Linux was faster than OpenSolaris 2008.11 in nearly all of the tests, but as mentioned in that article, OpenSolaris is still dependent upon GCC 3.4 where as Ubuntu and most other Linux distributions are now shipping with the newer and much-improved GCC 4 series. Following that article being published, Sun Microsystems had requested some compiler tests since they were confident the results would be different had their Sun Studio compiler been used. Well, in this article we now have some OpenSolaris benchmarks from the same AMD setup using GCC 3.4, GCC 4.0, and Sun Studio 12.”
I stopped reading here.
What’s the point of benchmarking GCC 4.0.2 in 2009?
GCC 4.1 was a big improvement over GCC 4.0.
GCC 4.2 was a big improvement over GCC 4.1.
GCC 4.3 was a big improvement over GCC 4.2.
GCC 4.3 was released one year ago. Even debian stable has it. GCC 4.4 is nearing release and is an especially important upgrade.
I can’t see any reason why one would want to benchmark GCC 4.0 in 2009 except trying to make GCC4 look bad, or at least, less impressive than it really is.
If any OS is still stuck at GCC 4.0, all that means is that you can’t use this OS to benchmark GCC4 in 2009, end of the story. That’s by no means an excuse to publish a “GCC4” benchmark without at least a very prominent notice that “This benchmark is of course irrelevant because we tested a GCC version that’s completely outdated”.
EDIT: I read a bit the article, and there’s a small mention of that fact on page 6, but that’s not enough for something that makes the whole benchmark totally irrelevant (since the other gcc benchmarked here, 3.4, is even more outdated)
Edited 2009-02-20 17:10 UTC
Well there’s no 4.3 available for Solaris yet. I’m sure whenever it’s released someone will test it. As for now, they’re still interesting to compare.
I for one would gladly see GCC vs MSVC on Windows for some OSS software. Hey, why not!
Your concerns are noted at the end of the article:
“Some of the Sun Studio slowdowns may be explained by a majority of the open-source projects targeting GCC as their compiler and catering their compiler flags to the GNU Compiler Collection with little or no optimizations for Sun Studio. At the same time, however, it is also worth noting the current stable series for GCC is 4.3. The BlastWave packages for GCC4 were at version 4.0.2, which is outdated and since then GCC has picked up support for SSE4 and various other features.”
Also consider that they are doing these tests on the same hardware and same OS. My very light googling suggests that getting a hold of 4.3 on OpenSolaris is a bit of task and is not at all a “stable” package. Not the best if you’re trying to operate psuedo-scientific tests.
They make always amateur benchmarks. Totally useless results also this time. I’m sure that they doesn’t know anything about optimization flags and even about behavior that by default compilers on x86 compile for i386 generic code. Very poor…
I don’t know if program compiled for the benchmark make uses of OpenMP, but if some of them does, then GCC won’t compare to sun studio. OpenMP was introduced in GCC 4.2, while sun studio 12 has openmp support.
So if some packages depend on it, that might be a reason why Sun’s compiler was faster for some test.
I was also told that Sun Studio on linux uses GCC internally, so it is not as fast as Sun Studio on Solaris… What I know for sure is that Studio 12 was compiling fine my program on linux and the executable was running fine too. But when ran on Solaris, the code horribly crashed… So I know _there is_ something different internally.
The biggest problem with the article is that they are benchmarking the compilation times, not the performance of the resulting code (OK, the first two looked at the performance of the generated executable — but the rest were measuring how long it took to compile something).
The thesis was that GCC was generating more efficient code and that’s why applications were faster under Ubuntu. In fact, their simple benchmarks seem to reflect that, to the extent they tested it, but they don’t recognize that.
Who cares if the compiler takes 100 seconds to rather than 60 to compile a project. Whichever generates the fastest running code without error is the winner.
From an end user stand point, this is true. From a developer’s stand point, however, compilation time is key to productivity.
I can see it now, “But Sir! Our build times are cut in half? Who cares if the applications run twice as slow? Is that really my concern?”
Long pause, “You’re right. It’s not your concern. Your time is too important to work here. Security will escort you from the premises and you’ll get your last 2 weeks by tomorrow.”
I think it’s pretty obvious that in that case you would use both compilers. The faster compiler at your desk and the better optimizer on the build farm. Infact, companies pretty much already do this with debug builds for developers.
Jesse
Edited 2009-02-21 06:45 UTC
That sounds like a really bad idea. What if it compiles fine on one compiler but doesn’t on the other? An application should really be tested with the compiler it is going to be built with.
Thats not what he said. The OP said the difference of 40 seconds in a build was irrelivent, he said that it was incredably relevent to a developer, not that it was more important then any other given feature.
As far as I could see, only a 1 out of 9 benchmarks was about compilation time.
While less important than other benchmarks (read: actual performance), if you have ever tried to building a (really) huge project project on both GCC (Linux) and VC/CL (Windows), you’ll know how it annoying it gets when the Windows build (even with GNU/Make Makes w/ multiple jobs) takes ~3-5 times the time it’s takes to build the Linux counter-part. (And I’m being polite)
… Plus, having one useless (?) benchmark doesn’t invalidate the rest of thier results.
– Gilboa
I would have really enjoyed seeing ICC and a new version of GCC4 in the line up.
If your working on Linux AVOID this.
We tried this on a project at work and it was a huge mess.
In order to use their compiler you then have to deploy ALL SORTS of custom libraries like custom version of libC etc… its nasty and makes a mess of your system.
I’m really not surprised. It’s incredibly difficult adding on a closed, third-party compiler to an open source system that traditionally defaults to another open source compiler.
This was always the refrain when using open source software on Solaris, and especially SPARC hardware, when the performance wasn’t up to much: “Recompile in Forte!” or Sun Studio Compiler as it is now. I don’t think they’ll ever learn really.
Edited 2009-02-25 17:29 UTC