"Earlier this month we published an article looking at the Linux versus OpenSolaris performance when using the new AMD Shanghai Opteron CPUs. Ubuntu Linux was faster than OpenSolaris 2008.11 in nearly all of the tests, but as mentioned in that article, OpenSolaris is still dependent upon GCC 3.4 where as Ubuntu and most other Linux distributions are now shipping with the newer and much-improved GCC 4 series. Following that article being published, Sun Microsystems had requested some compiler tests since they were confident the results would be different had their Sun Studio compiler been used. Well, in this article we now have some OpenSolaris benchmarks from the same AMD setup using GCC 3.4, GCC 4.0, and Sun Studio 12."
Post a Comment
Member since:
2006-01-02
I stopped reading here.
What's the point of benchmarking GCC 4.0.2 in 2009?
GCC 4.1 was a big improvement over GCC 4.0.
GCC 4.2 was a big improvement over GCC 4.1.
GCC 4.3 was a big improvement over GCC 4.2.
GCC 4.3 was released one year ago. Even debian stable has it. GCC 4.4 is nearing release and is an especially important upgrade.
I can't see any reason why one would want to benchmark GCC 4.0 in 2009 except trying to make GCC4 look bad, or at least, less impressive than it really is.
If any OS is still stuck at GCC 4.0, all that means is that you can't use this OS to benchmark GCC4 in 2009, end of the story. That's by no means an excuse to publish a "GCC4" benchmark without at least a very prominent notice that "This benchmark is of course irrelevant because we tested a GCC version that's completely outdated".
EDIT: I read a bit the article, and there's a small mention of that fact on page 6, but that's not enough for something that makes the whole benchmark totally irrelevant (since the other gcc benchmarked here, 3.4, is even more outdated)
Edited 2009-02-20 17:10 UTC
Member since:2005-08-09
Member since:2005-07-08
Your concerns are noted at the end of the article:
"Some of the Sun Studio slowdowns may be explained by a majority of the open-source projects targeting GCC as their compiler and catering their compiler flags to the GNU Compiler Collection with little or no optimizations for Sun Studio. At the same time, however, it is also worth noting the current stable series for GCC is 4.3. The BlastWave packages for GCC4 were at version 4.0.2, which is outdated and since then GCC has picked up support for SSE4 and various other features."
Also consider that they are doing these tests on the same hardware and same OS. My very light googling suggests that getting a hold of 4.3 on OpenSolaris is a bit of task and is not at all a "stable" package. Not the best if you're trying to operate psuedo-scientific tests.
Member since:
2009-02-20
Member since:
2006-01-04
I don't know if program compiled for the benchmark make uses of OpenMP, but if some of them does, then GCC won't compare to sun studio. OpenMP was introduced in GCC 4.2, while sun studio 12 has openmp support.
So if some packages depend on it, that might be a reason why Sun's compiler was faster for some test.
I was also told that Sun Studio on linux uses GCC internally, so it is not as fast as Sun Studio on Solaris... What I know for sure is that Studio 12 was compiling fine my program on linux and the executable was running fine too. But when ran on Solaris, the code horribly crashed... So I know _there is_ something different internally.
Member since:
2007-12-13
The biggest problem with the article is that they are benchmarking the compilation times, not the performance of the resulting code (OK, the first two looked at the performance of the generated executable -- but the rest were measuring how long it took to compile something).
The thesis was that GCC was generating more efficient code and that's why applications were faster under Ubuntu. In fact, their simple benchmarks seem to reflect that, to the extent they tested it, but they don't recognize that.
Who cares if the compiler takes 100 seconds to rather than 60 to compile a project. Whichever generates the fastest running code without error is the winner.
Member since:2005-09-22
From an end user stand point, this is true. From a developer's stand point, however, compilation time is key to productivity.
Member since:2005-11-21
From an end user stand point, this is true. From a developer's stand point, however, compilation time is key to productivity. "
I can see it now, ``But Sir! Our build times are cut in half? Who cares if the applications run twice as slow? Is that really my concern?''
Long pause, ``You're right. It's not your concern. Your time is too important to work here. Security will escort you from the premises and you'll get your last 2 weeks by tomorrow.''
Member since:2009-02-17
I think it's pretty obvious that in that case you would use both compilers. The faster compiler at your desk and the better optimizer on the build farm. Infact, companies pretty much already do this with debug builds for developers.
Jesse
Edited 2009-02-21 06:45 UTC
Member since:2005-07-07
That sounds like a really bad idea. What if it compiles fine on one compiler but doesn't on the other? An application should really be tested with the compiler it is going to be built with.
Member since:2006-02-05
Member since:2005-07-06
As far as I could see, only a 1 out of 9 benchmarks was about compilation time.
While less important than other benchmarks (read: actual performance), if you have ever tried to building a (really) huge project project on both GCC (Linux) and VC/CL (Windows), you'll know how it annoying it gets when the Windows build (even with GNU/Make Makes w/ multiple jobs) takes ~3-5 times the time it's takes to build the Linux counter-part. (And I'm being polite)
... Plus, having one useless (?) benchmark doesn't invalidate the rest of thier results.
- Gilboa
Member since:
2005-09-03
Member since:
2005-07-06
Member since:2005-07-06
I'm really not surprised. It's incredibly difficult adding on a closed, third-party compiler to an open source system that traditionally defaults to another open source compiler.
This was always the refrain when using open source software on Solaris, and especially SPARC hardware, when the performance wasn't up to much: "Recompile in Forte!" or Sun Studio Compiler as it is now. I don't think they'll ever learn really.
Edited 2009-02-25 17:29 UTC