When doing research for my evaluation of Solaris 9 on my Ultra 5, I kept running into one comment over and over again: Sun’s C compiler produces much faster code than GCC does. However, I couldn’t find one set of benchmarks to back this up. (If you know of any, drop me an email.) Could this be yet another case of rumor-taken-as-fact?
If indeed Sun’s compilers are faster, how much faster? 5 percent? 20 percent? twice as fast? I’ve heard them all from various people, so I decided to do some tests on my own to see if I could see this supposed performance difference between GCC and Sun’s compiler.
Another potential question would be which version of GCC to use: That latest (as of this writing) 3.3.2, or GCC 2.95 which is still in wide use (especially on Linux, FreeBSD, NetBSD, and OpenBSD running on SPARC). So I included them both, to also answer the questions of which GCC produces the better code (on Solaris/SPARC). 64-bit binary creation was also tested, for binaries created by GCC 3.3.2 and Sun’s compiler.
But Tony, Sun’s compiler is so expensive! Yes, it is expensive, especially when compared to the free (as in beer and freedom) GCC. However, Sun does offer a free 60-day evaluation license for their compiler suite, which is what I used.
For Sun’s compiler, I used the 60-day trial for Sun ONE Studio 7, which can be found here. It includes C, C++, and Fortran compilers, as well as Java other development tools. The full version lists for $2,995.
I’m using the same applications as I did for my previous article on 32-bit versus 64-bit binaries. In fact, the tests for this article and the 32-bit versus 64-bit article were all done at the same time, and these articles done separate in the interest of brevity. Those applications are OpenSSL 0.9.7c, GNU gzip 1.2.4a, and MySQL 4.0.17.
For the GCCs, the compiler options were “-O3 -mcpu=ultrasparc” for optimized 32-bit binaries, and “-O3 -mcpu=ultrasparc -m64” for optimized 64-bit binaries. For Sun’s compiler, I used “-fast -xarch=v8plusa” for optimized 32-bit binaries, and “-fast -xarch=v9a” for optimized 64-bit binaries.
GNU gzip 1.2.4a
I used the latest version from GNU’s ftp site, and compiled in the 32-bit and 64-bit variations with the various compilers. The gzip and gunzip operations were run 3 times each, and the results averaged.
In this test, Sun’s c compiler created a GNU gzip binary that zipped up the 624 MB file about 11% faster than GCC did. Surprisingly, GCC 2.95 produced a gzip binary slightly (2%) faster than GCC 3.3.2. The unzip operation, GCC 2.95 won at 89 seconds, with GCC 3.3.2 coming in at 90, followed by Sun’s compiler at 91 seconds.
In the 64-bit run, Sun’s compiler produced the faster code.
As with the 32-bit tests, the gunzip operation produced very little delta between the three binaries.
After seeing my 64-bit versus 32-bit article, Shane Pearson wrote me posing the question whether or not disk I/O was a bottleneck in the gunzip operations, thus causing the results to come out essentially the same. After I ran some tests, it turns out he my very well be right. The 624 MB file is written in 91 seconds, which is about 6.9 MB/s, and then you include reading the 126 MB gzip file in, for a total of around 8.4 MB/s, read and write. This is pretty close to the effective maximum of the system, which for a 100 MB dd if=/dev/zero test and 100 MB mkfile test, comes to about 10 MB/s, with no other CPU operations going on.
And here is a graph showing how they all stacked up:
Sun’s compiler was the clear winner. Surprisingly, the older version of GNU’s GCC beat 3.3.2 by a very slim margin.
For the OpenSSL 0.9.7c tests, I used the static builds of the binaries (static as in no separate libssl.s0 or libcrypt.so). I ran openssl speed rsa dsa for each of the 5 binaries (3 32-bit and 2 64-bit) 3 times, and averaged the results.
The differences were not nearly as profound with OpenSSL as with the other tests. Generally, Sun’s compiler did slightly better than the GCCs. The two GCCs were about evenly matched. Several of the operations showed little or no difference between either three.
There was much more significant performance difference when comparing 64-bit binaries created by the GCC and Sun compilers. For 64-bit code, Sun was the clear winner.
These graphs show how all 5 binaries stacked up against each other.
MySQL 4.0.17 has both C code and C++ code, and I compiled it with the three compilers in question. Getting 4.0.17 to compile under GCC 2.95.3 took the most finagling, but I was eventually able to sort it out. Because the tests take around 3 hours to run, I ran each iteration twice, and used the best run. There was very little disparity between the runs.
For the 32-bit runs, Sun’s compiler actually got beat by one or even both GCCs for many of the operations. In general, the results were quite sporadic between the 3 compilers.
As with the previous article, the 64-bit MySQL server binary was used, but the 32-bit Perl and MySQL client libraries were used to run the client end of the benchmarks in order to keep the client end of the tests consistent.
Sun’s compilers produced a binary that beat GCC in all but the select operation.
Here’s a view with all the compilers accounted for.
The results were all over the map for these runs. I went back to double check and make sure sql-bench wasn’t producing sporadic results (which took quite a bit of time), but the runs were consistent.
The biggest surprise though was that Sun’s 64-bit compiler created a binary that for most operations beat out all the others, whether they be 32-bit or 64-bit.
These were a limited set of tests on a limited number of applications run in limited ways, and I only tested the C compiler (and for MySQL, the C++ compiler).
Still, we can draw a few interesting generalizations. First, Sun does indeed appear to produce faster code, although not always, and the difference in speed varied quite a bit, and toped out at about 22% faster.
The greatest differences were seen comparing 64-bit binaries between GCC and Sun’s compiler, with Sun’s compiler having a greater advantage. Another surprise was that GCC 2.95 tended to produce slightly faster code compared to 3.3.2, with a difference in the 2% range and not much more.
The difference in speed varied greatly throughout the tests and specific operations, so it all depends on the application as well as the particular function it performs.
As with the 32-bit versus 64-bit tests, it all depends on your application and what you’re doing with it. If you’ve got an application that you really need optimized, trying with the three compilers listed here may be worth your time, especially since you can use Sun’s compiler free in an evaluation mode.
Seeing that GCC performance isn’t all that far off from Sun’s compiler suite (and in a few operations, bests Sun), says quite a bit about the quality that GCC has been able to achieve, especially considering the magnitude of platforms that GCC operates (beautifully) on.