“Sun today announced new benchmark results involving the performance of the open source MySQL database running online transaction processing workload on 8-way Sun Fire V40z servers. The testing, which measured the performance of both read/write and read-only operations, showed that MySQL 5.0.18 running on Solaris 10 executed the same functions up to 64 percent faster in read/write mode and up to 91 percent faster in read-only mode than when it ran on the Red Hat Enterprise Linux 4 Advanced Server Edition OS.” Take a look at the item below, though.
Which item below? Am I being thick or is there no obvious item below? Nothing strikes me in the article and the PDF of the results isn’t much more enlightening. Is it something in the MySQL options they used?
Colour me confused. Anyone know?
Edited 2006-05-25 16:23
The news item on OSNews.com below this one.
http://osnews.com/comment.php?news_id=14710
Hardware made by a company runs faster on OS made by the same company.
More news at 11!
Yeah, thats kind of the point as well. From Sun you can
get the hardware and the OS – see the advantage yet?
From Sun you can get the hardware and the OS
Thats what people love about Apple, yetwhen it comes to Sun people tend to dismiss it. Interesting.
The AMD cpu’s, Seagate harddrives, and Tyan motherboards than Sun sells are Solaris optimized?
Possibly, but surely not to the tune of a 91% difference.
> and Tyan motherboards than Sun
I don’t think its a Tyan mobo. The v40z is a Newisys 4300 with a different bezel and modified firmware on the system manager board (so far as I know, anyway).
It would be surprising if Sun and MySQL were not able to do this, geven that they both control their codebases, both have dedicated paid R&D, and both have something to gain from this.
The x2100, and other in that family, are based on modified Tyan motherboards. You are probably right about the 20/40Z though.
Actually you’re wrong. The Sun 40z and 20z are a rebranded system. The Tseriers are the first inhouse Sun Opteron boxes. You can buy the same v40z for Western Scientific. The systems are just a quad Opteron dual core boxes.
Personally I could not care about performance of toy DBs like MySQL. If they started comparing PostgreSQL or Oracle, I would be interested.
Why does Sun even bother releasing benchmarks? We all know the results are biased. Personally I don’t trust any study that’s overseen by the same company.
Yes, because benchmarks claiming that linux is faster than any OS made by linux users aren’t biased …
No, this topic is not about benchmarks done by “linux users”.
No, but if you apply the same logic that companies are biased about their benchmarks, why shouldn’t Linux users be biased when running benchmarks?
Because most linux users aren’t in it for the money, but for the pride. Therefore fakish tests are close to non-existent.
However, companies like Redhat, Novell and IBM are alike to Sun in this regard. I’d expect nothing else, considering their need for a steady income.
Pride and other emotional attributes have no value either in benchmarking. Benchmarking its self is an interesting yet rigorous *scientific* approach.
Benchmarking its self is an interesting yet rigorous *scientific* approach.
Yes, and there is a correct way to do things and an incorrect way to do things.
Pride and other emotional attributes will lead to doing benchmark the “scientific” way, while the lack of these will lead to b0rked benchmarks.
There is in science something called “Good Laboratory Practices”. Marketing Departments have a poor understanding of “Good Laboratory Practices”.
”
Pride and other emotional attributes will lead to doing benchmark the “scientific” way, while the lack of these will lead to b0rked benchmarks. ”
Bzzt… Wrong!
You need to understand what the Scientific Method is:
“Among other facets shared by the various fields of inquiry is the conviction that the process must be objective so that the scientist does not bias the interpretation of the results or change the results outright.”
http://en.wikipedia.org/wiki/Scientific_method
Yes, and securing the process is objective requires high ethical standards.
Lack of those will lead to lack of objective processes, and lead to “Marketing Department”-Science.
If you want scientific methods then get away from the money. Only in that way can you avoid bias. A multibillion company in fierce competition does not use objective processes.
Your quote:
“Pride and other emotional attributes will lead to doing benchmark the “scientific” way”
My response:
” process must be objective so that the scientist does not bias the interpretation of the results or change the results outright”
Any yes that excludes other emotional attibutes including greed, pride, etc. Arguing the virtues of contaminating the Scientic Process with one emotional trait over another is irrelivant.
Well, I wouldn’t be surprised that Solaris can beat Linux.
But for su much margin? Considering that mysql has always been biased and optimized toward Linux because it has been the main OS for mysql for a while and that we’ve seen mysql benchmarks where linux was beating solaris?
Not that it can’t be possible (ZFS could be the reason why solaris is winning here) but it just look worth of taking time to analyze it carefully.
I don’t think Solaris 10 has ZFS yet. So no ZFS had nothing to do with it.
> Why does Sun even bother releasing benchmarks? We all know
> the results are biased. Personally I don’t trust any study that’s
> overseen by the same company
Haha… Except you Linux zealots wouldn’t trust God himself if he told you that Solaris benchmarked faster than Linux because you are all so blinded by your platform loyalty.
I would be surprised if the combination of Solaris and ZFS didn’t beat Linux and Ext3 (even with BTrees enabled). It’s also worth remembering that the Redhat ES kernel is 2.6.9, which is quite old at this stage.
Where did you see that they were using ZFS? I looked for filesystem details, but couldn’t find them.
I should know better than to waste my time reading a press release benchmark too closely anyways 🙂
Unfortunately there is no information as to how the system was set up. It would have been nice to know more about how the operating systems were installed, filesystems were layed out and whether or not volume management was used.
Since a V40z can hold up to 6 drives, was the test conducted on a system with with 1 disk or more?
Yes but Sun do it for money and news, Linux does it for braggin rights. :p
There’s nothing wrong with either motive.
Other than both produce biased results? I see plenty wrong with that.
Bias will always be unavoidable. Question is, does that bias distort the results?
It’s almost always possible to produce a benchmark showing that system A runs faster than system B, and to produce another benchmark showing that system B runs faster than system A. You just find the differences, and design the benchmark to exploit the differences.
Now, it is possible that Solaris engineers studied MySQL performance, identified bottlenecks, fixed a few, and the result is that for the moment Solaris runs MySQL faster. If so, it won’t last long; Linux kernel developers will put the same tricks into Linux and everyone wins.
Like this:
http://developers.sun.com/solaris/articles/mysql_perf_tune.html
You lost.
Seriously, Solaris is not something that was slapped together in a garage. It’s always been superior to all free *nix variants when dealing with multiple processors and threaded apps.
It’s always been superior to all free *nix variants when dealing with multiple processors and threaded apps
You know, Linux 2.6 is also know to be SMP and multithread friendly – if you follow the linux kernel mailing list you’ll know that there’s has been *lots* of work in making Linux work well in platforms even bigger than what Sun offers, like 512-CPU SGI’s beasts.
Is not that Solaris can’t win the test because it’s not scalable. If you look at the numbers, Solaris beats Linux in those benchmarks even when running the *single* thread test. Hence, it’s not so much about SMP scalability or multithreading, IMO. If it falls behind with 1 thread, is not surprising that it falls even more with 16. My bet is that it’s ZFS which makes the difference. We know that ZFS rocks, but that doesn’t makes the rest of the linux kernel any worse. It’d be interesting to see solaris-UFS and linux-XFS just for curiosity.
You do know the difference Sun’s SMP and that offered by SGI– NUMA. Apple and Oranges. The 512 CPU are cluster on many small systems.
“You do know the difference Sun’s SMP and that offered by SGI– NUMA. Apple and Oranges. The 512 CPU are cluster on many small systems.”
FYI, an SMP Opteron box is also NUMA, albeit on a smaller scale than an SGI Altix (Sun also sells NUMA boxes, AFAIK).
The 512-CPU Altix isn’t a cluster, it runs a *single* Linux image (when you run “top” you see 512 CPUs – that’s why recent versions of top default to showing only aggregate CPU statistics). It’s made up of nodes, each with a couple of CPUs, but those nodes aren’t independent, they make up a single, unified machine.
You do have Altix clusters (with “Columbia”, at NASA’s NAS, for instance), but those are clusters of several 256/512-CPU Altixes (which, like I said, are each a single machine).
Apples and Oranges? I don’t think so.
”
The 512-CPU Altix isn’t a cluster, it runs a *single* Linux image (when you run “top” you see 512 CPUs – that’s why recent versions of top default to showing only aggregate CPU statistics). It’s made up of nodes, each with a couple of CPUs, but those nodes aren’t independent, they make up a single, unified machine. ”
And the projection of a single contiguous memory structure between node is what NUMA provides. And your point is? The fact that the systems communicate on a high speed interconnect is different the the Sun Fire 6800 which I admin. In fact NUMA patches are required and possess different locking primative that the localized SMP model– the big difference between NUMA and the large Sun Fire models is latency and thus the model is different.
Just try to run the Linux kernel on the 6800. You start seeing locking contentions after 12 CPUs and the scalability leveling off.
Don’t waste your time, this is the standard Linux troll argument about the “superior” scalability of Linux over Solaris. And let’s not confuse the issue with facts!
Indeed. I noticed that there are similarities between todays Linux advocate and Win NT advocates in the 90s. For the most part knowledge and mindless advocacy is inversely proportional. I agree, Linux Advocates should be procmailed to /dev/null.
Solaris, Linux, FreeBSD, OpenBSD,etc each offer advantages within specific areas.
Edited 2006-05-26 17:51
Exactly. I found that the more technical you get, the more likely you are to have insults hurled at you because they can no longer discuss the topic because you are “over their head”.
“My bet is that it’s ZFS which makes the difference. We know that ZFS rocks, but that doesn’t makes the rest of the linux kernel any worse. It’d be interesting to see solaris-UFS and linux-XFS just for curiosity.”
Not likely that ZFS is the difference. The performance was run on Solaris 10, which does not yet include ZFS. However, ZFS should first appear in Solaris 10 in the upcoming weeks in Solaris 10 Update 2 (s10u2).
Well, it’s sun who is doing the benchmark – not that they need to wait for a release…
The article states that Sun worked with the MySQL team to optimize their systems for MySQL performance, so I’m not shocked that Solaris outran the RHES system. I am surprised, however, by *how large* the performance gap was. I’d like to see third-party benchmarks, of course, and it would also be nice if the MySQL guys would share what made the Solaris system so speedy with the rest of the *nix world. I wonder if their optimizations also had an effect on PostGreSQL performance as well?
just test ignore
Edited 2006-05-25 17:51
“Sun today announced”
Today? LOL. Actually on Friday April 21, 8:05 am ET
Old news, Nothing to see here, move along.
http://hup.hu/node/25441
(english article)
Which release of Solaris 10 was used because there are some issues using 3/05 with the T1000/T2000:
http://docs.sun.com/source/819-2544-15/index.html
ZFS is not in Solaris 10. It will be in Solaris 11, if it’s stable enough.
ZFS should ship in Solaris 10 Release 6/06.
Here’s my take on the this benchmark:
ZFS was not used because ZFS it not available for Solaris 10, only Solaris 11.
The loads used were very large, and this was a massively parallel implementation. Sun used DTrace to optimize MySQL, saw a reasonable 10 – 15% performance increase (or something verabouts, I’m guessing here) over single server performance against Red Hat. They added more load and more servers in parallel to emphasize the performance benefit.
Yes, Solaris is faster than Red Hat under high loads. Yes, Solaris scales better to many processors. These claims have been made for some time, now here is the proof. Folks running single CPU servers under light loads may not see much performance difference between the two. But yes, Solaris scale up very, very well.
Sun is currently adding Solaris optimizations to PostGres as well. Looking forward to those numbers.
I don’t think we will see any benchmarks of Oracle on Solaris Vs Red Hat. Oracle forbids publishing bechmark numbers in their licensing.
I have only seen one 3rd party Oracle benchmark and this was done by PC Magazine some years ago. They decided to break the license agreement, and good for them.
RedHat Enterprise 4 is now two years old in every sense of the word. Why not go back and compare against Solaris 9… FYI.
Edited 2006-05-25 19:24
Ah because RedHat Enterprise 5 is not out. So we should hold back on one shipping version because the other vendor have chosen not to ship within the last 2 years?????? Grasping for straws.
RedHat Enterprise 4 is now two years old in every sense of the word. Why not go back and compare against Solaris 9… FYI.
FYI, Solaris 10 is one and a half years old. RHEL 4 is at update 3 which includes patches and improvements. Both of them are current shipping version, it is perfectly fair.
Anybody remember when Sun got caught when it claimed that its Java VM ran an order of magnitude faster than any of the other existing JVMs? It turned out that Sun’s JVM contained logic that looked for a specific sequence of bytecodes matching the benchmark algorithm. When it detected the sequence, it ran an optimized piece of assembly code and, not surprisingly, none of the other JVMs were tuned in this way. However, Sun’s JVM was pretty much identical (or worse) in performance when you ran specific cases that it wasn’t tuned to handle.
In order to be credible, benchmarks need to be practical (ie. apply to scenarios performed in the real world), open to review, and orchestrated by an organization/individuals with no financial incentive in the outcome. Note: I’m not criticizing Sun’s results here. It could very well be that its code is faster than competitors; however, one should take the results with a SERIOUS GRAIN OF SALT.
>Ah because RedHat Enterprise 5 is not out. So we
>should hold back on one shipping version because the
>other vendor have chosen not to ship within the last
>2 years?????? Grasping for straws.
How about not testing against an OS that hasn’t been updated within 2006?
And, uh… Solaris 10 was officially released in mid-2005, if memory serves.
Maybe Red Hat will see the value in updating their software to something more modern… and maybe I’ll give a damn about them again if they do.
What you used RH E4? It has gone though 4 updates including kernel patches and changes. Yet that’s not even the point.
If we were to use your “argument” We should not be testing Red Hat E4 against Windows Servre 2003 since RH4 was initially relased on 15 February 2005 while Windows 2003 Server was release almost 2 years ealier (April 2003).
”
And, uh… Solaris 10 was officially released in mid-2005, if memory serves. “– Try Jan. 31, 2005
Looking at the release dates beteen RH E4 and Sol 10,
your release date argument is non-existent 1.5 months difference?
Your grasping.
take solaris 10 running on a sparc (say a v240), and linux running on a dell (a poweredge, or even a higher end optiplex for instance). compile binutils on both using about the same version of gcc. time it.
in my case, about 4-5 hours on solaris. roughly 5 minutes on the dell…
I’m sorry, but several times where solaris is just getting through the configure script on something, I’ve seen linux done with the entire build of a collection when run in parallel next to each other.
it’s not that solaris is such a bad system, but speed is not one of the traits it’s renowned for…
haha…
Now try it on a comparable hardware box… Like Solaris 10 running on an Ultra 20. And watch Solaris blow away Linux on the Dell.
so, now you’re claiming that solaris x86 (on a freaking low end workstation…) blows away solaris sparc on a proper server? wow, who’s the zealot here?
> so, now you’re claiming that solaris x86 (on a freaking low
> end workstation…)
I’m claiming that Solaris x86 on Opteron blows away Sparc, yes. The Sparc has fallen behind in recent times.
And dkeep in mind that if you were compiling on a dedicated server, that server wwas probably balencing a bunch of other tasks at the same time. hardly a fair test.
take solaris 10 running on a sparc (say a v240), and linux running on a dell (a poweredge, or even a higher end optiplex for instance). compile binutils on both using about the same version of gcc. time it.
in my case, about 4-5 hours on solaris. roughly 5 minutes on the dell…
That is rubbish. Are you cross compiling x86 binaries on SPARC? Compiling is the worst possible benchmark one can come up with ecspecially on two different ISAs.
um no, I’m compiling binutils for both platforms. it isn’t limited to x86 in case you didn’t know…
that’s just one example however (though admittedly one of the worse). I’ve seen this disparity all over the place in preparing packages for both platforms. and why is compiling such a bad benchmark, it is a real world task that I have to do on both. one’s slow, one isn’t…
Linux zealots has 80/20 rule:
80% time is spent for compiling, 20% for running the program.
I’d agree that the time it takes to compile something is a horrible benchmark, especially with gcc. When I had an Alpha, it took a while to compile things. I read somewhere online that the register allocation phase of the compilation runs in O(n^2) with the number of registers to fill. With RISC processors having many times the number of registers as x86, it will take quite a bit longer for that step.
Maybe gcc uses an improved algorithm for amd64 and x86, since they’re common, but for SPARC, it’s possible the developers just try to generate code that is correct and have compile speed as a lesser concern. There’s probably a dozen other places where the differences in architecture and gcc developer time makes a compiling benchmark suspect.
I have nothing against Solaris, but it always seems to me a useless debate when it comes to compare performance, scalability and stability. We can all argue this and that, but in the end the facts are:
Among the top 500 supercomputers in the world, which cost millions of $$, which have the very best engineers to look after them, which run performance and stability critical tasks, Linux has about 78% of the market share. Solaris has 0.8%.
Do you really think it’s just to save a few bucks in the OS? Oh wait, isn’t Solaris even free now too?
Take a look at top500.org for the numbers.
I think you need to look at pat9912’s comment about Solaris and SGI and the difference between Symmetric Multiprocessing (SMP) and Massively Parallel Processing (MPP):
http://en.wikipedia.org/wiki/Symmetric_multiprocessing
http://en.wikipedia.org/wiki/Massively_Parallel_Processing
Solaris can compete in that space if someone chooses to use it in that role. In fact 4 of the entries in the Top 500 are Sun machines (141, 327, 369, 370).
> Linux has about 78% of the market share. Solaris has 0.8%.
A misleading statistic at best. Most of those supercomputers are running highly highly hacked special purpose versions of the Linux kernel that are not even usable for general tasks outside of their specific task they carry out.
if i follow Anandtech and the way that they used to understand their MYSQL benchmaks between OS X and Linux, i sould say now that Linux is not a good server OS, its performance are miserable compared to Solaris, the Linux kernel has design flaws, dam!!! the Linux kernel is a micro kernel, Linux is slow to create threads, there are whatever wrappers wrapping something that make Linux slow, etc… Given that MySQL runs so much better on Solaris than on Linux, it has to be that Linux is a poor OS for servers, right?
Linux zealots will say that NO!!!! those results are bullshit, that’s crap benchmarking, but as Linux zealots have said bout the Anandtech benchs, i should reply that No!!!!! that’s dam good results, well done, nothing to complain about, it’s like this you can*t change it, Linux loses miserably, right?……
Or maybe we should be smarter than Anandtech and take care about what’s exactly behind those numbers!!!!
But it is intersting to see that the people who defend Linux against Solaris regarding to its performance on MYSQL are now in the same situation than the people who were defending OS X against Linux on the same subject: MYSQL performance. And probably amoung the people who were bashing OS X performance on servers, the same people are now trying to defend Linux on the topic: MYSQL performance.
Actually the difference between MySQL on Linux and on Solaris isn’t that big for one machine.
It’s when using several machines in parallel that it starts to show.
What we can conclude is that a Sun-optimized MySQL runs faster on a Solaris-system than on a RedHat system. Hardly a surprise. Most of the talk about 91% and stuff like that, is merely Marketing Speech.
Talking about design flaws because of approx. 15% difference in performance is a bit exaggerated, and not even close to a miserable performance by Linux. And poor performance in Mac OS X in regard to Apache and MySQL is a known issue, no matter what one may think of the blunders in Anandtech’s benchmarks.
The problem with Mac OS X was that the same machine performed 2-5 times better (that’s 100 – 400% better) with linux than with Mac OS X Server.
Hakime. You’d do wisely in not behaving like a moron. Get rid of those !!!!!! all over your posts. It makes you look like 14-year old pimplefaced boy.
ZFS will be officialy released in the soon to available 6/06 release of Solaris 10. (and offcourse is already available in Solaris Express right now)
“MySQL and Sun have worked to optimize the performance of MySQL Network software certified for the Solaris 10 OS,”
Wow. A test conducted by Sun, for Sun, and they’ve worked to optimise MySQL and they get some speed increases? Incredible. Did they tune MySQL on Linux in the same way, or in a way that would increase its performance? What happens when you take an unmodified Solaris 10 and an unmodified MySQL installation, compiled from source with nothing additional, or otherwise? The PDF indicates none of this. I get the feeling they maybe didn’t like the results from that, otherwise they wouldn’t have thrown the marketing soundbite in on ‘Sun and MySQL working together for your benefit!’
I think this shows how flawed a proprietary OS is (and Solaris is still very much proprietary as a whole). If you were to produce benchmarks like this for a Linux distribution, people will almost immediately know what you’ve done, whether you’ve tuned both database systems equally, whether you’ve been using a different compiler and whether your results are totally flawed or not. It’s just that much more difficult to get away with.
Edited 2006-05-26 10:41
And this is different from the pdf that RedHat distributes to customers showing RHEL wiping the floor with Solaris 10 on a V20z running certain benchmark tests how? I would link the pdf file but it is marked “Red Hat Confidential”, but let me summarize it. A 21 page document that has three pages of setup infomation (as equally minimal as the Sun documentation) and 18 pages of “results”.
What I think it shows is the flawed logic of two vendors who are interested in getting customers to buy their products and services that use benchmarks and limited information to show the benefits of their products.
And if someone benchmarked MySQL on Linux how would they know what was done? I could very easily produce a set of benchmark results and say nothing about what I did or how to produce them, so how would you figure out what I did? This is the same argument I used against Tony Bourke’s MySQL benchmark article and was essentially flamed out of existence by Linux zealots.
If people really want “honest” benchmark results, then everybody needs to raise the believability flag everytime a set of benchmark results is published that does not contain information necessary for a third party to conduct the same tests and show the methodology in full detail. Until that happens, companies and individuals will publish whatever benchmarks they want to promote whatever product or agenda they want.
And this is different from the pdf that RedHat distributes to customers showing RHEL wiping the floor with Solaris 10 on a V20z running certain benchmark tests how?
It’s surprising they give that to customers, but I suppose any relevance it has depends on what Solaris they were comparing to, what Red Hat had specifically done with MySQL and their Linux distro in the results they give or whether it is an ‘as shipped’ set up.
This is the same argument I used against Tony Bourke’s MySQL benchmark article and was essentially flamed out of existence by Linux zealots.
Tony Bourke’s set of benchmarks was far more complete and explanatory than this joke that’s come out of Sun. At least he went some way to trying to work out what had happened in some cases. If that’s the argument you used I’m not surprised you got flamed.
Solaris 10 came out of that set of benchmarks quite respectably, as did Linux, but it showed absolutely nowhere near (nor gave any indication whatsoever) what Sun is trying to show here.
If people really want “honest” benchmark results, then everybody needs to raise the believability flag everytime a set of benchmark results is published that does not contain information necessary for a third party to conduct the same tests and show the methodology in full detail.
Yep. I suppose the problem with that is it takes a great deal of time and effort to produce results, how you did it and a way to reproduce exactly what you got. It would take you many weeks and months to put together.
I questioned a number of things starting with his methodology of “stock with default options” installation method, what the Hell does that mean? Comparing two operating systems in Beta develpment against a “production ready” OS that is compiled in its entirety with no mention of compiler flags. And to top it off, use a filesystem that is recommended by MySQL (ReiserFS), while it is an option it certainly isn’t the default! Explain to us just how this is better than Sun’s offering? And lets not forget the classic line “I’m going to teach you how to benchmark your system using MySQL” and forget to mention that when you compile MySQL you have to use “–enable-thread-safe-client” or sysbench will not work.
Actually no, Solaris did not come out that well, Tony made the claim that in the 10 million row test that he had to use forcedirectio in order to get better results. In all of the MySQL testing I have done, the only time I even came close to approaching Tony’s “problem” results is when I tried to duplicate the 10 million row test at home as he did it. The /usr partition was close to full when I ran the test, no wonder the results sucked. And this is using the same “methodology” as Tony, the closest he came to mentioning a problem is when he said that some tests took almost an hour to complete. If nothing else a person reading the article not familiar with Solaris would come away with the idea that MySQL performs poorly unless you use forcedirectio. But then most people don’t install a database on a 7 GB /usr partition either.
No I got flamed because I had the audacity to question a questionable benchmark article and despite numerous attempts to point out various problems with the article and its results, I was shouted down because “Linux won”, what bullshit.
Actually it would not take all that much more time to spell out what was done and why, but people would have to demand it. Instead, the vast majority of readers don’t question anything and just blithely agree with almost anything tossed in front of them.