Some Sun Engineers have started blogging about how Sun does performance work on Solaris during development. They mention that they felt they should start talking about it after Linus Torvalds said more regular performance testing of Linux would be usefull. The posts are here and here.
Suns Internal Performance Testing
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
Just wanted to say very cool, good to hear about all the hard work going at Sun and there development of Solaris.
It would be good to get some more blog’s or articles on how other operating systems get tested. Interesting stuff 🙂
Linux users are always saying that linux become more and more faster, but linus is saying the opposite.
What the … ?
Sun find some bench that shows Linux 2.6 slower than Linux 2.4. Big news.
Some benchs :
http://bulk.fefe.de/scalability/
http://developer.osdl.org/craiger/hackbench/
http://www-106.ibm.com/developerworks/linux/library/l-web26/
http://www.2cpu.com/articles/98_1.html
http://kerneltrap.org/node/903
Did you actually read the links? No where in either blog posting do the sun engineers say anything about the performance of Linux. They both link to an article where Linus comments on feedback about the performance of 2.6 versus 2.4 which was supplied by Intel.
Stop making up FUD when no FUD exists.
The more I give Sol10 a chance and the more I ‘play’ with it, the more I like it. For those looking for a little more propriety and corporation backed OS with a long history, look no further. Solaris finally has the product to be “cool” using.
Still has a little work (well maybe a lot..) on the desktop front to compete with some of the more desktop oriented linuxes for home use (JDS promising for corporate desktop), but one thing at a time. Take care of the hardcore bigiron/serving/performance markets first and hopefully opensolaris will take care of and gain momentum on the desktop side.
Good time to be a Solaris fan. It also only will get better. Latest Solaris Express already ‘feels’ faster as a desktop system to me than the release version.
Now who do I talk to about bug submissions and feedback without paying? This is the one thing that baffles me in that Sun charges to be a fan and a willing tester of their OS. Seems against all good sense to me for a software developer to do and even more so an OS developer.
desktop
http://shots.osdir.com/slideshows/slideshow.php?release=279&slide=3…
installer
http://shots.osdir.com/slideshows/slideshow.php?release=279&slide=1
I find this rather interesting that Linus wants daily benchmark testing to determine potential performance improvements to the Linux kernel. Unlike the Sun articles I just read, the Linux kernel developers have to establish a baseline for the performance of the kernel before they can start benchmarking it. To do this correctly could take months away from actual development time. And they would have to do this with every version of the kernel to derive maximum benefits.
A better option would be to create a separate group to benchmark the kernel, create the baseline, then test the kernel and analyze the results. This would also slow down kernel development considerably and I wonder how the “community” would deal with that?
You can post your concerns on the newsgroups comp.unix.solaris and alt.solaris.x86. Several Sun employees monitor those groups regularly for issues with Solaris.
This is why I have so much faith in Sun products. Whether it is Solaris, Java or Studio they are methodical in their testing, the end result being I the end user get one hell of a product. What should be interesting is how the external developer community is able to interface with Sun on OpenSolaris whilst still maintaining this level of regression testing and benchmarking.
This healthy competition between Sun and Linux benefits us all.
Linux users are always saying that linux become more and more faster, but linus is saying the opposite.
What the … ?
There have been some degradations in I/O performance between 2.4 and 2.6. Although in spite of what Linus says, there have been many benchmarks proving the opposite — that the 2.6 series are in generally faster than 2.4 series.
The specific benchmark Linus was commenting about is a comparison of a heavily modified Red Hat 2.4 kernel, containing many backported 2.6 enhancements, versus a vanilla 2.6 kernel.
So unilaterally saying 2.4 is faster than 2.6 is misleading and untrue.
Benchmarks are done before each kernel release, this is usually done by OSDL, atleast 2,500 regression and stability tests are performed before a release.
Regardless of what Linus says 2.4 and 2.6 are both faster than Solaris 10, atleast according to this benchmark.
http://software.newsforge.com/article.pl?sid=04/12/27/1243207
The “benchmark conparison” you specified is invalid since Tony chose to test a Beta Build of Solaris 10 against a “production” build of Gentoo Linux. If you are going to test something, then the playing field should be equal, and using Gentoo does not make the field equal. It stacks the deck against any other OS in the test. If I was performing the test I would use a distro of Linux that I did not have to compile.
@Adam
Regardless of what Linus says 2.4 and 2.6 are both faster than Solaris 10, atleast according to this benchmark.
Did you even read that benchmark?
1) There are several cases where Solaris does win
2) That was a beta version of Solaris 10 from *last year* with debugging enabled most likely, I’d prefer to see it on the shipping version…
I did read the benchmark, and no where does it say Solaris 10 wins any of the tests.
Also you have to consider this Gentoo system was not compiled with ICC, although rarely done, significant performance increases would occur. It is the best compiler in the area of performance for Linux systems. If it were it would be much faster. Solaris 10 has the advantage of being compiled with Sun’s own compiler.
Also when Tony did the test, you realize that 2.6 was not the default kernel for Gentoo — so the similar arguement also applies for the Gentoo 2.6 benchmark as with the Solaris 10.
OSDL could take the highest score ever recorded as the baseline (like Sun does) if they wanted to; this doesn’t sound difficult.
Likewise the whole point of a separate benchmark group is that they don’t slow down development. There would be a bottleneck if all patches had to be benchmarked before Linus accepted them, but he’s not that disciplined.
While I’d like to see a pre-compiled version of Linux in the tests, I don’t see anything wrong with including Gentoo – the point is to get the best performance possible from Linux and the best possible from Solaris, then compare them.
1) There are several cases where Solaris does win
Really? I looked it over and don’t see a single Solaris win, except for a tie with 2.6 on scalability.
2) That was a beta version of Solaris 10 from *last year* with debugging enabled most likely, I’d prefer to see it on the shipping version…
I’d like to see a shipping version as well, but I’d be shocked if that version had debugging enabled. There’s an entire section of the article devoted to how he improved performance within Solaris, and I’m sure he would have mentioned if it was debugging. He even tried talking to employees at Sun, so it’s not like he wouldn’t have known.
Tony chose to do the benchmark on build 69 of Solaris 10, Instead of the then released build 72. According to a Sun engineer build 72 had performance enhancements in areas that would benefit the MySQL benchmarks.
I wouldn’t lay too much emphasis on the MySQL benchmarks as they don’t represent an adequate real world customer environment.
Most corporations would never use Gentoo and None of them will ever use build 69 of Solaris
I don’t know what your background is, but mine prior to getting into IT was photographic sciences. When you conduct a test between two or more samples, the idea is to make the tests the same. By compiling Gentoo (regardless of compiler used) the test was invalidated because Tony compiled Gentoo and not the others.
His “performance tweaks” for Solaris was not the only options he could have used. He could have also set maxphys which sets the maximum data transfer rate of the hard disk (which defaults at 1,048,576 bytes for the sd driver):
http://docs.sun.com/app/docs/doc/817-0404/6mg74vs9m?a=view
With the amount of concern from the BSD side of the house as well, most people have dismissed the tests as biased. Comments by various BSD users, both on SourceForge and Slashdot point out there were options Tony could have used to improve BSD’s results.
And he could have collected sar data on all of the OS’s he tested to see if there were problem areas where additional tweaking could have improved performance and reported that information as well.
The issue was raised when Intel employee Kenneth Chen announced some performance figures for various versions of the 2.6 kernel. The tests found that versions 2.6.11, 2.6.9, 2.6.8 and 2.6.2 of the kernel performed 13, 6, 23 and 1 percent slower respectively than the Red Hat Enterprise Linux 3 baseline–which runs on version 2.4 of the kernel, with some added features from version 2.6.
What’s the big deal?!?
Linus what’s the performance weekly or possibly daily, so what?
It isn’t hard to believe that security/patches could have a performance impact.
i.e. WinXP sp2.
That is not necessarily correct. I sat in a “30,000 ft.” overview of SELinux given by RedHat’s lead developer and he said there were features of SELinux the NSA wanted to implement that RedHat had issues with. Although the developer was not specific, he did state that RedHat was not comfortable with the performance hit RHEL would take if these features were implemented. I would like to know what those features are!
ASDF, It is not a big deal per se, but it is the implication that Linux is not benchmarked prior to release — in reality it is. In a matter of fact OSDL does nightly kernel builds.
I do not know how the test is invalidated because of compiling Gentoo, as all the systems there were compiled at one point. I too though am curious of how a pre-compiled distribution would fair, I am almost sure that performance would not degredate to change the results.
I don’t think the version of Solaris he was using was compiled with gcc, which has noticably worse performance than Sun’s own Studio 10 compiler thingy — so that is another inconsitancy.
There are many options in Linux for enhancing performance, not even a small amount of them were explored. If anything the test was biased towards Linux.
I do not know any knowledgeable BSD person who honestly thinks the open-source BSDs can perform as well as GNU/Linux and Solaris on SMP computers.
I seem to remember there was a fair bit of tension between Linus and the SELinux guys regarding the SELinux hooks in the kernel. I am sure there is some sort of informal benchmarking carried out by Linus, I don’t really think the point here is whether there were issues between 2.4 & 2.6, but having a formalised benchmarking process in place.
Having run both RHEL3 and 4 against S10, I’ve found S10 to be by far more performant, and esp. more stable. I’ve seen consistent RH hangs and core dumps – and not once been able (even after all kinds of bad behavior) to get S10 to fall over… gotta give Sun credit. We may not abandon RH yet, but we’re seriously looking at options…
Adam and Robert: this was a database test that was nearly completely I/O bound. Compiler options should have a minscule on the result.
But perhaps, controllers drivers could have a big impact on performance?
Seriously, I would like to see a test where the kernel performance could be a bit more isolated from the others components. If you look at Linux kernel change logs you will see that Intel engineers do work on drivers for their hardware (as, I suppose, do engineers from others chipset builders), and don´t know if they do it on the others projects too (or if developers of the others projects borrow some technique from them).
I know that isn´t possible to isolate totally the drivers from the kernel (as it create the needed infrastructure and shapes the intercommunication layer) but should be great if they could agree in a common driver communication layer so that drivers could run unchanged (or at most, almost unchanged) on Linux, *BSD, Darwin and Solaris although, even the release license would be a huge problem.
This could give a boost on the development of alternative systems (or could feed up a tremendous duplicated effort waste, consuming all human resources and on end cause the extinction of our modern society).
Anyway, would give a better comparison ground.
Of course, as always, I can be also tottaly wrong.
I agree with you. Recently I have moved my 4 Linux systems into a Solaris 10 system running zones. I have had no crashes at all, and everything is running great.
The reason I moved from Linux was because the systems would crash after running for a month or so. I could ping the system, and get to a login prompt. When I type root, and hit enter, it just hung and would nenver get the password prompt. I reboot and nothing in the error logs. Most of the hardware was new, and the way it crashed pointed to the OS dying on me.
I thought about moving to another dist, but instead of playing that game, just went to Solaris 10. I have no regrets, runs wonderful.
Undoubtedly, the test could have been improved. Your point about the drivers was particularly good. I was just pointing out that Gentoo vs RedHat really has little to do with it. If anything, it’s a disadvantage to Linux, because RedHat ships particularly good kernels.
The regression has already been found and fixed. It is a simple scheduler parameter that was set incorrectly for the itanium’s large cache.
And in the process, a number of improvements to the efficiency of the block IO layer were put forward so expect performance to increase further.
Well this is a pleasant surprise, I logged in and saw that my blog had got a couple of thousand hits yesterday, I assume Seans got the same – glad to see that people enjoyed our articles.
The comment about benchmarks running as I/O bound. This is one of the things that we have to avoid, I will explain this a lot more in my next blog posting (I was debating what to write about next, this has made up my mind).
One of the problems with a lot of the benchmark results you see posted on the web is that the box is I/O bound in some form, and hence you will have wait time. And wait time pretty much makes your measurements, if you are only interested in kernel or userland performance, irrelevant.
Even if its a driver, if its blocking on I/O your not going to get an accurate measurement of its capabilities (unless of course its the cause of the blocking – and this is where we love DTrace
).
Anyway more in my next blog post, once I get the day job done.
Bug reports, when the opensolaris site is public you will be able to report stuff there, but for the moment the solaris usenet groups etc are good places to post for help. As is the yahoo solaris x86 group.
If you want please feel free to send problems to me, at my blog user name @sun.com. What kind of issues are you having anyway?
One of the problems with a lot of the benchmark results you see posted on the web is that the box is I/O bound in some form, and hence you will have wait time. And wait time pretty much makes your measurements, if you are only interested in kernel or userland performance, irrelevant.
Well, actually in this case kernel performance can become very important (depending on the situation). Performance maybe not in terms of CPU efficiency, but of caching and cache replacement, memory footprint of data structures, disk scheduling, and aspects of filesystem efficiency (mainly in terms of data layout).
And likewise performance aspects of the application can be important. Especially databases which often tend to do their own caching and take more control of IO ie by flushing (fsync) or direct IO.
But I know what you meant. In the case where all else is being equal, such a benchmark will only show improved numbers with respect to improved CPU efficiency until the point where things become IO bound rather than CPU bound.
Say, what’s the feeling with respect to Linux versus Solaris multiprocessor scalability these days? Any idea?
In the good old days Solaris was undoubtedly much better here than Linux. With Linux 2.6, the kernel runs 512 CPU Altix systems (the kernel.org kernel is now the development target of SGI’s R&D engineers, I think), and will likely move to 1024 or 2048 when the dual core Itanium is released.
Now this is obviously in different environments to where your 144 CPU E25K is used, and no doubt some of your scalability pressure comes from the hardware side (eg. a crossbar interconnect probably can’t feasably scale that high without blowing out the cost). But it does lend Linux a lot of credibility nevertheless.
Extrapolating results from publically available benchmarks shows that Linux may be very close to Solaris. For some benchmarks I’ve seen, Linux on POWER5 scales *better* than Solaris on Sun (although the difference in hardware makes the comparison pretty shaky).
I’m sure you guys must do tests. Though I guess you can only give me one answer to my question, being a Sun employee. It smells funny that Sun has started attacking Linux on the low end performance ground while being unusually silent about the high end. But maybe that’s just where they feel they need to gain mindshare.
Well, actually in this case kernel performance can become very important (depending on the situation). Performance maybe not in terms of CPU efficiency, but of caching and cache replacement, memory footprint of data structures, disk scheduling, and aspects of filesystem efficiency (mainly in terms of data layout).
true, but all of this is relevant to the software (kernel or application) not the hardware, once your disk or other io subsystem becomes the bottleneck you are no longer measuring the performance of the software, which is our aim – no matter what data structures or disk scheduling your using, but I see from your last comment you understand exactly what we are trying to measure.
The reality is that your not going to see a benchmark number being posted on something like spec.org where the system has IO problems, but a lot of the benchmarks you read about are not run like that (and this is not a criticism, my colleagues and I are in a privileged position to have access to enough hardware to do this work in a much more exacting manner, well actually in the privileged position to get to do this work as our day job – I really do admire the efforts that people go to benchmark things in their spare time [although I am using my own spare time writing this now
, even with a grand prix on]).
And likewise performance aspects of the application can be important. Especially databases which often tend to do their own caching and take more control of IO ie by flushing (fsync) or direct IO.
yep, but that still comes through the kernel at some point, even if it might bypass some sub systems based on what it wishes to do. And in all cases the userland application is of interest to us as well, but we look at it as an entire system. I’m planning on going into this a lot more detail for the next blog posting, with some examples of io bound versus non io bound systems etc.
Say, what’s the feeling with respect to Linux versus Solaris multiprocessor scalability these days? Any idea?
You’ll have to excuse me for dodging the bullet on this one so to speak. I know what our feeling is, and what it generally is internally, but anything I say here will be construed as marketing speak, so I’ll leave this to the marketing folks to address if they so wish. Lets just say that 1 to 144 cpus on the same OS image, no modifications needed at all, does say a lot though.
snip Altix
As a rule I don’t write about things I either don’t understand or haven’t worked with, and while I understand what the SGI folks have done, I haven’t worked with an Altix box, or the heavily customised Linux kernel that they use, so I’ll decline to comment. Out of curiosity has anyone here used one of the large Altix boxes, I would be interested in finding out what they are like.
I’m sure you guys must do tests. Though I guess you can only give me one answer to my question, being a Sun employee. It smells funny that Sun has started attacking Linux on the low end performance ground while being unusually silent about the high end. But maybe that’s just where they feel they need to gain mindshare.
Yep, I can only give you one answer – its not about attacking Linux, it never has been, and never will be – Linux has a place, as does Solaris, as does Windows, as does BSD etc, etc, and as each OS improves it raises the bar for all the others, so the only beneificary is the consumer.
What it is about is competing with our competitors and that happens to mean that a Linux vendor, in Red Hat, is one of them, as is IBM with AIX, as is Microsoft with Windows, as was HP with HP-UX, as was SGI with IRIX etc. The competition should be on the merits of the OSes, not on anything else. I blogged about this before, back in December last year, please see http://blogs.sun.com/roller/page/fintanr/20041207#cars_all_belong_t…
You’ll have to excuse me for dodging the bullet on this one so to speak. I know what our feeling is, and what it generally is internally, but anything I say here will be construed as marketing speak.
Thanks for the response. I understand you’re not in an easy position with your answers.
As a rule I don’t write about things I either don’t understand or haven’t worked with, and while I understand what the SGI folks have done, I haven’t worked with an Altix box, or the heavily customised Linux kernel that they use, so I’ll decline to comment. Out of curiosity has anyone here used one of the large Altix boxes, I would be interested in finding out what they are like.
I have not, no. I should get the chance to work (on the kernel) with a 64-way in coming months, maybe higher if I’m lucky. But unfortunately I haven’t used solaris on anything above a 4 way, so I’d have the opposite problem.
With regards to the kernel, as I said, SGI developers are now working with plain 2.6 kernels which can boot and run the 512 ways. Their 2.6 production kernels will be SUSE kernels. In general 2.6 apparently doesn’t need heavy modifications in order to get good performance, although there is obviously room for improvement.
Yep, I can only give you one answer – its not about attacking Linux, it never has been, and never will be – Linux has a place, as does Solaris, as does Windows, as does BSD etc, etc, and as each OS improves it raises the bar for all the others, so the only beneificary is the consumer.
Yeah attacking maybe wasn’t a good choice of word because of the negative implications. Competing with / focusing on.
It’s funny that you should mention I/O and benchmarks because the article I am working on deals with that topic. I think a lot of people have no idea what goes on “behind the scenes” in a benchmark test that could influence their choice more than the results of the benchmark itself. I have started benchmarking Fedora Core 3 using iozone and on my hardware I am seeing high wio in the sar -u output during the iozone run(this is all preliminary and I intend to share my information with RedHat). I plan to start Solaris Express testing later today and see if I get similar results.
I think way too many people look at benchmarks as the “holy grail” of vindication methods for their particular OS and don’t ask the more relevant questions such as tuning options and I/O mesaurements during the tests (if they were taken at all). If people actually saw I/O results along with the benchmark results, we might actually start addressing real issues of performance rather than benchmark results that at the best of times are questionable.
I have not, no. I should get the chance to work (on the kernel) with a 64-way in coming months, maybe higher if I’m lucky. But unfortunately I haven’t used solaris on anything above a 4 way, so I’d have the opposite problem.
Its all swings and roundabouts
. I would still be interested in hearing what its like though. I will do a bit more in depth reading about Altix boxes in the mean time.
With regards to the kernel, as I said, SGI developers are now working with plain 2.6 kernels which can boot and run the 512 ways. Their 2.6 production kernels will be SUSE kernels. In general 2.6 apparently doesn’t need heavy modifications in order to get good performance, although there is obviously room for improvement.
I stand corrected, thanks for pointing this out. Are their changes in the mainline kernel tree? I haven’t followed the lkml in quite some time, Solaris keeping me somewhat occupied.
I stand corrected, thanks for pointing this out. Are their changes in the mainline kernel tree? I haven’t followed the lkml in quite some time, Solaris keeping me somewhat occupied.
Much of their stuff, yes. Much of the most fundamental performance work in their 2.4 propack was naturally backports of 2.6 features like the per-cpu scheduler.
Apparently (and I say apparently because it is still second hand information, albiet from @sgi.com employees) the last *big* thing they needed was their own block device and SCSI layer (X-SCSI or XIO, IIRC). This need was relieved when the last global lock was removed from Linux’s block layer sometime early in 2.6.x
But on that note I think there are things in the propack that still haven’t made it into the kernel (a patch for better scalability of threaded page faults is one off the top of my head). So I’m pretty sure there are still areas where 2.6 trails their heavily modified 2.4 kernel in performance.
I just recently found the Yahoo group and will check out the newsgroups as well (but the lack of a good modern newsreader that is actively developed and I LIKE has leared me away from Usenet. Any suggestions for my shiney new Solaris workstation?).
But the problem I have been having with the latest SolXpress (NV9) is I keep getting a Terminal Window to popup after entering in the Network Info during Interactive Install. And then it just sits there. I got an Option three install (interactive/desktop/text) and got it to install but now some things are broken (like Performance meter). I had none of these problems with the GA SOL Release.
newsreader : I use a modified mutt which probably isn’t the best thing in the world – its handy though. Webwise bloglines lets you sub to these groups – although its a really bad way to try and read them.
Just to clarify the install problem, you enter the network info and then a terminal window pops up and nothing further happens? This is a cd based graphical install? Do you have some details on the box?
Performance meter, this is the performance meter on CDE yes?
Email sent (to fin…@sun.com). Thanks