John Finigan has made a few timing benchmarks of a kernel compile on NetBSD 2.0RC1 on single, dual, and quad SPARC configurations.
NetBSD 2.0RC1 timings
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
does this mean that netbsd is better than solaris was at taking advantage of the 4 cpus? or 2? and how does that compare with linux?
The link for timing becnhmarks, does not seem to be working.
Just to let you know this benchmark is about NETBSD running SMP code on SPARC.
Actually Linux scales pretty linearly past 4 CPUs (while NetBSD is impressive, it is clearly starting to taper off at 4). I’d imagine Solaris would be pretty linear here too.
Have a look at this page:
http://es.tldp.org/Presentaciones/200211hispalinux/blanchard/talk_2…
To show that a 24CPU system compiled a kernel in 247.44 CPU seconds[1] (time*cpus), while a 32 way system did it in 240.64. Actually the positive scaling factor can probably be attributed to some scalability work that the author had done.
Note that this was done at the very start of 2.5, before much scalability work was done. The big thing at this point was the percpu O(1) scheduler, I guess. But you can see 2.4 is pretty scalable too.
[1] Note that the linux kernel compile has sections that can’t actually keep that many CPUs busy (the final link stage, and the startup stages, for example).
does this mean that netbsd is better than solaris was at taking advantage of the 4 cpus? or 2?
Without a doubt the answer is no. 2.0 will be the first release of NetBSD capable of SMP. IIRC Sun had the SparcStation 20 shipping with SMP in 1994. The SMP capabilities of NetBSD 2.0 seem limited, but still nice for making the system feel more responsive and for compiling things.
how does that compare with linux?
As far as I know many point versions of linux 2.6.* won’t boot with multiple processors on SPARC 32.
But in a general sense I would say that when it comes to SMP scaling, Solaris is better than Linux which is itself better than any shipping BSD.
Actually Linux scales pretty linearly past 4 CPUs (while NetBSD is impressive, it is clearly starting to taper off at 4). I’d imagine Solaris would be pretty linear here too.
Well, my guess is that my test is mostly limited by memory bandwidth at 4 processors. Also the note about all stages of the compile not being able to keep all CPUs busy all of the time is probably applicable here as well.
For embarrasingly parallel tasks like a kernel compile, scaling is likely going to be mostly limited by bandwidth assuming the OS is at all decent. The p690 compiling the linux kernel in the above link has, as best as I recall, something like 32 GB/s theoretical memory bandwidth (sorry, cant seem to find the figure online). The same test running on an 8 way compaq P3 xeon will not show scaling anything like that, seeing as all processors in that situation will be sharing something like 800 MB/s peak bandwidth to memory. In short, 8-way x86 hardware is ugly
For less parallelizable tasks, linear scalability is essentially impossible.
As far as I know many point versions of linux 2.6.* won’t boot with multiple processors on SPARC 32.
I think you’re right there.
But in a general sense I would say that when it comes to SMP scaling, Solaris is better than Linux which is itself better than any shipping BSD.
I’d be very interested in claims about Solaris scaling better than Linux nowadays. In the past that was definitely true, but today Linux has run on machines with twice as many CPUs as Solaris (and SGI is aiming for 1024, which will become 2048 or 4096 when multicore IA64s come out).
Also notice the distinct lack of benchmarks coming out of Sun. If Solaris really is more scalable than Linux, why don’t they run a couple of tests and show everybody?
Lastly, I don’t mean to be a troll, but the comment about Linux being more scalable than any shipping BSD sort of implies there is a BSD not yet shipping that is more scalable than Linux; which is doubtful.
Well, my guess is that my test is mostly limited by memory bandwidth at 4 processors. Also the note about all stages of the compile not being able to keep all CPUs busy all of the time is probably applicable here as well.
Probably. You did say your test was probably running mostly out of the caches, but I can believe that the hardware scalability comes into play even at 4 CPUs (shared bus, is it?).
For embarrasingly parallel tasks like a kernel compile, scaling is likely going to be mostly limited by bandwidth assuming the OS is at all decent. The p690 compiling the linux kernel in the above link has, as best as I recall, something like 32 GB/s theoretical memory bandwidth (sorry, cant seem to find the figure online).[i]
Well yeah, but your operating system internals are still going to start causing problems at high CPU counts. I guess it will be scheduler, process creation/destruction, memory management that are stressed by kernel compiles.
[i]The same test running on an 8 way compaq P3 xeon will not show scaling anything like that, seeing as all processors in that situation will be sharing something like 800 MB/s peak bandwidth to memory. In short, 8-way x86 hardware is ugly
I agree.
For less parallelizable tasks, linear scalability is essentially impossible.
By definition, yes
Lastly, I don’t mean to be a troll, but the comment about Linux being more scalable than any shipping BSD sort of implies there is a BSD not yet shipping that is more scalable than Linux; which is doubtful.
Right, maybe. Depends on what you mean. I should have specified the FOSS BSD’s, for which I’ll stick by that. Whether there is a proprietary system with BSD guts that scales better than linux, I dont know OTOH. And before anyone gets upset, I’m a big BSD fan. Tried FreeBSD years ago and loved it, have never stuck with any UNIX since. Though actually, Net 2.0 may be my next desktop.
I’d be very interested in claims about Solaris scaling better than Linux nowadays. In the past that was definitely true, but today Linux has run on machines with twice as many CPUs as Solaris (and SGI is aiming for 1024, which will become 2048 or 4096 when multicore IA64s come out).
All depends on your definition of scaling. The Altix and say, a Sun E25K are not even close to being for the same workload or market. If you define scaling as “Will boot a lot of CPU’s and run CPU bound tasks on them well” then you are right. If you worry about the things that are stressed in a commercial workload, ie CPU’s needing to access “far” memory constantly and a lot of time being spent in the kernel, no way. Altix is a pure technical computing machine.
Actually, Linux Journal article aside, I remain puzzled why people always point to the Altix as proof that Linux scales better than Solaris when there is another machine that they could point to that is actually comparable, the HP Integrity Superdome. Perhaps not as impressive by the numbers, but its a 64 (soon 128) way Itanium server which can be ordered with Linux and is targetted squarely at the same market as a Sun E25K.
Also notice the distinct lack of benchmarks coming out of Sun. If Solaris really is more scalable than Linux, why don’t they run a couple of tests and show everybody?
Sun publishes benchmarks, check out TPC-H. If you want to see SPECcpu for UltraSPARC IV, it was submitted about a month ago.
Also try this whitepaper:
http://www.sun.com/servers/highend/whitepapers/Sun_Fire_Enterprise_…
Probably. You did say your test was probably running mostly out of the caches, but I can believe that the hardware scalability comes into play even at 4 CPUs (shared bus, is it?)
Mostly is a logical conclusion. But not wholly, of course. Yeah, its a shared bus, in this case sharing a 40 Mhz (360 MB/s?) bus across 4 CPU’s. Pretty impressive for a 1994 design, I think. Larger Suns of the time used multiple busses.
For less parallelizable tasks, linear scalability is essentially impossible.
By definition, yes
Heh. Guess you could say I didnt like “Actually Linux scales pretty linearly past 4 CPUs”.
Rats, since I checked it around 10 hours ago, the page seemingly has gone down. Ive set up a mirror:
http://www.dsm.fordham.edu/~finigan/netbsd_smp_timings.html
Sorry guys.
Now the darn thing is back up. Seeing as Im using free hosting, I guess you get what you pay for. If you still have problems, Im sure my alma mater will not have a problem with serving a few pages from the above link.
For re-igniting my interest in older SPARC hardware. There are certainly some interesting results, looks like I’ll have to dust off the ole SPARCStation20, and let `er rip. Hook up some grunty external storage, and we’ll be smokin’.
I’m very keen to get SMP working on SPARC64, apparently it’ll only spin up the second CPU, and that’s all. We’ve got some E250’s, and an E420’s with multi-CPU’s which would be cool to get running with SMP.
Glad to hear it Phil. Speed aside, to me the form factor, expandability, and general quality engineering and cohesiveness of these machines is hard to beat.
As for the new NetBSD, I give it a thumbs up. The install was about the easiest Ive ever seen, and the base system is very nicely laid out and uncluttered.
E420…Id love to have one of those. Actually, if I were in your situation I would give FreeBSD 5.3 beta a shot. They have SPARC64 isos built and supposedly SMP is fully supported.
Looks nice.
All depends on your definition of scaling. The Altix and say, a Sun E25K are not even close to being for the same workload or market. If you define scaling as “Will boot a lot of CPU’s and run CPU bound tasks on them well” then you are right. If you worry about the things that are stressed in a commercial workload, ie CPU’s needing to access “far” memory constantly and a lot of time being spent in the kernel, no way. Altix is a pure technical computing machine.
Altix doesn’t just do that, though. I mean, they’ll be connected to *massive* IO systems – they don’t just pull their data out of thin air.
SGI still stress and try to find bottlenecks in the kernel on 512 CPU machines. Page fault scalability for multiple threads was a recent one that was being improved.
Actually, Linux Journal article aside, I remain puzzled why people always point to the Altix as proof that Linux scales better than Solaris when there is another machine that they could point to that is actually comparable, the HP Integrity Superdome. Perhaps not as impressive by the numbers, but its a 64 (soon 128) way Itanium server which can be ordered with Linux and is targetted squarely at the same market as a Sun E25K.
I never used it as proof that Linux scales better. You actually just came out with the assertion (although not stated as a hard fact) that Solaris is more scalable than Linux.
Sun publishes benchmarks, check out TPC-H. If you want to see SPECcpu for UltraSPARC IV, it was submitted about a month ago.
I don’t really like the TPC benchmarks for a good comparison of OSes. They use wildly different clients, servers, databases, hardware, etc.
Also try this whitepaper:
Actually, it does though. If we’re talking about kernel operations, then yes it scales pretty linearly. If some userspace program can’t scale well, that is nothing to do with the kernel.
Altix doesn’t just do that, though. I mean, they’ll be connected to *massive* IO systems – they don’t just pull their data out of thin air.
Thats a given.
I never used it as proof that Linux scales better. You actually just came out with the assertion (although not stated as a hard fact) that Solaris is more scalable than Linux.
Yes yes, thats very cagey and all. What you did is present it as an example of how maybe possibly there are large linux systems that scale better than large solaris systems. But what about the issue? Is it appropriate to compare the Altix to absolutely anything out there with a bunch of CPU’s, or are there more appropriate comparisons available?
Actually, though, I was wrong about the Superdome shipping with Linux. Windows and HP-UX now, Linux (Red Hat, Suse) and VMS in a few months.
I don’t really like the TPC benchmarks for a good comparison of OSes. They use wildly different clients, servers, databases, hardware, etc.
Your exact quote which I was responding to was “Also notice the distinct lack of benchmarks coming out of Sun. If Solaris really is more scalable than Linux, why don’t they run a couple of tests and show everybody?” I was pointing out that there is no lack. That whitepaper contained plenty more things than TPC-H anyway. If you wonder why Sun does not compare linux to solaris on the same 144 way (72, etc), hardware and show the world, it is because Linux doesnt run on that hardware.
Boring. No Linux comparisons.
You could always do a little research yourself. A start would be that I know NEC has published a few results for its 32-way Linux+Itanium systems.
Yes yes, thats very cagey and all. What you did is present it as an example of how maybe possibly there are large linux systems that scale better than large solaris systems. But what about the issue? Is it appropriate to compare the Altix to absolutely anything out there with a bunch of CPU’s, or are there more appropriate comparisons available?
Considering the *lack* of comparisons available, the fact that Linux has run more CPUs that Solaris is the basis for my doubting that Solaris is hands down more scalable.
Your exact quote which I was responding to was “Also notice the distinct lack of benchmarks coming out of Sun. If Solaris really is more scalable than Linux, why don’t they run a couple of tests and show everybody?” I was pointing out that there is no lack.
There is a lack of verifiable, reproducable and fair comparisons versus Linux. For Sun it would be trivial to pump out a few of these benchmarks.
That whitepaper contained plenty more things than TPC-H anyway. If you wonder why Sun does not compare linux to solaris on the same 144 way (72, etc), hardware and show the world, it is because Linux doesnt run on that hardware.
No, it is because they don’t want to compare with Linux.
Look, they’re having big problems with Linux eating away their low end market, right? Well why not have Linux and Solaris go head to head on a 4-way and an 8-way SPARC, to show how much better Solaris is?
You could always do a little research yourself. A start would be that I know NEC has published a few results for its 32-way Linux+Itanium systems.
Dude, what part of “operating system comparison” do you not understand? I told you I’m not interested in seeing Linux on NEC’s I2 versus Solaris on SPARC.
There is a lack of verifiable, reproducable and fair comparisons versus Linux. For Sun it would be trivial to pump out a few of these benchmarks.
That whitepaper contained plenty more things than TPC-H anyway. If you wonder why Sun does not compare linux to solaris on the same 144 way (72, etc), hardware and show the world, it is because Linux doesnt run on that hardware.
No, it is because they don’t want to compare with Linux.
Wrong. It would not be trivial in any sense because Linux currently will not run on any US III machines that have >4 processors. Actually Im not even sure if anyone supports the v440 or v480. For, say, 8 processors you have a choice of the v880 or the Fire 3800, and there is no support for either. They could perhaps run your test on an ’98-era US II box, but would you pay attention to that? Personally I do not think it is reasonable for Sun to port linux to these machines just for a test.
But honestly, what would be the point? No vendors support Linux on SPARC. Who would spend tens or hundreds of thousands of dollars on a system no one, including Sun, would support? Such a thing would make Solaris on x86 look, well, like Linux on x86.
To be totally clear I am not trying to insult anyone with this statement, but simply to say that no one has yet thought it important enough to support these machines under linux. I dont blame them.
You could always do a little research yourself. A start would be that I know NEC has published a few results for its 32-way Linux+Itanium systems.
Dude, what part of “operating system comparison” do you not understand? I told you I’m not interested in seeing Linux on NEC’s I2 versus Solaris on SPARC.
I brought up an Itanium system because you mentioned Altix. Thought maybe you were a fan. Really I was just trying to find any published commercial workload results for linux on >= 32 processors anywhere.
Considering the *lack* of comparisons available, the fact that Linux has run more CPUs that Solaris is the basis for my doubting that Solaris is hands down more scalable.
Tenuous; see above reasons. Perhaps soon we will see an 8-way opteron head-to-head, but that is about all I would hope for.
Wrong. It would not be trivial in any sense because Linux currently will not run on any US III machines that have >4 processors. Actually Im not even sure if anyone supports the v440 or v480. For, say, 8 processors you have a choice of the v880 or the Fire 3800, and there is no support for either. They could perhaps run your test on an ’98-era US II box, but would you pay attention to that? Personally I do not think it is reasonable for Sun to port linux to these machines just for a test.
Maybe, but there is a subset of SPARC hardware that both Linux and Solaris run. There is plenty of x86 hardware that they both run as well.
If Solaris was faster anywhere, we’d have heard about it. Why do you think it takes MS so long between pumping out “independant” benchmarks? Because they have to really search hard to find setups that are (or seem to be) fair where NT beats Linux.
I brought up an Itanium system because you mentioned Altix. Thought maybe you were a fan. Really I was just trying to find any published commercial workload results for linux on >= 32 processors anywhere.
Altixes are very cool. If that makes me a fan then OK.
Here is a good one: SPECJBB on a SunFire 15K with 104 CPUs gets a score of 602270. A 16 core P5 running Linux gets 542145. Heh, that’s a good “different hardware, different os, different server software” comparisons for you.
Tenuous; see above reasons. Perhaps soon we will see an 8-way opteron head-to-head, but that is about all I would hope for.
It isn’t tenuous at all. That is a very good basis for *me doubting Solaris is hands down more scalable*.
In the mean time, 2.0_RC4 has been tagged.
You can find binaries and ISOs at http://netbsd.student.utwente.nl
Maybe, but there is a subset of SPARC hardware that both Linux and Solaris run. There is plenty of x86 hardware that they both run as well.
If Solaris was faster anywhere, we’d have heard about it.
So we have gone from large scale SMP scalability to plain old faster? The subset you speak of is the smallest machines. Nothing that can tell us what we wanted to know.
I brought up an Itanium system because you mentioned Altix. Thought maybe you were a fan. Really I was just trying to find any published commercial workload results for linux on >= 32 processors anywhere.
Altixes are very cool. If that makes me a fan then OK.
Here is a good one: SPECJBB on a SunFire 15K with 104 CPUs gets a score of 602270. A 16 core P5 running Linux gets 542145. Heh, that’s a good “different hardware, different os, different server software” comparisons for you.
Sorry, I meant ‘thought maybe you were an Itanium fan’, but perhaps that was unclear.
POWER5 is very impressive. Yet I think that the below quote speaks for itself–make up your mind.
Dude, what part of “operating system comparison” do you not understand? I told you I’m not interested in seeing Linux on NEC’s I2 versus Solaris on SPARC.
And by the way, seeing as I never mentioned NEC before that point, there is no way you could have “told me that”. Search the page if you dont believe this.
—
It isn’t tenuous at all. That is a very good basis for *me doubting Solaris is hands down more scalable*.
Yes, it is certainly purely logical to assume that a currently impossible test not having been performed leads to your assumption.
By the way, if you are in the mood for SPECjbb, here are a few more results, with submission dates:
Sun 15K, 104 CPU, Solaris (2002): 602,270
IBM POWER5, 16 core, AIX (2004): 633,106
IBM POWER5, 16 core, Linux (2004): 542,145
Fujitsu 2500, 112 CPU, Solaris (2003): 1,420,177
Sun E6900, 48 core, Solaris (2004): 421,773
HP rp8420, 32 core, HP-UX (2004): 574,452
The top Sun result is 2 processor generations old, the Fujitsu result is 1 processor generation old. Everything else is a current product.
So we have gone from large scale SMP scalability to plain old faster? The subset you speak of is the smallest machines. Nothing that can tell us what we wanted to know.
Nope. It would be smiple to test Solaris vs Linux on a 16-way (or higher if you like) x86 system. Turn off half the CPUs and test again, turn off half and test again, etc. This would give you some nice looking scalability curves.
I really don’t know why you think this would be so difficult.
By the way, if you are in the mood for SPECjbb, here are a few more results, with submission dates:
Yeah, it is interesting to see AIX ahead of Linux. Supposedly this comes about mostly due to the POWER’s odd memory management scheme. While AIX is optimised for it, it is very tied in to that architecture (not surprisingly).
POWER5 was meant to have some features to help Linux along in this regard, but IBM reckons it is still only about 90-95% as good as AIX.
Another interesting metric is the Linux vs AIX scalability. Linux is very impressive (although not surprisingly a bit behind).
SPECjbb
Cores 2 4 8 16
Linux 1.00 1.95 3.62 6.56
AIX 1.00 1.97 3.81 7.34
Throw in Solaris (on PRIMEPOWER)
Cores 8 16
Linux 1.00 1.81
AIX 1.00 1.92
Solaris 1.00 1.88
(scores are normalised)
And, gee, it looks like Solaris mightn’t actually be the duck’s guts afterall. It is obviously still not a definitive comparison because of different hardware and JVMs, but you get the picture.
Yes, it is certainly purely logical to assume that a currently impossible test not having been performed leads to your assumption.
No, that’s a bit naive. I come from a realist point of view that says if a company feels threatened by something, it will trash it if at all possible. Sun’s got a big mouth about how much better Solaris and how it is x percent better than Linux at this and that. They’re hilariously short on real numbers though.
If you honestly believe Sun isn’t throwing benchmarks at Linux because they can’t get it to run on their hardware, then I’m lost for words. You seem like an intelligent fellow…
Nope. It would be simple to test Solaris vs Linux on a 16-way (or higher if you like) x86 system. Turn off half the CPUs and test again, turn off half and test again, etc. This would give you some nice looking scalability curves.
I really don’t know why you think this would be so difficult.
Really it boils down to two things. Firstly the hardware. Feel free to correct me, but as far as I know you can currently buy 2 such 16-way machines: the IBM x445 and a Unisys “mainframe”. The Unisys is, I hope we can agree, out of the picture–its a weird anomoly. The x445 looks like a decent candidate except that Solaris x86 wont even boot. The machince is a NUMA design using IBM’s in house “Summit” chipset. Not unlike Altix in that sense, except Altix is much better.
Secondly-Could Sun write drivers? Sure, but how could that expense be justified? If youre looking at an x445, I bet you want to run Windows. If you dont, they’d sure love to sell you a SPARC machine instead.
Perhaps if/when Sun puts out its big Kealia Opteron boxes we’ll see.
If you honestly believe Sun isn’t throwing benchmarks at Linux because they can’t get it to run on their hardware, then I’m lost for words. You seem like an intelligent fellow…
Thanks, as do you.
But yes, I honestly believe this. At least I believe that Linux wont run on big (“FirePlane” — NUMA) Sun machines and making it do so would be nontrivial and for Sun, expensive. Since this expense wouldnt gain them anything but the ability to do a test, why would they bother? Having a Linux-on-SPARC ecosystem does not help them. Why would anyone buy that? Especially because people have Solaris binary investments, and enterprise Linux isnt really much cheaper, if at all, on a machine that size (purchase price).
Besides, your interesting comments on Linux on P5 bring up another point–they would not just have to get it to run on a E25K, they would have to get it optimized, if the test was to be fair.
Throw in Solaris (on PRIMEPOWER)
Cores 8 16
Linux 1.00 1.81
AIX 1.00 1.92
Solaris 1.00 1.88
Interesting. The one thing that seems to hold those big PRIMEPOWER machines back is their old, ready for the glue factory memory bus. The newer 90 nm CPU’s have more cache than the ones in this test (and 50% better clock, yikes), but SPECcpu still shows a nice dropoff. Luckily, SPARC64 VI will bring a new bus.
But generally, I think Fujitsu CPU’s and systems rock.
On the other hand, the one thing that truly impresses me about P5 is its memory subsystem. Outrageously expensive design, but impressive. I dont know why IBM is so shy about the memory numbers for P4 and P5. Probably because they want us to ohh and ahh over the MPU design only
Actually, and Im really not trying to be contentious here, just to point out something interesting–I got a real chuckle out of the P5 16 way TPC-C submission. The machine would have turned in a great score one way or another, but IBM felt it necessary to equip it with 610 TB of disk attached. Before that, the biggest I’d seen was around 75 TB. I think I would have respected a lower, more realistic score more, just because seeing the machine get a sizable chunk of a current 64-way machine’s score on a level field would have been cooler. Kind of makes me wonder what theyll attach to the p690 successor when it comes out. 2 PB?
Then again, lots of people say TPC-C is crap. Could well be.