The homegrown Linux operating system has come a long way from its origins as a college kid’s pet project and computer hobbyist favorite. Refined in recent years by professional computer programmers at IBM, Hewlett-Packard, Novell and Red Hat, Linux now has become so technically powerful that it lays claim to a prestigious title–it runs more of the world’s top supercomputers than any other OS.
Linux Rules Supercomputers
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
And yet it still isn’t an operating system. Amazing this kernel completely replaces an operating system all on it’s own.
Linus just looks so happy with that pinguin in his lap 🙂
And only 2 of the top 500 run BSD ? Ah well at least it’s 1 better than Windows. It would be interesting to see what the 7 custom jobs are like.
Where are the “Linux-doesn’t-scale” trolls?
Didn’t the reknown experts from EDS just tell us that it is insecure and doesn’t scale well?
I was just told that it was horrible by that other group.
But still, the Macs are still at #7, I thought they were going to get knocked down a little lower. And Tyr, did you make this one of the BSDs?
The issue is more complex:
1. Linux scales much better on platforms like POWER or IA-64 than 32-bit x86. On the other hand, Linux might not have squeezed the last quetch of SMP scalability on the x86 platform, which one however could argue, is not important anymore (with AMD64 and IA-64 available).
2. Security is almost completely irrelevant for supercomputers. Most of them are sitting behind tightly configured firewalls.
I especially like the part where it says Linux outguns Solaris and AIX. I don’t think my Solaris brethrens are awake yet. Sorry, I have to quote it:
Moreover, Seager says Linux outguns popular Unix operating systems like AIX and Solaris from Sun Microsystems (nasdaq: SUNW – news – people ) because those systems contain features that make them great for commercial users but add a lot of system overhead that ends up limiting overall performance. One example: a “virtualization” feature in AIX lets many applications share the same processor but “just hammers performance,” Seager says.
But still, the Macs are still at #7, I thought they were going to get knocked down a little lower. And Tyr, did you make this one of the BSDs?
Nope, I just read the posted article. Just 497 to go now I guess 😉
This is proof that it does, and performs spectularily well.
In a lot of the Linux vs Solaris flame wars here on OS news, the Solaris crowd kept saying Linux doesn’t scale. Solaris is powerful, but Linux does in fact scale and is very fast.
The proof is in the pudding.
[the quote]
Well, the statement is sketchy at best.
1. Sun’s limiting factor would first be their CPU, which is (compared to alternatives) rather unsuitable for numerical computation. And compared to companies like IBM, SGI, or Cray, Sun has at best very limited experience with low latency interconnect.
2. POWER5 Micro-Partitioning is available for both AIX 5L and Linux. So how is this just affecting AIX? Or does he want to say in reality that the POWER Hypervisor “hammering performance”?
Linux rules SuperComputers is right!
Linux also rules AverageComputers too :^)
Just a guess, not even an good guess at that!
Linux rules SuperComputers is right!
Linux also rules AverageComputers too :^)
Yes we’ve heard that for a couple of years… and it’s yet to be seen…
But the Microsoft/Sun-lead alliance, Agility Alliance, says that Linux isnt secure, scalable and makes a bad coffee!
How can you run it on super computers? Doesnt everybody know that windows is what powers supercomputer??
But seriously, I’ve just read the Agility Alliance new story and felt abit sick. I’m just waiting for a client to call me up and ask why we’re using linux on his servers when Microsoft/Sun says otherwise.
Article:
http://www.zdnet.com.au/news/software/0,2000061733,39184795,00.htm
> In a lot of the Linux vs Solaris flame wars here on OS news, the Solaris crowd kept saying Linux doesn’t scale. Solaris is powerful, but Linux does in fact scale and is very fast.
For the millionth time, these quotes from the Top 500 list mean absolutely nothing in respect to Linux and its ability to scale. 99.9% of all systems in the Top 500 list are massively parallel computers that put absolutely no emphasis on the scalability of the OS running on the nodes. The only reason Linux became so popular in HPC is because Linux can run on cheap hardware (which adds up really fast when you’ve got thousands of computational nodes) and the Linux licensing up until recently has very low which also adds up when you’ve a lot of nodes. There is not much of technical merit to choosing Linux, it is mostly financial. Pretty much any other OS (even lowly Windows) could have done this job just as well as Linux provided it was cheap and it ran on the commodity hardware. There are actually a lot of negatives to Linux compared to many commercial Unix version, in particular Linux is a poor NFS client and can make things a nightmare and absense of a true distributed filesystem for Linux doesn’t make things easiser either. Technically and financially Solaris 10 should be a much better platform for HPC than Linux as Solaris is now cheaper than Linux and can run on pretty much the same commodity hardware. Solaris 10 also has the best observability tools (D-trace, etc.) that can be a huge differentiator in HPC arena. Solaris 10 is also a much better NFS client with NFSv4 and RDMA. I think when OpenSolaris officially comes out of the door there should be a pretty big exodus from Linux in HPC space — the benefits in Solaris 10 and OpenSolaris are just impossible to ignore.
Anonymous,
Linux is used on supercomputers because it scales and performs better than Solaris. Yeah, it’s cheap, but so are the *BSDs. Yeap, good thing Solaris 10 is playing catch up in the HPC world, oh wait it’s still post beta. And can you tell us how the faster supercomputer in world constitutes cheap hardware?
Moreover, Seager says Linux outguns popular Unix operating systems like AIX and Solaris from Sun Microsystems (nasdaq: SUNW – news – people ) because those systems contain features that make them great for commercial users but add a lot of system overhead that ends up limiting overall performance. One example: a “virtualization” feature in AIX lets many applications share the same processor but “just hammers performance,” Seager says.
Don’t waste your time with the Anonymizer Solaris Fanboy. For him, Solaris can do no wrong and Linux can do no good. It’s a black and white world for him, and the fact that he goes out of his way to use an anonymizing service should be enough to tell you he doesn’t want a real debate.
> Linux is used on supercomputers because it scales and performs better than Solaris. Yeah, it’s cheap, but so are the *BSDs. Yeap, good thing Solaris 10 is playing catch up in the HPC world, oh wait it’s still post beta.
Tying up hundreds or thousands of single or dual processor machines into a cluster is not a definition of scalability and Linux technically has absolutely nothing to do with it. The overhwelming majority of these massively parallel supercomputers still run all of their computational units in the context of one or maybe two processors because this is how big all computational nodes are — what scalability are you talking about. Intrinsic OS scalability counts only when the OS for the system is running in single image. And Linux is not very good at running in single image, according to Linus’s own words Linux will have problems scaling beyond 16 CPU’s in the single image, something Solaris will eat for breakfast. Sorry, but Linux has a lot of catching up to do with commercial Unix OS’s in the scalability department.
> And can you tell us how the faster supercomputer in world constitutes cheap hardware?
Look at the majority of Linux clusters on Top500, with the exception of IBM PPC clusters and SGI Altix systems almost all of them run either commodity Xeon/Opteron hardware or Apple XServe commodity boxes.
> And can you tell us how the faster supercomputer in world constitutes cheap hardware?
I expect that what Mr Anonymiser is getting at is that compared to Sun’s mid-range Sparc based hardware, the fastest supercomputers in the World probaby do look cheap!
> I expect that what Mr Anonymiser is getting at is that compared to Sun’s mid-range Sparc based hardware, the fastest supercomputers in the World probaby do look cheap!
Out of all vendors selling midrange gear, gear from Sun is probably the cheapest (much cheaper than HP or IBM). So if you’re talking midrange, comparatively Sun is a low cost vendor, there is no secret about that.
Anonymizer,
Perhaps you should take sometime to understand what scalability is all about. The fastest computer in world, running Linux of course, is optimized for bandwith, scalability as well as the ability to handle large amounts of data. Read up a little more on why people design supercomputers, and you’d realize scalability is a primary reason.
Not only is Linux horizontally and vertically scalable, it scales up and down Solaris’ ass. From mainframes down to wrist watches, Linux is there. If Solaris was so scalable, it would be on most of the worlds supercomputers. And no, your definition of supercomputers as a connection of clusters is wrong. Wikipedia is your friend.
Anyway, I’m tired, Linux runs most of the worlds most powerful and scalable computers, an achievement that cannot be attributed to Solaris 10. And single image systems are so 1970s, nobody uses them in this century. Well, nobody with a brain and basic economic sense. They are effectively termed legacy hardware in many domains.
I ain’t arguing with you any longer, if this article doesn’t prove to you that Linux is scalable, powerful, reliable and trusted in the HPC arena, nothing will.
Good day.
according to Linus’s own words Linux will have problems scaling beyond 16 CPU’s
Not according to this ZDNet article:
“HP on Tuesday demonstrated Linux scaling to 64 processors on its Superdome hardware as it seeks to reassure customers who are increasingly considering the open source operating system for enterprise applications.
Running three different benchmarks on a standard Linux distribution based on the 2.6 kernel, the Superdome showed linear improvements for kernel compiling, memory bandwidth, and the HPL common supercomputer benchmark.”
From the article “BigTux project shows Linux scaling to 64 processors”:
http://news.zdnet.co.uk/0,39020330,39184546,00.htm
That’s for vertical scaling. However, the other posters are right: scalability isn’t only vertical, it can also be horizontal. And in that area, Linux has proven itself:
“Has Linux really lagged behind in scalability?
[Sam] Greenblatt: Linux has not lagged behind in scalability, [but] some vendors do not want the world to think about Linux as scalable. The fact that Google runs 10,000 Intel processors as a single image is a testament to [Linux’s] horizontal scaling. The National Oceanographic and Atmospheric Administration (NOAA) has had similar results in predicting weather. In the commercial world, Shell Exploration is doing seismic work on Linux that was once done on a Cray [supercomputer].”
From the article “Forget the FUD about Linux scalability”:
http://searchenterpriselinux.techtarget.com/originalContent/0,28914…
That’s not even counting the SGI Altix running Linux on its 256-processors:
http://www.techbuilder.org/news/59201414
Anonymizer Guy, just admit that Linux is a good OS, just like Solaris is a good OS. I’m sure Mr. McNealy won’t mind.
> Not only is Linux horizontally and vertically scalable, it scales up and down Solaris’ ass
Get this through your thick skull, there is absolutely no technical feat in holizontal scalability. Any OS, I repeat *any* OS that has a networking stack can be horizontally scalable, Windows 95 could be just as horizontally scalable as Linux, therefore there is nothing amazing about these horizontal clusters on Top 500 — it is not indicative of the intrinsic scalability of the OS, which can be demonstrated only in the single image. When Linux becomes capable of scaling from 1 to lets say 128 processors on the transactional database workload (peanuts for Solaris), make sure to let me know cause right now Linux doesn’t look all that bright at all as far scalability for databases goes in the single image.
> And single image systems are so 1970s, nobody uses them in this century. Well, nobody with a brain and basic economic sense. They are effectively termed legacy hardware in many domains.
It looks like you’re absolutely clueless about this subject. I tell you what, graduate from that shitty university of yours, which aparently didn’t teach you much, get some experience and than try posting something here. You’re just as ignorant as the most Linux fan boys.
> “HP on Tuesday demonstrated Linux scaling to 64 processors on its Superdome hardware as it seeks to reassure customers who are increasingly considering the open source operating system for enterprise applications.
On what workloads they’ve done it and how linear is it? I would take this HP marketing mambo jumbo with a serious grain of salt. Then next thing HP is going to tell you is that Itanic is the processor architecture of the future. LOL.
It’s a fact that Linux sucks at scaling beyond 16 processors, a tribute to that is the fact that almost no vendors are shipping any Linux servers capable of more than 32 processors. Unisys could sell you a server with 32 procs, but they will be quite to tell you that you need slice the server into 8 or 16 processor partitions to take the real advantage of the 32 processors — Linux does not really scale beyond 16 processor especially on database workloads.
I bet you would start pointing at SGI Altix after reading the above, but here this Altix can show decent performance on 256 processors only and *only* on the highly parallel computational workloads (HPC) and not on anything else. This is the reason SGI does not even attempt to target Altix at business/database space — Altix performance would be so abismal, it wouldn’t be funny.
Remember that Linux kernel is still very coarse grained and not multithreaded as Solaris for instance, Linux will always suck on workloads that require a lot of multithreading and lock management. Databases will always belong to Solaris until Linux kernel gets completely rewritten from scratch.
Any OS, I repeat *any* OS that has a networking stack can be horizontally scalable, Windows 95 could be just as horizontally scalable as Linux, therefore there is nothing amazing about these horizontal clusters on Top 500
Sure. If you don’t mind, I’ll take the word of the experts over the guy who uses an anonymization service.
As for the Windows 95 comment, that was pretty funny. Ha! Making a supercomputer out of Win95 boxes would be quite a feat, indeed!
Remember that Linux kernel is still very coarse grained and not multithreaded as Solaris for instance, Linux will always suck on workloads that require a lot of multithreading and lock management. Databases will always belong to Solaris until Linux kernel gets completely rewritten from scratch.
Oh, I see. This is a territorial thing for you: databases belong to Solaris (nevermind that the topic is about supercomputing, not databases).
Well, I know one database that doesn’t belong to Solaris. You may have heard of it, they recently went public:
http://www.google.com
Go tell these guys that Linux isn’t scalable, that horizontal scalability doesn’t count.
How do you like that, Belafonte?
For the millionth time, these quotes from the Top 500 list mean absolutely nothing in respect to Linux and its ability to scale. 99.9% of all systems in the Top 500 list are massively parallel computers that put absolutely no emphasis on the scalability of the OS running on the nodes.
Regardless if you say it for one million times: This is not the whole truth. Massively parallelism also has its scale limit. And several of the Top 10 Linux machines (e.g. Altix, or pSeries) are not purely MPP. NASA’s Altix in fact has at the order of 256-512 CPU per shared memory NUMA node, this is more than any typical Sun hardware.
And also from my side for several times: Scalability is not just an OS issue. Maybe you should try running Solaris x86 with > 64 CPU in SMP yourself.
As a System Admin of one of the larger Linux clusters on the top500 list (and yes, Linux fanatic), here’s a few things to straighten things out…
Linux basicly has a monopoly on clusters of x86 and x86_64 hardware. Linux has a very significant fraction of the G5 hardware (Linux does about 20% better than OSX for some jobs). The other few hundred entries in the top500 list aren’t clusters.
Linux on clusters has nothing to do with some others’ comments about scability. Having Linux perform well on a thousand single-proc or smp machines doesn’t require the vertical scaling they are talking about.
Linux did very well in the beginning because it was free, stable, and ran on cheap hardware. It does very well now because it is free (or really expensive if you want to pay for it), stable, runs on most all hardware, and has the inertia of an entire industry behind it.
This year we’ll see more Sol10, MacX, and Windows (yes, Windows) clusters popping up. Lots of people like Sol and MacX and they have some very good technical reasons, so they will naturally appear.
MS is debuting their cluster OS this year, so that will naturally appear.
I don’t see anyone totally displacing Linux’s current virtual monopoly in HP clustering.
The early Sol10 adopters will have a hard time because a lot of the required scientific software and a lot of hardware drivers will need to be ported.
I bet you would start pointing at SGI Altix after reading the above, but here this Altix can show decent performance on 256 processors only
1. If I am not mistaken, Altix scales currently upto 1024 CPU per node.
2. Using Itanium 2 powered HPC computer to run non-numerical applications like databases is for sure a deadly wrong investment.
Go tell these guys that Linux isn’t scalable, that horizontal scalability doesn’t count.
Why? Because clustering hurts your ego?
If the solution serves its purpose, whether it is clustered or not, is quite irrelevant. And also there is natural limit for purely SMP, which you – in your affection for a non-NUMA company – obviously just do not realize.
Oh, and by the way: x86 scalability doesn’t count. But you will know this soon enough if you have tried Solaris x86.
Oh, I see. This is a territorial thing for you: databases belong to Solaris (nevermind that the topic is about supercomputing, not databases).
I guess you have to explain why Sun is unable/unwilling to compete in TPC-C, then.
Just wondering : what kind of antivirus and anti-spyware are used on Windows supercomputers ?
Linux on clusters has nothing to do with some others’ comments about scability. Having Linux perform well on a thousand single-proc or smp machines doesn’t require the vertical scaling they are talking about.
Well, if you read carefully my remarks: Whether Linux scales on x86 has nothing to do with Linux scalability in general. And Linux scales quite well on IA-64 and POWER5.
If you want Solaris to compete in the top 10 HPC range, on the other hand, you will notice that simple clustering will not get you far enough, especially your system should be able to handle larger amount of message passing than you can squeeze through a loosely interconnected cluster.
And here you will ultimately face the problem that Solaris supports neither of the fastest platforms currently available, nor does it support scalability beyond SMP (NUMA, for example).
Oh, and by the way: Scalability is not just scalability in SMP, scalability in terms of its performance in NUMA environment is also for sure _scalability_.
You mixed up my post with Anonymizer’s. I’m basically agreeing with you.
> And here you will ultimately face the problem that Solaris supports neither of the fastest platforms currently available, nor does it support scalability beyond SMP (NUMA, for example).
Err, Solaris supports Operon very well, which is the fastest platform on a number of benchmarks. Plus what makes you think that Solaris doesn’t support NUMA — all Sun multiprocessor gear in the current product lineup can be considered NUMA as UltraSparc chips have memory controllers on chip and therefore any memory accesses going off chip are non-uniform. High end Sun servers have been NUMA for years (SF15K/SF25K).
And here you will ultimately face the problem that Solaris supports neither of the fastest platforms currently available, nor does it support scalability beyond SMP (NUMA, for example).
Oh, and by the way: Scalability is not just scalability in SMP, scalability in terms of its performance in NUMA environment is also for sure _scalability_.
You are absolutely wrong. Solaris supports NUMA Topolgy and has for a while now.
http://www.aceshardware.com/read_news.jsp?id=80000582
http://blogs.sun.com/roller/page/dp/20050301#what_s_new_in_solaris3
But Linux has dominated the OSS scene for a long time. Chances are, when you talk to someone who is interested in OSS, more than likely, you’re talking to someone who works with Linux as well.
Linux is also a community, whose membership depends on if one wants to be a part of it or not. No one can exclude another from the community because they disagree, meaning some Linux community members will be raving idiots to communistic hippies to capitalistic moguls to professionals to everyday users. Sun is company, it’s membership is exclusive by means of employment pactices and hiring policies. Sun’s management should expect their employees to act professionally and dignified (unless for some reason, they don’t want that).
[email protected], it’s good to see you know how to be a professional. At least you decided to use an anonymizer to hide your face from your bosses but you’re doing a great job representing and selling Sun to the public.
Before you start mouthing off here, whether you’re right or not, think about all the bad will you’re spreading for Sun then reconsider what you’re saying.
I’ve seen better manners in McDonald’s employees than I have with you. I know if I was your boss and I found out who you were and what you were saying, I’d fire you instantly. You may know something about computers, and you may even wear a suit, but you are not a professional.
And if you’re not a Sun employee, than comments like “You’re just as ignorant as the most Linux fan boys,” speaks volumes about you.
You are absolutely wrong. Solaris supports NUMA Topolgy and has for a while now.
I do not think so. Solaris has very limited scalability beyond SMP. And here also lies the “beyond SMP” scalability problem with Solaris or Sun hardware (situation for x86 is even worse): the largest installation possible is at the order of 128 CPU on Fujitsu-Siemens Primepower, which proves a SMP sized scalability, and only with their SPARC64 CPU, but not really an impressive NUMA scalability, and that is not even the situation with UltraSPARC IV, which is limited at about 72 CPU.
And given UltraSPARC’s performance relation to IA-64 and POWER5, this is also the reason why Sun does not shine in HPC.
Err, Solaris supports Operon very well, which is the fastest platform on a number of benchmarks.
Which benchmarks? IA-64 is pretty sure the superior platform (but maybe not the cheapest) for HPC. And in my opinion, IA-64 has a better memory architecture, which one could also easily see by why there is nobody having 256 and beyond Opteron CPU run in shared memory.
Most Opteron HPC solutions are purely clustered solutions, and not NUMA.
High end Sun servers have been NUMA for years (SF15K/SF25K).
And here is also the problem. Even in NUMA the number of CPU is limited to 72 CPU with UltraSPARC IV. This is not really an impressive NUMA or proves a high degree of NUMA scalability in any way.
I do not think so. Solaris has very limited scalability beyond SMP. And here also lies the “beyond SMP” scalability problem with Solaris or Sun hardware (situation for x86 is even worse): the largest installation possible is at the order of 128 CPU on Fujitsu-Siemens Primepower, which proves a SMP sized scalability, and only with their SPARC64 CPU, but not really an impressive NUMA scalability, and that is not even the situation with UltraSPARC IV, which is limited at about 72 CPU.
The Sunfire 25K system with 72 UltraSPARC IV (dual core cpus) cpus looks like 144 cpus to Solaris. Solaris scales fine on these boxes.
Also Solaris lgroups are primarily for NUMA, So yes Solaris supports the NUMA topology.
Read up on these links:
http://iforce.sun.com/protected/solaris10/adoptionkit/general/featu…
http://www.google.com/search?q=sunfire+NUMA&sourceid=mozilla&start=…
http://www.ukhec.ac.uk/publications/reports/ewomp2002bull.pdf
And here is also the problem. Even in NUMA the number of CPU is limited to 72 CPU with UltraSPARC IV. This is not really an impressive NUMA or proves a high degree of NUMA scalability in any way.
I am begining to think you don’t know what NUMA means.
BTW on the Supercomputer comment you might want to read up on this also.
http://pr.fujitsu.com/en/news/2002/08/22.html
http://www.top500.org/sublist/System.php?id=6166
Top500.org shows two Fujistu systems running Solaris
with 2304 and 1472 cpus each.
http://www.top500.org/lists/plists.php?Y=2004&M=11
32 National Aerospace Laboratory of Japan
Japan/2002 PRIMEPOWER HPC2500 (1.3 GHz) / 2304
Fujitsu 5406
11980
33 Kyoto University
Japan/2004 PRIMEPOWER HPC2500 (1.56 GHz) / 1472
Fujitsu
So two machines in the first 33 shows what? Superiority of Sun hardware? (Oh, it is not even Sun hardware…)
So two machines in the first 33 shows what? Superiority of Sun hardware? (Oh, it is not even Sun hardware…)
This article was about Linux and the discussion you began was about Solaris scaling and it’s support for NUMA.
Why did you suddenly change the subject to hardware? BTW Fujitsu and Sun have been reselling each others hardware for years and now they are even developing server platforms together. Fujitsu has always sold SPARC servers with Solaris, which is the topic of discussion here.
I just love it when people run out of arguments and change the subject. Keep it up.
So two machines in the first 33 shows what? Superiority of Sun hardware? (Oh, it is not even Sun hardware…)
OH because you are so clueless, let me clue you in. The link was supposed to show Solaris scaling to 2304 and 1472 cpus in a supercomuter since you brought up Solaris’ scalability beyond SMP.
Fujitsu according to thier site claim the max config of that system is 16000+ cpus. So they have tested and are ready to sell a 16000+ cpu supercomputer running Solaris.
So Solaris scales swimingly on all topologies and has for a long time, that supercomputer was running Solaris 8. How ws linux scalability in 2000?
People have installed 112+ cpu NUMA nodes with Solaris 2.6 in the 1990s. Read up on the wildfire protoype.