Linked by Thom Holwerda on Thu 16th Apr 2009 19:54 UTC
General Unix Even though the old-world UNIX operating systems, like IRIX and HP-UX, have been steadily losing ground to Linux for a long time now, they do still get updated and improved. HP-UX 11i v3 is supposed to get update 4 tomorrow, with a host of new features that won't excite you if you're used to Linux, but they're still pretty useful for HP-UX users.
Thread beginning with comment 359046
To read all comments associated with this story, please click here.
Linux does not scale
by Kebabbert on Fri 17th Apr 2009 11:44 UTC
Member since:

When we talk about scalability, (llama): We're not talking about clusters. We're talking about single-system-image big iron, where _one_ kernel runs on a single machine with > 16 CPUs in a cache-coherent shared-memory system. The most cost-effective machines for cluster-building, in CPU power per dollar, are dual-socket quad core Intel Core2-based machines. i.e. 8 cores per node. That's great if you have a workload that has some coarse-grained parallelism, or is embarrassingly parallel, e.g. processing 100 separate data sets with single-thread processes that don't depend on each other. That's not so great if you have a lot of processes that need fine-grained access to the same shared resource. The canonical example here is a database server handling a database with a significant amount of write accesses. Otherwise you could just replicate it to a big cluster and spread the read load around. Locking for write access in a big cluster, even with low latency interconnects like infiniband, is still _way_ higher overhead than you'd get in e.g. a 4 or 8 socket quad-core machine. Even NUMA big iron is better suited for this than a cluster.

CLUSTERS DON'T COUNT AS BIG IRON. They're just a pile of normal machines. They do have their uses, though.

Get it? Large cluster is not big iron. It is NOT scalability. Get it?

If I have to modify Linux to be able to run on several CPUs, then Linux is not scalable. It is modifiable. I could redesign an C64 emulator and spread the load on several nodes - does that mean that C64 is scalable? NO. Solaris on the other hand, uses the same installation DVD on small intel Atom CPUs up to big iron with hundreds of CPUs - THAT is scalability. The same kernel, without modifications, scale up. Standard Linux does not scale, you have to modify it. I could also modify C64 emulator - but C64 is not scalable. Neither is Linux. BIG IRON and LARGE CLUSTERS are completely different.

Reply Score: 1

RE: Linux does not scale
by tylerdurden on Fri 17th Apr 2009 22:43 in reply to "Linux does not scale"
tylerdurden Member since:

LOL. I don't think the term "big Iron" means what you think it means.

BTW, Linux has been scaled to over 512 processors for a single image. Neither HP-UX nor Solaris, nor AIX for that matter have proven to scale to that degree in single image configurations. And to be honest, the markets that those unix target are not so much focused on things like scalability, as they are focused on reliability and redundancy.

Saying that Linux "does not scale" is disingenuous at best, or you simply have no clue what you are talking about.

Edited 2009-04-17 22:45 UTC

Reply Parent Score: 1

RE[2]: Linux does not scale
by dvzt on Fri 17th Apr 2009 23:42 in reply to "RE: Linux does not scale"
dvzt Member since:

BTW, Linux has been scaled to over 512 processors for a single image. Neither HP-UX nor Solaris, nor AIX for that matter have proven to scale to that degree in single image configurations.

Sun Fire M9000 has at maximum 64 pyhsical CPUs x 4 cores x 2 threads equals 512 virtual CPUs exposed to the OS. That is no customised machine and hacked OS, but standard hardware and same installation CD as you would put on any other kind of SPARC machine!

Reply Parent Score: 1

RE[2]: Linux does not scale
by Kebabbert on Sun 19th Apr 2009 21:33 in reply to "RE: Linux does not scale"
Kebabbert Member since:

Big Iron does not mean a cluster, which is a bunch of computers. Read my first post again. Slowly.

Regarding Solaris that has not scaled to 512 cpus or is not able to scale, why dont you read this link?

"These types of techniques allow the Solaris kernel to scale to thousands of threads, up to 1 million I/Os per second, and several hundred physical processors. Conveniently, this scaling work can be leveraged for CMP systems. Techniques such as those described here, which are vital for large SMP scaling, are now required even for entry-level CMP systems. Within the next five years, expect to see CMP hardware scaling to as many as 512 processor threads per system, pushing the requirements of operating system scaling past the extreme end of that realized today."

Later this year, rumours say that SUN will release a machine with 2048 hw threads, which will be presented as 2048 CPUs to Solaris. It will have 8 Niagara III cpus (256 threads each).

Here we that prior to Linux kernel 2.6.27, Linux was 250 times slower in 64cpu configurations. 250 times slower is hardly good scaling? I bet there are other limitations still which will not allow Linux to scale.

To clarify, of course Linux will run on lots of CPUs, the question is how GOOD Linux does it. The question is not if Linux is capable of that. Any OS is capable to run on a machine with lots of CPUs. The question is whether the cpus will be utilized in an efficient manner. For Linux, the answer is no.

Linux sucks badly as a file server, due to limitations in the kernel:

It takes years and a long time to scale well. Solaris in it's first iterations didnt scale well. But now, after 30 year, Solaris scales extremely good - with the same install DVD. Without no hacks or modifications of the Kernel. Linux redesigns all the time, as Andrew Morton explains, this approach leads to Linux having lots of bugs. The quality of the code sucks, as Linux kernel Developer Andrew Morton explains. How in earth can Linux scale well if the bugs never get ironed out, due to redesigns all the time?

"I used to think [code quality] was in decline, and I think that I might think that it still is. I see so many regressions which we never fix."

Reply Parent Score: 2

RE: Linux does not scale
by SEJeff on Sat 18th Apr 2009 00:46 in reply to "Linux does not scale"
SEJeff Member since:

The default number of CPUS the _upstream_ linux kernel supports for x86 is 64.


Do the world and unplug your computer. Please quit trolling. Some of the SGI Linux supercomputers push > 4096 cpus in a SSI (single system image for those that don't know). What the heck do you think the SLUB memory allocator was for? It was for dealing with memory on these crazy NUMA machines with terabytes of memory. How many Solaris boxes do you see with > 1024 cpus in a SSI? None!.

I think HP-UX does this better than anything though. HP Nonstop has a theoretical limit of infinite cpus if memory serves. The rolling upgrades make it pretty much bulletproof.

Moral of the story, "Linux is here to stay and quit being an idiot". Unix is good and it is where most of the ideas in Linux come from.

Reply Parent Score: 2

RE[2]: Linux does not scale
by Kebabbert on Sun 19th Apr 2009 22:09 in reply to "RE: Linux does not scale"
Kebabbert Member since:

I would hardly call 64 cpus good scaling. Prior to Linux 2.6.27 the kernel was 250 times slower on 64 cpu systems. (Solaris people have long time been talking about hundreds of cpus and many more threads).

Those SGI supercomputers with 4096 cpus, how old are they? Which Linux kernel version did they use? Linux v2.2? v2.4? Oh yes, the 2.4 Linux scales very well. Here we have some Linux scaling experts debunking the FUD that Linux scales badly. They clarify everything about the FUD:,289142,s...

"Linux has not lagged behind in scalability, [but] some vendors do not want the world to think about Linux as scalable. The fact that Google runs 10,000 Intel processors as a single image is a testament to [Linux's] horizontal scaling.

Today, Linux kernel 2.4 scales to about four CPUs

-With the 2.6 kernel, the vertical scaling will improve to 16-way. However, the true Linux value is horizontal scaling.

Q: Two years from now, where will Linux be, scalability-wise, in comparison to Windows and Unix?
A: It will be at least comparable in most areas"

Linux scales to 10.000 cpus in one single image in current v2.4, and in Linux 2.6 the kernel will improve to 16-way. Huh? Is it only me that sees a contradiction? You have bought everything about Linux scaling well and being well coded.

When I say that Linux scales bad, even the Linux experts agree, as I have proved. I am not trolling. I find it extremely hard to believe that in v2.4 Linux scaled bad (2-4 CPUs) and in 2.6 it suddenly scales better than Solaris does with hundreds of cpus? It takes decades to scale well. What am I, a fool? Can't I think? Do I buy everything? No critical thinking? Do the world a favour, and apply some critical thinking on everything you hear.

Linux scales well on large clusters, yes. But that is NOT Big Iron. How many times must I repeat that? Read my first post again. When people says Linux scales well (which it does) then they talk about clusters.

In other words; Linux scales well HORIZONTALLY, but sucks VERTICALLY. Get it? How many times must I say this? It is explained on wikipedia. Read it. Twice. Slowly.

Linux Kernel developer Andrew Morton

"it would help if people's patches were less buggy."

Let me say this again, apparantely some people have problems understanding this: If I modify a kernel to scale well, the kernel does not scale well. The MODIFIED kernel scales well.

I can modify C64 to run on a large cluster, but no one would hardly say C64 scales well. The modified version does, but not C64. If modifier Linux runs on large cluster, fine. But that does not mean standard Linux scales well. Due to limitations, standard Linux does not run without modifications. But Solaris does run on big iron with hundreds of CPUs with the same install DVD. It scales well.

trasz; The fact that Linux uses spinlocks is one of the reasons that its performance drops noticeably under high load on many CPUs. Other operating systems use fully functional mutexes, along with interrupt threads.

Reply Parent Score: 2