Linked by Eugenia Loli on Tue 21st Oct 2003 17:48 UTC
SGI and IRIX SGI today announced that large-scale SGI Altix 3000 systems have once again generated breakthrough performance results on demanding scientific applications. Recent achievements include attaining record levels of sustained performance and scalability on a 256-processor global shared-memory SGI Altix 3000 system, the largest such system ever to run on the Linux operating system.
Order by: Score:
But...
by Christopher X on Tue 21st Oct 2003 18:03 UTC

Linux can't scale like a real Unix (read - Solaris), ask Bascule! :-P

But seriously, thats very, very cool. :-D

...
by Err on Tue 21st Oct 2003 18:08 UTC

Interesting certainly.

Of course judging by MS reports people may actually need an Altix 3000 to run Longhorn. I think perhaps a bigger desk is going to be required :>.

v review...
by andy richter on Tue 21st Oct 2003 18:09 UTC
...
by philmes on Tue 21st Oct 2003 18:23 UTC

A SSI as well. Very kinky.

Beautiful!
by Eu on Tue 21st Oct 2003 18:57 UTC

Tears of joy are falling down my face... ;)

Welcome to a new world of knowledge buildging and sharing. My god, does Linux rock!

----
But remember Sun, Linux has no place on the server! LOL

SGI Altix is running SuSE Linux
by Gil Bates on Tue 21st Oct 2003 19:13 UTC

http://www.suse.com/us/company/press/press_releases/archive03/sgi.h...

Linux scales. Can't dispute that anymore.

Seems like Sun and SGI both like SuSE a lot for some reason.

SuSe and Sun/SGI
by Anonymous on Tue 21st Oct 2003 19:23 UTC

They both like SuSe because Redhat has their head up their ass in the licensing area. You can bet both companies tried to work with Redhat first, and decided it wasn't worth it.

Questions.
by Tim on Tue 21st Oct 2003 19:54 UTC

What kernel version are they running?

I just know this is going to get me flammed, but I have to ask....Is the scalability of this machine a function of Linux or more of a function of SGI's NUMAflex architecture


Only the SGI Altix 3000 family of servers is designed around this scalable shared-memory architecture that analyzes data sets as whole entities, without breaking them up into smaller segments to be handled by individual processors. The Altix architecture has proven ideal both for complex shared-memory applications running on a large single system image, and for communication-intensive applications optimized for clustering.


Linux should certainly be appreciated for the inroads it has made in its ability to scale (NUMA support, NPTL, 0(1) scheduler, etc), but isn't the scalability of a shared-memory SMP machine more of a function of how the underlying architecture manages/accesses memory to avoid cache coherence conflicts and lengthy execution of spin locks?

Don't flame, Inform.

WOW
by Cheapskate on Tue 21st Oct 2003 20:03 UTC

what an incredible machine...

maybe Santa Clause will bring me one for christmas...

re: Questions.
by Anonymous on Tue 21st Oct 2003 20:09 UTC

> I just know this is going to get me flammed, but I have to >ask....Is the scalability of this machine a function of Linux >or more of a function of SGI's NUMAflex architecture
The below would indicate this is NOT a NUMA box.
After all, NUMA is about breaking "data"/memory into segments to be handled by certain processors.

>>Only the SGI Altix 3000 family of servers is designed around
>>this scalable shared-memory architecture that analyzes data
>>sets as whole entities, without breaking them up into >>smaller segments to be handled by individual processors. The
>>Altix architecture has proven ideal both for complex >>shared-memory applications running on a large single system
>>image, and for communication-intensive applications >>optimized for clustering.

re: re: Questions.
by Tim on Tue 21st Oct 2003 20:26 UTC

The article contains the following:

The Altix family of servers combines industry-standard 64-bit Linux with the Intel ® Itanium ® 2 processor family and SGI ® NUMAflex architecture to enable global shared memory systems from a few to hundreds of processors, which is a first for Linux OS-based computing. Powered by the third-generation NUMAflex supercomputing architecture, even the largest data sets can be handled and analyzed with ease and in record time.

Linux scalability
by hmmm on Tue 21st Oct 2003 20:43 UTC

Brought to you by SGI! ;)

SGI internal architecture
by MattPie on Tue 21st Oct 2003 20:44 UTC

*Assuming SGI has bascially used the same arch as on the Origin Mips boxes*

The Origins have CPU modules, each of which have 2-4 processors and memory slots. Those are then interconnected with high-speed NUMA links. The OS generally tries to keep threads on the same group of processors so it isn't send large amount of data over the NUMA links, but any processor can access any memory in the system.

This system fits an incredibly small niche
by Bascule on Tue 21st Oct 2003 21:26 UTC

Linux is an effective operating choice for this system because it fits the same role as a scientific computing cluster, which would most likely be running Linux as well.

When used for anything but scientific computing, Linux is not an ideal choice for these sorts of systems because it lacks resource partitioning and management features, such as Solaris's Fair Share Scheduler which can be used to divy system resources into shares that are assigned to specific project groups, thus it becomes impossible for any single user/program on the system to monopolize system resources.For the purposes of scientific computing (which is what this system is designed for) these sorts of features aren't really necessary, as the entire resources of the system will most likely be monopolized by a single scientific computing application which parallelizes across the entire system with shared memory MPI.

I see this system as filling an extremely small niche. Its only advantage over a cluster is bandwidth, meaning to actually leverage the full power of the hardware the application using the system must perform relatively minimal computations on a large amount of data, enough that when parallelized the program becomes I/O bound.

We run an MPI-enabled atmospheric modelling program here called RAMS (http://atmet.com/) which is soon to be released under the GPL. This program is very much CPU bound, to the point that using GigE instead of 100bT for interconnect between nodes of a cluster has a negligable effect on the performance of the model. Purchasing an Altix system for this purpose would be a horrific waste of money compared to a cluster with the same processors (although we always go x86 for cost reasons, although some are moving to Opteron). The performance of our scientific computing application on an Altix system or a cluster with 100bT interconnect would most likely be negligable, as RAMS would not benefit from the extra bandwidth afforded by an Altix system.

All of that said, keep in mind that the Altix should not be seen as a competator to some of Sun's high end offerings. Sun's systems fit roles of mission critical systems where Solaris's resource management is needed. The resources of a system of this caliber need to be explicitly dedicated through Solaris mechanisms that Linux lacks, such as a Fair Share Scheduler.

I haven't taken too much time looking through the Altix whitepaper, but I'm still not certain if Linux/Altix has been patched to include a number of facilities that are present on Solaris. For example, since Solaris 7 a scrubber has been available for Solaris, so that if a CPU malfunction is detected, provided it doesn't bring the kernel down all processes on that CPU are moved off of it. The same functionality is provided for system RAM as well. Any of these components can then be hot swapped for good ones after a failure has been detected. In this way a mission critical system can continue running despite the failures of individual components.

The Altix does not seem designed for such mission critical roles, only for scientific computing purposes where it fits the same role as a cluster of low end systems. I don't see it as being competative with clusters in any way, and to reiterate it simply fills the very small niche of highly parallelizable I/O bound scientific computing applications. In such a role the system scheduler isn't taxed as each processor will be effectively used for a single process, so it's possible for Linux to be used as a platform for these purposes as "high end" features like project groups aren't necessary.

However, don't expect a Linux-powered system to be filling the role of a Sun Fire 15k any time soon.

re: SGI internal architecture
by hmmm on Tue 21st Oct 2003 21:28 UTC

They used Craylink interconnects on the Origin series systems. I think what they are using today is a generation ahead of the Origin systems. Probably faster interconnects with optimized hardware and software for a more efficient and scalable system.

Arch: Origin 2000
CPU: 300Mhz MIPS R12k
RAM: 4GB/brick
Interconnect: 1.6 GB/s Craylink

Arch: Altix 3000
CPU: 1300Mhz Intel Itanium2
RAM: 32GB/brick
Interconnect: 6.4 GB/s Numalink

I think O2000 had more dynamic hardware configurations, but the Altix is probably at least an order of magnitude faster. I helped build an O2000 64 processor Reality Monster a few years ago. Very nice system to work with!

Replies
by Bascule on Tue 21st Oct 2003 21:37 UTC

Christopher X (IP: ---.ok.ok.cox.net)

Linux can't scale like a real Unix (read - Solaris), ask Bascule! :-P

As I stated in my above post, Linux fits the role of providing a platform for MPI applications, as in this role it will manage a single process per processor. In terms of scalability for a general purpose server, it lacks the facilities needed to effectively manage the system, unless SGI has provided these through kernel patches and system-specific management tools.

Tim (IP: ---.sympatico.ca)

Linux should certainly be appreciated for the inroads it has made in its ability to scale (NUMA support, NPTL, 0(1) scheduler, etc), but isn't the scalability of a shared-memory SMP machine more of a function of how the underlying architecture manages/accesses memory to avoid cache coherence conflicts and lengthy execution of spin locks?

I think it's much more a function of how the scientific computing application the system is being used for parallelizes itself. When using shared memory MPI as traditionally implemented by MPICH, features like the O(1) scheduler (that assuming they're using a 2.6 kernel) and NPTL are irrelevant, as processes are used instead of threads and communicate using SVR4 shared memory (at least in the MPICH implementation). The system scheduler in such a setup manages little, as each processor will be monopolized by a single process.

I can only hope that SGI is providing their own MPI implementation which has been optimized specifically for the Altix, as a typical MPI implementation (such as MPICH) would not be able to utilize this system effectively.

Management utilities
by Bascule on Tue 21st Oct 2003 21:41 UTC

From SGI's four color glossies of the system, it does appear they have resource partitioning and high availability features:

System management tools: Partitioning, FailSafeTM (for high-availability environments), Comprehensive System Accounting, job containers

I'd greatly appreciate a link to the whitepaper on these features if someone is able to find one. I'd certainly like to see how it has been implemented and how actual management tasks are carried out.

It'd also be nice to know if the source code to these components has been released (as I don't believe any of them are available in the Linux mainline)

MPI
by Bascule on Tue 21st Oct 2003 21:44 UTC

It also appears that SGI does provide their own MPI implementation as part of their Message Passing Toolkit:

http://www.sgi.com/software/mpt/overview.html

(perhaps I should read a bit more before posting)

re: Bascule
by philmes on Tue 21st Oct 2003 22:29 UTC

fyi, Oracle, SuSE and SGI are beginning to promote the Altix for DB server use.

Re: philmes
by Bascule on Tue 21st Oct 2003 22:47 UTC

fyi, Oracle, SuSE and SGI are beginning to promote the Altix for DB server use.

The only endorsement I was able to find for Oracle on the Altix comes from this SGI press release:

http://www.sgi.com/newsroom/press_releases/2003/august/suse.html

It contains this quote from Dave Dargo, vice president of Oracle's Linux Program Office and the Performance Engineering team within the Platform Technologies Division at Oracle Corporation:

"Oracle welcomes SGI and SuSE's alliance, because it will enable Oracle customers on that platform to avail themselves of Oracle's support for Linux," said Dave Dargo, vice president of Platform Partners, Linux and Performance, Oracle Corp. "SGI servers, coupled with solutions based on Oracle Database and SuSE Linux Enterprise Server, will give customers increased choice for their enterprise Linux applications."

With that as the only endorsment for the Altix from Oracle, somehow I don't see many people flocking to the Altix as an Oracle platform.

One of the big questions with the Altix is how do you determine who is competant to administrate it? Are the tools described in the four color glossies simply ports of the Irix tools? I would hope so, but there's no way of knowing without reading the whitepaper.

I found this link to the Altix whitepaper but registration is required in order to view it:

http://www.sgi.com/servers/altix/whitepapers/index.html

Linux Kernel...
by Wayne Scotting on Wed 22nd Oct 2003 00:22 UTC

I bet 10,000 quatloos that the Linux kernel running on that box isn't something you can get your paws on...

re:
by philmes on Wed 22nd Oct 2003 01:15 UTC

bascule:

With that as the only endorsment for the Altix from Oracle, somehow I don't see many people flocking to the Altix as an Oracle platform.

There was quite a bit of talk at Oracleworld. OTOH, Oracle have clearly indicated that they think the days of the megaboxen are limited and the way forward is RAC.

I'm inclined to agree with them. We'll certainly be migrating some of our clients to RAC's during the next upgrade cycle. The price performance ratio compared to single monolithic boxes (of whatever flavour) is killer in this scenario.

I bet 10,000 quatloos that the Linux kernel running on that box isn't something you can get your paws on...

http://oss.sgi.com/projects/sgi_propack/

I'd like a tripple breasted whore with my quatloos please.

Re: Wayne Scotting
by Bascule on Wed 22nd Oct 2003 01:19 UTC

I bet 10,000 quatloos that the Linux kernel running on that box isn't something you can get your paws on...

As Linux is licensed under the GPL, this would most likely be illegal. I don't believe it would be possible for Linux to operate effectively/at all on this system without directly altering the kernel source.

More than likely SGI is either giving away or selling the source code (selling would be an interesting scenario, as they can only sell it for however much they're factoring the cost of Linux itself into the Altix)

Bascule's comments
by Tim on Wed 22nd Oct 2003 03:00 UTC

It seems that someone can't handle an informed analysis of the posted article. What exactly was it about Bascule's comments that was abusive? I'm a little curious.

A bit of lawyering...
by Kobold on Wed 22nd Oct 2003 03:04 UTC

As Linux is licensed under the GPL, this would most likely be illegal. I don't believe it would be possible for Linux to operate effectively/at all on this system without directly altering the kernel source.
GPL states that if someone gives you the moidifed binary, they have to provide you the source. In this case, only NASA has right to demand the access to modified kernel code.

:re Tim
by Christopher X on Wed 22nd Oct 2003 03:11 UTC

I agree, who's doing that to Bascule? The guys opinionated, but he's no troll.

Kobold
by Bascule on Wed 22nd Oct 2003 04:04 UTC

GPL states that if someone gives you the moidifed binary, they have to provide you the source.

Correct

In this case, only NASA has right to demand the access to modified kernel code.

Careful, I don't think you have enough information to draw that conclusion. You would have to know if SGI is charging anything for Linux on the Altix. If they are, then under the "equivalent access" clause of the GPL (http://www.gnu.org/licenses/gpl-faq.html#DoesTheGPLAllowDownloadFee) they must make the source available for a price equal to or less than what they're charging for the Linux binaries.

More lawyering...
by Kobold on Wed 22nd Oct 2003 05:00 UTC

http://www.gnu.org/licenses/gpl-faq.html#GPLRequireSourcePostedPubl...
--------------8<--------------
The GPL does not require you to release your modified version. You are free to make modifications and use them privately, without ever releasing them. This applies to organizations (including companies), too; an organization can make a modified version and use it internally without ever releasing it outside the organization.

But if you release the modified version to the public in some way, the GPL requires you to make the modified source code available to the program's users, under the GPL.

Thus, the GPL gives permission to release the modified program in certain ways, and not in other ways; but the decision of whether to release it is up to you.
--------------8<--------------
Therefore, the only thing they have to do is to provide NASA with the source of their kernel version.

Altrix development
by dubhthach on Wed 22nd Oct 2003 09:04 UTC

Interesting article i found there that appeard on linuxjournal back in jan. It's by the "Linux engineering director" at sgi
http://www.linuxjournal.com/article.php?sid=6440

Re: Kobold
by Bascule on Wed 22nd Oct 2003 20:34 UTC

The GPL does not require you to release your modified version. You are free to make modifications and use them privately, without ever releasing them. This applies to organizations (including companies), too; an organization can make a modified version and use it internally without ever releasing it outside the organization.

But if you release the modified version to the public in some way, the GPL requires you to make the modified source code available to the program's users, under the GPL.


Please click this link: http://www.gnu.org/licenses/gpl-faq.html#DoesTheGPLAllowDownloadFee

You will clearly see that if you sell binaries of a GPL program, you must make the source available for an equal or lesser price.

Therefore, the only thing they have to do is to provide NASA with the source of their kernel version.

If they are selling the kernel binaries, they must make the source available to anyone for an equal or lesser cost. See the link.

Of course, this is all moot given this: http://oss.sgi.com/projects/sgi_propack/

...although I don't see the high availability tools there. How does one hot swap a defective CPU or RAM? Is this even possible on the Altix, or would a defective CPU or RAM bring down the entire kernel?