Linked by Thom Holwerda on Tue 17th Sep 2013 22:04 UTC, submitted by garyd
General Development

ZFS is the world's most advanced filesystem, in active development for over a decade. Recent development has continued in the open, and OpenZFS is the new formal name for this open community of developers, users, and companies improving, using, and building on ZFS. Founded by members of the Linux, FreeBSD, Mac OS X, and illumos communities, including Matt Ahrens, one of the two original authors of ZFS, the OpenZFS community brings together over a hundred software developers from these platforms.

ZFS plays a major role in Solaris, of course, but beyond that, has it found other major homes? In fact, now that we're at it, how is Solaris doing anyway?

Thread beginning with comment 572497
To read all comments associated with this story, please click here.
Solaris is doing well
by Kebabbert on Thu 19th Sep 2013 10:51 UTC
Kebabbert
Member since:
2007-07-27

Sun had 30.000 customers. Oracle has 340.000 customers. Oracle is betting heavily on Solaris, for high end large >32 socket servers. Larry Ellison said that "Linux is for low end, Solaris for high end". Oracle owns the whole stack now: hardware (SPARC), OS (Solaris), middleware (Java), applications (database). So Oracle is adding things into the stack that boosts the Oracle database, for instance Solaris is 17% faster than other OSes running Oracle Database. The Database is the heart of every company, without databases you can not run a company. The most important piece at a company is the database. Oracle owns the database market (80% or so).

So, Oracle is now boosting Oracle Database throughout the whole stack. The results shows in latest TPC-C and TPC-H database benchmarks: Oracle is much much much faster than anyone else, including IBM POWER7, Intel Xeon, HP, etc. The fastest database servers in the world are from Oracle, here are several database benchmarks:
https://blogs.oracle.com/BestPerf/entry/21030612_sparc_t5_4_tpc

So, companies are not really interested in a certain technology, instead they are interested in solutions. If Oracle has a Database solution (running Solaris on SPARC) that is much much much faster than anyone else, the companies will buy it. Management dont care about tech talk. They want to see fast results. Oracle has 10x performance/price advantage over IBM POWER7, for instance. Just check the official benchmarks.

Also, Oracle is targeting large servers, serving many clients with extreme throughput. Which means Oracle is also fastest at Java server work loads, SAP, etc. All types of server work loads. Just read the link above.



Why said Larry Ellison that "Linux is for low end"? Why not use Linux for high end as well? Well, the largest Linux servers today, are 8-socket servers. There have never existed larger Linux servers. You can not find a 16 socket Linux server, for instance. No one has ever sold a 16 cpu Linux server. How can Linux kernel developers improve scalability on Linux servers, when the Linux hardware does not exist? How can they test their code? The largest Linux servers they can test their code on, is a common 8-socket x86 server from Dell, HP, IBM, etc.

Sure, there are SGI UV1000 servers with 10.000s of cores and loads of RAM, but it is a cluster consisting of several PCs on a fast switch. Also, the ScaleMP Linux server with 1000s of cores, is also a cluster. So, all Linux servers with 1000s of cores are clusters. They are all NUMA servers, and NUMA servers are clusters:
http://en.wikipedia.org/wiki/Non-uniform_memory_access#NUMA_vs._clu...
"...One can view NUMA as a tightly coupled form of cluster computing...."

NUMA servers are primarily used for HPC number crunching, and can not run databases in large configurations, the database performance would be extremely bad. So no Enterprise companies use NUMA servers, because they are primarily interested in databases. Only rendering farms use NUMA servers from SGI or ScaleMP to do number crunching.

Sure, you can run a distributed database over all nodes in a HPC cluster - but you can not run a monolithic database such as Oracle on a HPC cluster. So if you ever see a database benhcmark on large NUMA clusters it must be a distributed database, Uridium, Virtual Void. You would never see a Oracle database running on SGI UV1000 cluster, for instance.


If you look at a Solaris or IBM AIX server with 32 sockets, it is built as a SMP server. SMP servers are a "single, huge server", not a HPC cluster. The difference between a HPC cluster and SMP server, is the RAM latency. The HPC cluster has >10.000ns in worst case RAM latency, because data cells might be far away in another node - so you need to make sure your data are close in adjacent nodes, etc.

SMP servers have worst case latency of a few 100ns, so you program clusters totally different from a true SMP server. An SMP server is just an ordinary server, and you dont need to redesign your software, just program as normal. Just copy your normal binaries to the SMP server and run it. If you try to copy normal binaries to a HPC cluster, it will not work because you need to redesign software so data is close to adjacent nodes, etc - otherwise performance will be very bad.

You can not run normal software on a HPC cluster, it can only run a certain type of software with embarassingly parallel workloads. The big difference is RAM latency. Clusters have very bad worst case latency, an SMP server has good worst case latency.

The new Oracle M6 server next year, has 96 sockets, with 9.216 threads and 96TB RAM. It is not a true SMP server, it shares some NUMA features - but worst case latency is very very good. So you treat it simply like a true SMP server, no need to treat it as a HPC cluster. It is designed to run Oracle database in huge configurations, all from RAM. So SAP HANA memory database will not be a threat, Larry said.

(All 32 socket Unix servers share some NUMA features, but they have very good RAM latency, so you treat them all as a true SMP server).

So, for high end we will continue to find IBM and Oracle and HP with 32 socket servers. For low end with 1-8 socket servers there will be Linux and Windows. Until Linux handles larger than 8-socket servers, Linux will never venture into high end enterprise. So Linux scalability continues to be a myth.

Reply Score: 3

RE: Solaris is doing well
by arsa on Fri 20th Sep 2013 08:28 in reply to "Solaris is doing well"
arsa Member since:
2009-10-26

You seem to have some insight into Oracle businees plans which is not just a mere forecast. That's fine and well. What is not fine and well is stating "GNU/Linux is for low-end only" and "GNU/Linux scalability is a myth".

Oracle presents its Engineered Systems as the best machine they have. When I remember the way Oracle is marketing this machine (OpenWorld keynotes, "Oracle Magazine" and, oh dear, trailers for IronMan movies) it's clear they consider it a big deal. Description certainly seem impressive: http://www.oracle.com/us/products/engineered-systems/index.html
Should we also describe what operating systems they run on?
- Exadata, Database appliance, Exalytics, Big data appliance, Network applications platform run on Oracle Linux
- Exalogic can either run on Oracle Linux or Solaris
- Supercluster and ZFS storage applicance run on Solaris
- Virtual compute applicance is running Oracle VM but I don't know much about it
It appears that if it weren't for the facts that SPARC can be properly used in Solaris only and ZFS storage OS was already nicely built, then GNU/Linux is the only choice.

Now, the question for you. You are certainly aware of the expandibility of Engineered Systems. Can you explain how come GNU/Linux is not scalable? Were you able to deduce that from Oracle's behaviour?

Once again, I am all for the success of Solaris, but I'm even more against false claims over GNU/Linux.
All the best.

Edited 2013-09-20 08:30 UTC

Reply Parent Score: 2

RE[2]: Solaris is doing well
by Kebabbert on Fri 20th Sep 2013 11:58 in reply to "RE: Solaris is doing well"
Kebabbert Member since:
2007-07-27

You seem to have some insight into Oracle businees plans which is not just a mere forecast. That's fine and well. What is not fine and well is stating "GNU/Linux is for low-end only" and "GNU/Linux scalability is a myth".

Larry Ellison said that Linux is for lowend in an official interview. Read here for Oracle's official stand on Linux vs Solaris:
http://searchenterpriselinux.techtarget.com/news/1389769/Linux-is-n...


Oracle presents its Engineered Systems as the best machine they have. When I remember the way Oracle is marketing this machine ... Now, the question for you. You are certainly aware of the expandibility of Engineered Systems. Can you explain how come GNU/Linux is not scalable? Were you able to deduce that from Oracle's behaviour?

Oracle's engineered systems are really good, and they surely run Linux and/or Solaris. However, these systems are small. For instance the ExaData has only a few cpus. The ExaLogic is a cluster:
http://en.wikipedia.org/wiki/Oracle_Exalogic
And as well all know, Linux scales excellent in clusters. No one has ever denied that.

The question is if Linux scales on a single fat server. And Linux does not scale on such a SMP server, because there have never existed such a Linux server with 16 or 32 cpus for sale. There are Linux clusters for sale, with 1000s of cores, but no 16/32 cpu Linux server for sale. So, if there does not exist any 32 socket Linux servers, how can Linux scale well on SMP servers? They dont even exist, how can anyone even benchmark and assess the Linux scalability?

So, the Oracle engineered systems running Linux are either tiny (up to 8 sockets) or they are a cluster. The Oracle Engineered systems also exist in a modified Solaris version, and they are called "Supercluster". The engineered systems were developed before Oracle bought Sun, and that is the main reason they mostly run Linux. "Supercluster the fastest engineered system at Oracle":
http://www.oracle.com/us/products/servers-storage/servers/sparc/sup...

The new Oracle M6 server will have more than 32 sockets, it will have 96 sockets. And it will run Solaris, not Linux. Because Linux can not scale beyond 8-sockets in a single fat server (SMP).


Once again, I am all for the success of Solaris, but I'm even more against false claims over GNU/Linux.
All the best.

These are not false claims. There have never existed a single fat Linux server with more than 8 cpus for sale. The largest Linux server ever sold, has 8 cpus, and is just a normal HP, DELL, IBM server x86. I suggest you google for larger Linux servers, you will not find any. Thus, I speak true. The proof I speak true is simple: no one has ever sold larger Linux servers than 8 cpus. If you find a larger Linux server, then I am wrong. You will not even find benchmarks on a large Linux server. It is like IBM, they never release benchmarks on their large IBM Mainframes, because Mainframes have really slow cpus, much slower than x86 cpus. The IBM POWER7 is a good cpu, though.

Linux scales excellent in clusters, up to 10.000s of cores and maybe even beyond that.

Linux scales very bad in SMP servers, up to 8 sockets. HP tried to compile Linux to their big 64-socket Unix server, and Linux failed miserably. Linux had ~40% cpu utilization, which means most cpus were idle when running on a 64 cpu server. That is quite bad. Google for "HP Big Tux Linux server" for more information.

Reply Parent Score: 1

RE: Solaris is doing well
by znby on Sun 22nd Sep 2013 05:56 in reply to "Solaris is doing well"
znby Member since:
2012-02-03

SMP servers have worst case latency of a few 100ns, so you program clusters totally different from a true SMP server. An SMP server is just an ordinary server, and you dont need to redesign your software, just program as normal. Just copy your normal binaries to the SMP server and run it. If you try to copy normal binaries to a HPC cluster, it will not work because you need to redesign software so data is close to adjacent nodes, etc - otherwise performance will be very bad.


SMP does not scale to any appreciable core count. In fact, on individual CPUs with large core counts (like the 16-core Interlagos Opterons) you will find that certain cores will be in one memory region, and other cores will be in other regions, and you need to concern yourself with this fact if you want to write code that makes optimal use of the chip. I also fail to see how something can have "features of NUMA". Either all CPUs can access memory at a latency that is unaffected by which address you are accessing, or the latency/access time is dependent on which region of memory you access, and it's a NUMA machine. You said yourself that these Sparc "SMP" machines have different latencies depending on which memory region you access, which would imply that you do need to worry about NUMA issues on one of these machines if you really wish to get top performance.

Also, given that these are hardware/architectural issues, I fail to see what the choice of operating system is, or what exactly prevents Linux from being useful in this scenario. The choice of Solaris probably has more to do with vendor lock-in and the fact that Solaris has been optimized for Sparc since before Linux even existed...

Edited 2013-09-22 06:00 UTC

Reply Parent Score: 1

RE[2]: Solaris is doing well
by Kebabbert on Sun 22nd Sep 2013 14:26 in reply to "RE: Solaris is doing well"
Kebabbert Member since:
2007-07-27

"SMP servers have worst case latency of a few 100ns, so you program clusters totally different from a true SMP server. An SMP server is just an ordinary server, and you dont need to redesign your software, just program as normal. Just copy your normal binaries to the SMP server and run it. If you try to copy normal binaries to a HPC cluster, it will not work because you need to redesign software so data is close to adjacent nodes, etc - otherwise performance will be very bad.


SMP does not scale to any appreciable core count. In fact, on individual CPUs with large core counts (like the 16-core Interlagos Opterons) you will find that certain cores will be in one memory region, and other cores will be in other regions, and you need to concern yourself with this fact if you want to write code that makes optimal use of the chip. I also fail to see how something can have "features of NUMA". Either all CPUs can access memory at a latency that is unaffected by which address you are accessing, or the latency/access time is dependent on which region of memory you access, and it's a NUMA machine. You said yourself that these Sparc "SMP" machines have different latencies depending on which memory region you access, which would imply that you do need to worry about NUMA issues on one of these machines if you really wish to get top performance.
"
Yes, all modern SMP servers are somewhat NUMA, read my post above on the new Oracle M6 server. The point is that there is a huge difference in programming when worst case memory latency is a few 100ns, or 10.000ns. The latter must be treated as a cluster.


Also, given that these are hardware/architectural issues, I fail to see what the choice of operating system is, or what exactly prevents Linux from being useful in this scenario. The choice of Solaris probably has more to do with vendor lock-in and the fact that Solaris has been optimized for Sparc since before Linux even existed...

Benchmarks show that Linux has problems utilizing all cpus well, on as few as 8-sockets. Which is to be expected because there does not exist any larger Linux servers than 8 sockets. 8-socket servers are on the verge of what Linux can handle. Or do you mean that Linux would scale well even on 32 socket servers, when it had not even been tested nor optimized for such servers?

So we all agree that Linux can not scale well, when it stops at 8-sockets?

Reply Parent Score: 2