Linked by Thom Holwerda on Tue 17th Sep 2013 22:04 UTC, submitted by garyd
General Development

ZFS is the world's most advanced filesystem, in active development for over a decade. Recent development has continued in the open, and OpenZFS is the new formal name for this open community of developers, users, and companies improving, using, and building on ZFS. Founded by members of the Linux, FreeBSD, Mac OS X, and illumos communities, including Matt Ahrens, one of the two original authors of ZFS, the OpenZFS community brings together over a hundred software developers from these platforms.

ZFS plays a major role in Solaris, of course, but beyond that, has it found other major homes? In fact, now that we're at it, how is Solaris doing anyway?

Thread beginning with comment 572620
To view parent comment, click here.
To read all comments associated with this story, please click here.
RE[3]: Solaris is doing well
by Alfman on Fri 20th Sep 2013 14:56 UTC in reply to "RE[2]: Solaris is doing well"
Alfman
Member since:
2011-01-28

Kebabbert,

"Larry Ellison said that Linux is for lowend in an official interview. Read here for Oracle's official stand on Linux vs Solaris:"

I was unable to find the interview in your link (membership required?) Of course he would say that though, he's biased. Bill Gates would have said the same thing regarding windows. They are not really going to admit that their products can be replaced by much cheaper linux alternatives, even if it were true. I hope you understand why asking Larry Ellison is meaningless. Consider his claim here:

http://www.x-drivers.com/news/hardware/4900.html
"Oracle claimed a legion of record-breaking benchmark performances for the T4-4. Ellison repeatedly compared the performance of the T4-based Sparc SuperCluster to IBM's Power line—and the Power 795 in particular. A one-rack T4 SuperCluster 'is twice as fast as IBM's fastest computer, at half the cost,' he claimed."

When in fact... "But the benchmarks that Oracle cited were mostly internal ones. Those may carry some weight for many Oracle customers, but there were only two that really hint at the T4-4's performance beyond software that has been tuned for that processor. One of those third-party benchmarks was the TPC-H benchmark for a 1,000 GB load, in which the T4-4 beat the IBM Power 780 and Itanium-based HP Superdome 2 on price/performance, raw performance, and throughput."

and

"The T4 is still outperformed on Oracle Database 11g by HP's BladeSystem RAC configuration running Oracle Linux, and edged out by HP's Proliant DL980 G7 running Microsoft SQL Server 2008 and Windows Server 2008 both on price performance and raw power. Both are x86 systems."

I'm not sure whether the additional cores with Sparc are actually beneficial. I just wasn't able to find much information about it.


"The question is if Linux scales on a single fat server. And Linux does not scale on such a SMP server, because there have never existed such a Linux server with 16 or 32 cpus for sale. There are Linux clusters for sale, with 1000s of cores, but no 16/32 cpu Linux server for sale. So, if there does not exist any 32 socket Linux servers, how can Linux scale well on SMP servers? They dont even exist, how can anyone even benchmark and assess the Linux scalability? "


Do you know if it's *really* a linux problem instead of an x86 SMP scalability problem? I honestly don't think x86 can scale efficiently beyond 8 cores under any OS. My understanding is that linux will run on the same Sparc architectures that Solaris does:

http://www.phoronix.com/scan.php?page=news_item&px=MTE5Nzc

Do you have a benchmark of an apples to apples comparison between solaris and linux on the same processors (ignoring that such processors are not being sold with linux)?


Mind you solaris *could* be better than linux for high end deployments. I'm genuinely curious about it, and if you have any evidence (benchmarks & case studies) that would be very informative to me.

For that matter, I'm very curious about the scalability of 64 core shared memory systems in general regardless of OS. Correct me if I'm wrong, but it seems to me that it would scale badly unless it were NUMA (or it had so much cache that it could effectively be used as NUMA).

It's fun to talk to others who are passionate about this stuff!

Edited 2013-09-20 15:02 UTC

Reply Parent Score: 3

RE[4]: Solaris is doing well
by Alfman on Fri 20th Sep 2013 15:33 in reply to "RE[3]: Solaris is doing well"
Alfman Member since:
2011-01-28

What I meant to quote before was this, but I was too hasty.

"some of the claims Oracle made about its performance on third-party benchmarks were based on selected interpretations of the data—which drew catcalls from IBM Systems and Technology chief technical strategist Elisabeth Stalh. "Oracle claimed nine T4 world records. 7 of the 9 are not industry standard benchmarks but Oracle’s own benchmarks, most based on internal testing," Stahl blogged. "Oracle’s SPECjEnterprise2010 Java T4 benchmark result, which was highlighted, needed four times the number of app nodes, twice the number of cores, almost four times the amount of memory and significantly more storage than the IBM POWER7 result."

It does suggest that Larry Elisson's claims should be taken with a grain of salt. Although in fairness to him, everyone else probably does the same thing.

Reply Parent Score: 3

RE[4]: Solaris is doing well
by Kebabbert on Sat 21st Sep 2013 10:43 in reply to "RE[3]: Solaris is doing well"
Kebabbert Member since:
2007-07-27

"The T4 is still outperformed on Oracle Database 11g by HP's BladeSystem RAC configuration running Oracle Linux, and edged out by HP's Proliant DL980 G7 running Microsoft SQL Server 2008 and Windows Server 2008 both on price performance and raw power. Both are x86 systems."

The SPARC T4 has a maximum of 4 sockets, there are no larger T4 servers. And as such, is a tiny server. The HP bladesystem is a RAC cluster, consisting of several PCs. It is easier to beat a single server if you are using clusters. The Windows SQL Server 2008 is a small server. Sure it competes with SPARC T4 on 4-socket servers, but the difference is that Windows SQL is topped out, whereas Oracle Database can continue to scale even to very large servers. Just because you can compete on 1-4 sockets, does not mean you can keep compete when you go above 8 sockets. I am convinced Windows SQL has scaling problems and you will see steep deterioration above 4 sockets or so. But on 1-4 sockets I think Windows SQL does excellent, because MS has optimized and tested the code for few sockets. It is difficult to scale to many sockets and keep the performance intact. There is a scaling problem.

But I agree that the SPARC T4 was not that good on all benchmarks because it stopped at four sockets. T4 is more of a server cpu, built to handle many threads and many users at a high throughput. A desktop cpu has few strong threads, so it can not service many many clients, only few of them at a time. On databases though, the T4 was very good. Oracle is a database company, so that is to be expected.

Regarding "Oracles internal benchmarks" said the IBM spokesperson. Sun Microsystems SPARC cpus had several world records running Oracle software, long before Oracle bought Sun. Oracle is an important enterprise software provider, so IBM and Sun and many others benchmarked Oracles software, "Peoplesoft", etc. And SPARC was always much faster than IBM POWER, because SPARC is a server cpu handling many clients. For instance, you needed 14 (fourteen) IBM POWER6 cpus to match 4 SPARC T2 cpus in Peoplesoft v8.0 benchmarks back then (Google "Peoplesoft v8 benchmarks" and compare IBM to Sun). SPARC T2 crushed IBM back then, in many server benchmarks which all favours high throughput. Some years later, Oracle bought Sun, and guess what? SPARC still continued to crush IBM on the same Oracle benchmarks, Peoplesoft, etc. But because IBM still could not beat SPARC in high throughput benchmarks using Oracle software, IBM changed strategy and said "no, we will not compete on Oracle benchmarks any more". But SPARC has always crushed IBM POWER on those benchmarks, even long before Oracle owned SPARC. Ergo, "internal benchmarks" is just FUD from IBM.



You should know that IBM is FUDing a lot and according to wikipedia, it was actually IBM that started to employ FUD on a wide systematic scale:
http://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt#Definition




The latest SPARC T5 cpu is twice as fast as the T4. And the SPARC T5 has twice the number of sockets, up to 8. So T5 servers are four times faster. And very good at databases, and also has the world record in many server benchmarks. It is almost three times as fast as IBM POWER7 on some database benchmarks. T5 utterly crushes IBM POWER7, IBM can not compete anymore.

And guess what IBM is saying in response to the worlds fastest cpu T5? "We dont understand why Oracle is talking about how fast cpus they have, no one is interested in cpu performance any longer. Talking about cpu performance is so 2000ish". I am convinced that if the coming IBM POWER8 is good, IBM will start to brag a lot how fast the POWER8 is. ;)
http://blogs.wsj.com/digits/2013/03/27/ibm-fires-back-at-oracle-aft...

“...[performance] was a frozen-in-time discussion,” Parris said in an interview Wednesday. “It was like 2002–not at all in tune with the market today.”...Companies today, Parris argued, have different priorities than the raw speed of chips. They are much more concerned about issues like “availability”–resistance to break-downs–and security and cost-effective utilization of servers than the kinds of performance numbers Ellison throws out..."

My point is that when IBM can not compete, IBM will start to FUD. As IBM always have done.




For instance, the largest IBM Mainframe with 24 sockets, is claimed to replace up to 1.500 x86 servers. But, if you happen to know that IBM Mainframe cpus at 5.26GHz are much slower than a decent x86 cpu, you will start to wonder. If you dig a bit, it turns out that all x86 servers are idling, and the Mainframe is 100% loaded! How can a Mainframe cpu replace even a single x86 cpu?? Let alone 1.500 x86 cpus! My point is that you should be cautious what famous FUDing company IBM says. The largest IBM Mainframe has 50.000 MIPS, which corresponds to 200.000 MHz x86 (see link below). An 10-core x86 cpu at 2GHz equals 20.000MHz. How many x86 cpus are needed to reach 50.000 MIPS? So, are the IBM Mainframes so fast as IBM claims, or is it only FUD?
http://www.mail-archive.com/linux-390@vm.marist.edu/msg18587.html




Another example of IBM FUD is this. The POWER7 has few but strong cores. The SPARC T5 has many but weaker cores. IBM says: POWER7 core is faster - this is true in some benchmarks. Therefore the POWER7 cpu is faster - this is false. Just because IBM has stronger cores, it does not make the entire cpu stronger. Imagine IBM has only one strong core, and SPARC has 1000 slightly weaker cores, which cpu is fastest you recon?

IBM claims that Oracle SPARC T5 world records are false, that SPARC T5 is not the worlds fastest cpu. Oracle is lying says IBM. Nothing is correct, says IBM:
http://whywebsphere.com/2013/04/29/weblogic-12c-on-oracle-sparc-t5-...
"....Oracle announced their new SPARC T5 processor with much fanfare and claiming it to be the “fastest processor in the world”. Well, perhaps it is the fastest processor that Oracle has produced, but certainly not the fastest in the world. You see, when you publish industry benchmarks, people may actually compare your results to other vendor’s results. This is exactly what I would like to do in this article.

...Being “fastest processor in the world” means that such processor must be able to handle the most transactions per second per processor core

...However IBM produced the world record result in terms of EjOPS per processor core – truly a measure of the fastest processor known to men..."

Here IBM claims that because IBM has stronger cores and therefore beats the SPARC T5 cpu. Well, the T5 has twice the number of cores compared to the POWER7 cpu. So the T5 cpu is actually faster. So IBM is FUDing again.
https://blogs.oracle.com/jhenning/entry/ibm_transactions_per_core

And actually, SPARC T5 core is faster. For instance in TPC-C and TPC-H benchmarks, the T5 core is 24% faster than IBM POWER7.

Reply Parent Score: 2

RE[5]: Solaris is doing well
by Alfman on Sat 21st Sep 2013 14:46 in reply to "RE[4]: Solaris is doing well"
Alfman Member since:
2011-01-28

Kebabbert,

Wow, that's a long post, I'll just respond to the main point.

"My point is that when IBM can not compete, IBM will start to FUD. As IBM always have done."

I wasn't denying that bias&FUD exists on the part of IBM, surely it does...however I'd take all statements with a pinch of salt, whether they've from IBM, Oracle (HP, SGI or anybody else).

I'll always trust third party benchmarks over the "horse's mouth". Unfortunately with solaris, oracle are actually guilty of banning third parties from publishing solaris benchmarks. While this by itself doesn't imply that their stuff performs badly, it does imply that oracle wants to hide the truth and that we only get to see benchmarks of their choosing.


http://www.phoronix.com/scan.php?page=news_item&px=ODc4Nw

Reply Parent Score: 2

RE[4]: Solaris is doing well
by Kebabbert on Sat 21st Sep 2013 11:05 in reply to "RE[3]: Solaris is doing well"
Kebabbert Member since:
2007-07-27

Do you know if it's *really* a linux problem instead of an x86 SMP scalability problem? I honestly don't think x86 can scale efficiently beyond 8 cores under any OS.

You mean to say x86 does not scale beyond 8 sockets, not 8 cores. Sure, there are no larger SMP servers than 8 sockets x86 today, and has never been. SGI UV1000 server, is actually a NUMA cluster with 100s of sockets x86. But it is HPC cluster, so it is ruled out from this discussion because we discussing SMP servers, not HPC clusters.

Regarding scalability of Linux. If you look at SAP benchmarks using 2.8GHz Opteron cpus and fast RAM sticks, Linux has 87% cpu utilization on 8-socket server. That is quite bad cpu utilization. Solaris on same opteron cpus but slower at 2.6GHz and slower RAM sticks, gets 99% cpu utilization and beats Linux on SAP benchmarks. Solaris is using slower hardware, and beats Linux. 8-sockets is the maximum what Linux has been tested for, and Linux does not handle 8-sockets well. Linux Big Tux with 64 sockets, had 40% cpu utilization shows experimental benchmarks from HP, so HP could not sell Big Tux. So with 8 sockets, Linux had 87% cpu utilization, at 64 sockets Linux had ~40%. I guess at 16 sockets Linux will have cpu utilization at 70%, and rapidly fall off. Because Linux has not been tested nor optimized to handle 16 sockets - how could Linux scale 16 sockets well, the hardware does not exist!


My understanding is that linux will run on the same Sparc architectures that Solaris does:

Yes, it will. But how well? HP tried Linux for their 64 socket server, with awful results. I believe Linux will stutter and be very bad at 96 socket SPARC servers.


Do you have a benchmark of an apples to apples comparison between solaris and linux on the same processors (ignoring that such processors are not being sold with linux)?

There are benchmarks with Linux and Solaris on same or similar x86 hardware. On few cpus, Linux tends to win. On larger configurations, Solaris wins. But that is expected, because all Linux kernel devs sit with 1-2 socket PCs at home. Not many Linux devs has access to 8 socket servers. Linux vs Solaris on same hardware:
https://blogs.oracle.com/jimlaurent/entry/solaris_11_outperforms_rhe...

https://blogs.oracle.com/jimlaurent/entry/solaris_11_provides_smooth...



Mind you solaris *could* be better than linux for high end deployments.

There does not exist high end Linux deployments, and has never had. So for high end deployments, you have no other choice than go to Unix servers with 32 sockets or larger, from IBM, Oracle or HP. So I would be very very surprised if Solaris not was better than Linux. Unix kernel devs have for decades tested and tailored the kernel for 32 sockets and above - of course Unix must handle large servers better?


I'm genuinely curious about it, and if you have any evidence (benchmarks & case studies) that would be very informative to me.

People routinely runs Unix on large 32 sockets, or larger, for decades. So Unix should be comfortable running large servers without effort, I suspect.


For that matter, I'm very curious about the scalability of 64 core shared memory systems in general regardless of OS. Correct me if I'm wrong, but it seems to me that it would scale badly unless it were NUMA (or it had so much cache that it could effectively be used as NUMA).

The canonical example of a large SMP workload, is running databases in large configurations. As a kernel developer explains and talks about NUMA, SMP, HPC, etc:
http://gdamore.blogspot.se/2010/02/scalability-fud.html
"....First, one must consider the typical environment and problems that are dealt with in the HPC arena. In HPC (High Performance Computing), scientific problems are considered that are usually fully compute bound. That is to say, they spend a huge majority of their time in "user" and only a minuscule tiny amount of time in "sys". I'd expect to find very, very few calls to inter-thread synchronization (like mutex locking) in such applications...

...Consider a massive non-clustered database. (Note that these days many databases are designed for clustered operation.) In this situation, there will be some kind of central coordinator for locking and table access, and such, plus a vast number of I/O operations to storage, and a vast number of hits against common memory. These kinds of systems spend a lot more time doing work in the operating system kernel. This situation is going to exercise the kernel a lot more fully, and give a much truer picture of "kernel scalability"

Reply Parent Score: 2

RE[5]: Solaris is doing well
by Alfman on Sat 21st Sep 2013 16:00 in reply to "RE[4]: Solaris is doing well"
Alfman Member since:
2011-01-28

Kebabbert,

"You mean to say x86 does not scale beyond 8 sockets, not 8 cores. Sure, there are no larger SMP servers than 8 sockets x86 today, and has never been."

Actually I meant this in the context of SMP versus NUMA. You said "All 32 socket Unix servers share some NUMA features, but they have very good RAM latency, so you treat them all as a true SMP server". I'd really like to know the difference between x86 NUMA and "Unix server true SMP", since as far as I know SMP requires NUMA in order to scale efficiently above 4-8 cores without very high memory contention. Saying that Solaris servers are different sounds an awful lot like marketing speak, but maybe I'm wrong. Can you point out a tangible technical difference?

"There are benchmarks with Linux and Solaris on same or similar x86 hardware. On few cpus, Linux tends to win. On larger configurations, Solaris wins."

"Linux vs Solaris on same hardware:"


I thank you for looking these up. I really wish they were using *identical* hardware and only switching a single variable between tests (instead of switching the OS AND the hardware vendor).

Still, the benchmarks are interesting.

This shows a glaring scalability problem with RHL. We're left to infer that RHL has a scalability problem compared to the Solaris chart on the same page.
http://blogs.oracle.com/jimlaurent/resource/HPDL980Chart.jpg

However another chart on a different blog post (on different hardware) doesn't show the scalability problem under RHL.
http://blogs.oracle.com/jimlaurent/resource/HPML350Chart.jpg

So was the problem with Red Hat Linux, was it the hardware, OS, software, the number of cores? We really don't know. Surely any employee worth his salt would have performed the benchmarks in an apples to apples hardware/software configuration, why weren't those results posted?

As before, I'm not asserting that Solaris isn't better, it very well may be, but it would be naive to trust Oracle sources at face value.


"Consider a massive non-clustered database. In this situation, there will be some kind of central coordinator for locking and table access, and such, plus a vast number of I/O operations to storage, and a vast number of hits against common memory."

I'd think this design is suboptimal for scalability. A scalable design would NOT have a single central coordinator, there should be many (ie one per table or shard) running in parallel even though it's not clustered. To be optimal on NUMA software should be specifically coded to use it, however you are probably right that vendors are treating it as clustered nodes instead. They haven't gotten around to rewriting the database engines to take advantage of NUMA specifically.



Can you disclose whether you are connected to oracle?

Edited 2013-09-21 16:06 UTC

Reply Parent Score: 2