Linked by Thom Holwerda on Tue 17th Sep 2013 22:04 UTC, submitted by garyd
General Development

ZFS is the world's most advanced filesystem, in active development for over a decade. Recent development has continued in the open, and OpenZFS is the new formal name for this open community of developers, users, and companies improving, using, and building on ZFS. Founded by members of the Linux, FreeBSD, Mac OS X, and illumos communities, including Matt Ahrens, one of the two original authors of ZFS, the OpenZFS community brings together over a hundred software developers from these platforms.

ZFS plays a major role in Solaris, of course, but beyond that, has it found other major homes? In fact, now that we're at it, how is Solaris doing anyway?

Permalink for comment 572777
To read all comments associated with this story, please click here.
RE[6]: Solaris is doing well
by Kebabbert on Sun 22nd Sep 2013 14:20 UTC in reply to "RE[5]: Solaris is doing well"
Kebabbert
Member since:
2007-07-27

Actually I meant this in the context of SMP versus NUMA. You said "All 32 socket Unix servers share some NUMA features, but they have very good RAM latency, so you treat them all as a true SMP server". I'd really like to know the difference between x86 NUMA and "Unix server true SMP", since as far as I know SMP requires NUMA in order to scale efficiently above 4-8 cores without very high memory contention. Saying that Solaris servers are different sounds an awful lot like marketing speak, but maybe I'm wrong. Can you point out a tangible technical difference?

Here is some information on these "different Solaris servers". I mean they are different, because they are well built, minimizing memory latency. Look at the last picture at the bottom:
http://www.theregister.co.uk/2012/09/04/oracle_sparc_t5_processor/

"...Turullols said you need one hop between sockets to scale. It usually takes two hops to get to eight-way NUMA given current designs, so this is where that near linear scalability is coming from...."

You see that this SPARC T5 8-socket server is connected to every other cpu, via 28 lanes. And a cpu can reach any memory cell in at most one jump - which means latency is very low. This is the reason this 8-socket server scales linearly. There are many 8-socket servers where you only get 5 cpus out of 8, or so. They scale bad with many hops.

Now imagine the Oracle M6 server with 96 cpus connected to each other, there would be 4560 lanes. That is too much and messy. So how to build a 96 socket server that scales well? Look at the bottom picture on the coming M6 server:
http://www.theregister.co.uk/2013/08/28/oracle_sparc_m6_bixby_inter...

Bronek:"...If you look at the picture carefully you will find that all CPUs can connect to others directly (7 cores), via single BX (12 cores) or a BX and a CPU (i.e. single hop, remaining 12 cores). This all with 4Tb/s bandwidth to maintain cache coherency across sockets - I think that's some really nice engineering..."

So, this M6 server has all cpus connected to another in only a few hops at worst case. It looks like the latency will be a few 100ns at worst. On the other hand, a HPC cluster have worst case latency of 10.000ns - which makes them only usable for parallel workloads where you dont need to access data far away.

This M6 server is for running huge nonclustered database configurations, all from memory. Oracle is concerned with the SAP Hana memory database, and this is Oracle's answer: a huge SMP-like server capable of running everything from RAM. So SAPs Hana RAM database is not a threat to Oracle's database. Thinks Larry Ellison.

This M6 server is very intricate built as we all can see. The are no other vendor building large database SMP servers with sockets more than 32, than Oracle and Fujitsu (the new 64-socket SPARC64 server M4-10s). As far as I know. HP has a 64 socket server, but it is old and not updated. I dont know if it is sold longer.

Anyway, you will not see a Linux NUMA cluster server running non clustered databases.




I thank you for looking these up. I really wish they were using *identical* hardware and only switching a single variable between tests (instead of switching the OS AND the hardware vendor).

Here are the 8-socket Solaris vs Linux SAP benchmarks I talked of. They use very similar hardware, opteron cpus of almost the same model, but Linux uses higher clocked. Linux has 128GB RAM and Solaris 256GB, because the Linux HP benchmarking team wanted to use faster RAM memory sticks, so they had to use 128GB RAM. Solaris uses slower memory sticks.
download.sap.com/download.epd?context=B1FEF26EB0CC34664FC7E80B933FCCAC 80DD88CBFAF48C8D126FB65D80D09E988311DE75E0922A14

download.sap.com/download.epd?context=40E2D9D5E00EEF7CCDB0588464276DE2 F0B2EC7F6C1CB666ECFCA652F4AD1B4C




This shows a glaring scalability problem with RHL. We're left to infer that RHL has a scalability problem compared to the Solaris chart on the same page.
http://blogs.oracle.com/jimlaurent/resource/HPDL980Chart.jpg

However another chart on a different blog post (on different hardware) doesn't show the scalability problem under RHL.
http://blogs.oracle.com/jimlaurent/resource/HPML350Chart.jpg

There are not the same scalability problem, but it has other problems. The Linux graph is very stuttering and not smooth. Linux struggles with the workload, and is very stuttery. Solaris is not.



As before, I'm not asserting that Solaris isn't better, it very well may be, but it would be naive to trust Oracle sources at face value.

Linux has never been tested on larger servers than 8-sockets, so I would be very surprised if Linux could scale well. But yes, I agree you need to be careful with Oracle marketing, too. I prefer independent benchmarks. If they dont exist, we can do nothing. But still, the Oracle benchmarks shows huge performance advantages to any other cpu or OS. I expect Oracle could tweak benchmarks slightly, but not completely? It should not be possible to make a lousy cpu look great? Or?



Can you disclose whether you are connected to oracle?

Sure. I am not connected to Oracle in any way. I work in finance, not IT. I just happen to be a geek liking the best tech out there. I admire good tech. I like the IBM POWER7 when it was released because it was the best, back then, better than SPARC. And I said so, in posts, yes. I acknowledged the superiority of POWER7, back then. I am also a fan of Plan9. In my opinion it might be the most innovative OS of them all. I prefer GO, to Java. etc. I just like the best tech. It does not really matter who is doing it. OpenBSD for security. Solaris for being the most innovative Unix. SPARC for the fastest cpu. ZFS for the safest fileystem. etc. If BTRFS would be better than ZFS, I would switch and dont look back. I am pragmatic, prefer the best tech.

But I dont like lies and FUD. To that I react and I want to dispel FUD.
-IBM Mainframes have very weak cpus, they are not strong. No matter what IBM says.

-Linux scales quite bad. No matter what Linus Torvalds say.

-Linux code quality is non optimal. Which Torvalds and other kernel devs agrees on. Here is what Con Kolivas, the famous Linux kernel developer, says when he compares source code quality of Solaris to Linux:
http://ck-hack.blogspot.se/2010/10/other-schedulers-illumos.html

http://www.forbes.com/2005/06/16/linux-bsd-unix-cz_dl_0616theo.html

http://www.theregister.co.uk/2009/09/22/linus_torvalds_linux_bloate...

Reply Parent Score: 2