Linked by Thom Holwerda on Thu 27th Oct 2016 19:23 UTC
Linux

With the final major capability for BPF tracing (timed sampling) merging in Linux 4.9-rc1, the Linux kernel now has raw capabilities similar to those provided by DTrace, the advanced tracer from Solaris. As a long time DTrace user and expert, this is an exciting milestone! On Linux, you can now analyze the performance of applications and the kernel using production-safe low-overhead custom tracing, with latency histograms, frequency counts, and more.

Order by: Score:
Comment by FlyingJester
by FlyingJester on Thu 27th Oct 2016 19:46 UTC
FlyingJester
Member since:
2016-05-11

Ah, so Linux is finally on par with Solaris from 2006 ;)

Of course, it's not really DTrace and it's not even a compatible interface, so it's unclear to me just yet if it's really just as good, better, or worse.

Reply Score: 3

RE: Comment by FlyingJester
by acobar on Thu 27th Oct 2016 20:54 UTC in reply to "Comment by FlyingJester"
acobar Member since:
2005-11-15

Hey, you forgot about ZFS !

Seriously, Solaris was good on some things but horrible (on x86 hardware) on others. Give me Linux any day.

About ZFS, very nice capabilities for servers, a bit too demanding on Desktops (well, used to be, new Desktops are really fast and full of memory).

Reply Score: 4

RE[2]: Comment by FlyingJester
by FlyingJester on Thu 27th Oct 2016 21:11 UTC in reply to "RE: Comment by FlyingJester"
FlyingJester Member since:
2016-05-11

Well, much of the desktop problems with ZFS are self inflicted by not disabling active deduplication on machines with less memory (which could really be the default on installation, but most maintainers don't seem to really have a good handle on ZFS anyways).

On UltraSparc, Linux is just a huge mess since most distros just cross compile everything, and have very minimal actual testing. I'm mostly using OpenBSD there, since they provide a MUCH more stable and well supported environment on UltraSparc than any Linux distro does.

Reply Score: 3

ZFS deduplication
by rleigh on Thu 27th Oct 2016 22:45 UTC in reply to "RE[2]: Comment by FlyingJester"
rleigh Member since:
2014-11-15

I have never seen deduplication enabled by default on any system to date (FreeBSD, Linux). Which systems do this?

Reply Score: 2

RE[3]: Comment by FlyingJester
by Drumhellar on Thu 27th Oct 2016 23:10 UTC in reply to "RE[2]: Comment by FlyingJester"
Drumhellar Member since:
2005-07-12

The non-dedup memory requirements of ZFS have always been way overblown. I think its because original (very rough) estimates of needed memory included dedup, and as other people repeated it around the internet, they dropped the "dedup" part.

FreeBSD with ZFS works comfortably on systems with 1GB of RAM, and with tuning, can work fine on 512MB or even 256MB configurations (Though, that tuning isn't the default).

Reply Score: 2

RE[4]: Comment by FlyingJester
by FlyingJester on Thu 27th Oct 2016 23:50 UTC in reply to "RE[3]: Comment by FlyingJester"
FlyingJester Member since:
2016-05-11

Yeah, this. On Solaris 10 and FreeBSD, I've never once had an issue.

I've heard that this idea that ZFS has crazy memory requirements (and deduplication being always on) comes from older AUR packages of ZFS, which were set up to enable deduplication by default. I had thought that some other packages did the same by default, but I've never used it on Linux outside Gentoo, so I can't say if it really is that way or not.

In any case, I would wager that a vast majority of the people who say ZFS uses a ton of memory have never actually used ZFS to begin with.

Edited 2016-10-27 23:51 UTC

Reply Score: 1

RE[3]: Comment by FlyingJester
by tidux on Sat 29th Oct 2016 19:48 UTC in reply to "RE[2]: Comment by FlyingJester"
tidux Member since:
2011-08-13

Even FreeBSD is a crashy, unstable mess on sparc64. OpenBSD's strict no-cross-compiler policy is a godsend for my little Ultra 5.

Reply Score: 2

RE[2]: Comment by FlyingJester
by flanque on Thu 27th Oct 2016 21:45 UTC in reply to "RE: Comment by FlyingJester"
flanque Member since:
2005-12-15

I loved Solaris when I had to support it.

ZFS & Live Upgrade... ahh, I can sleep.

Reply Score: 2

RE: Comment by FlyingJester
by flanque on Thu 27th Oct 2016 21:48 UTC in reply to "Comment by FlyingJester"
flanque Member since:
2005-12-15

Well in fairness it does say, similar to those provided by DTrace.

Reply Score: 3

RE: Comment by FlyingJester
by sergio on Fri 28th Oct 2016 03:01 UTC in reply to "Comment by FlyingJester"
sergio Member since:
2005-07-06

Ah, so Linux is finally on par with Solaris from 2006 ;)


Now Oracle Solaris must follow the lead and be on par with 2006's Solaris! ;)

I don't know maybe We just had very bad luck at work... but what a mess were the latest releases of Solaris 11!!! We had dozens of really serious problems horrible bugs on production system.

I'm not a hater I've working with Solaris since early 2000s... never had so many problems as today. As I said, maybe it's just my experience, some people love Solaris 11... to me it's the worst release ever.

I always recommended Solaris over RHEL for mission critical system... I'm not sure about it anymore... SPARC hardware still is incredible and ultra reliable and way better than even the best x86 servers... but Solaris is a PITA. ;)

Reply Score: 2

RE: Comment by FlyingJester
by Rxme on Fri 28th Oct 2016 06:10 UTC in reply to "Comment by FlyingJester"
Rxme Member since:
2016-10-28

DTrace was integrated in Mac OS X 10.5 in 2007.

Reply Score: 1

RE[2]: Comment by FlyingJester
by FlyingJester on Fri 28th Oct 2016 17:29 UTC in reply to "RE: Comment by FlyingJester"
FlyingJester Member since:
2016-05-11

OS X also had ZFS support around the same time.

Reply Score: 1

Bill Shooter of Bul Member since:
2006-07-14

Yeah, not sure how many people are running mission critical servers on macs these days. At least since the complete abandonment of the xserves. Anyone know if dtrace still works on OSX ?

Reply Score: 2

RE[3]: Comment by FlyingJester
by Drumhellar on Fri 28th Oct 2016 22:09 UTC in reply to "RE[2]: Comment by FlyingJester"
Drumhellar Member since:
2005-07-12

It appears so.

Adam Leventhal (One of the three authors of dtrace) has a dtrace-centric blog, and he mentioned using it to investigate some of the new features of the Sierra beta, including APFS

http://dtrace.org/blogs/ahl/

Reply Score: 2

RE: Comment by FlyingJester
by AdamR01 on Sat 29th Oct 2016 03:07 UTC in reply to "Comment by FlyingJester"
AdamR01 Member since:
2005-09-14

Or you could use Oracle Linux which has had actual dtrace since 2011.

Reply Score: 1

v Hacking Help
by Tammy5 on Thu 27th Oct 2016 23:11 UTC
Comment by Drumhellar
by Drumhellar on Thu 27th Oct 2016 23:18 UTC
Drumhellar
Member since:
2005-07-12

So, they're ditching systemtap instead of fixing/extending it? I've seen people complain about how unstable it was, causing kernel panics in relatively common circumstances, but it seems that it would be better to fix that, rather than tell users that it's time to ditch the work they've put in and start from scratch.

Though, that does seem to be SOP for how Linux is developed - see a feature in Unix/BSD, copy it rather poorly, ditch it, then copy it again to be more similar to the way Unix/BSD did it.

Reply Score: 2

RE: Comment by Drumhellar
by acobar on Fri 28th Oct 2016 11:11 UTC in reply to "Comment by Drumhellar"
acobar Member since:
2005-11-15

Though, that does seem to be SOP for how Linux is developed - see a feature in Unix/BSD, copy it rather poorly, ditch it, then copy it again to be more similar to the way Unix/BSD did it.

Well, it does seems a bit unfair to Linux developers to put things this way and I am sure there are lots of good tech invented in Linux circle but, being frank, most of the technical background around the basics an OS must have, how it should be layered and how it should operate, was already reasonably established when Linux was created.

Also, as there are many Linux developers all around the world, with many of them shooting for glory, there is a kind of rush to implement new ideas and we see the result of this on how quickly things are created, rewritten and even substituted. You don't see, usually, this kind of vertiginous experience done on traditional Unix and I agree with you on that, it is more a kind of phenomenon you would meet on Linux side, not necessarily a bad thing, though.

Reply Score: 4

RE[2]: Comment by Drumhellar
by Kebabbert on Mon 31st Oct 2016 13:21 UTC in reply to "RE: Comment by Drumhellar"
Kebabbert Member since:
2007-07-27

"Though, that does seem to be SOP for how Linux is developed - see a feature in Unix/BSD, copy it rather poorly, ditch it, then copy it again to be more similar to the way Unix/BSD did it.

Well, it does seems a bit unfair to Linux developers to put things this way and I am sure there are lots of good tech invented in Linux circle but, being frank, most of the technical background around the basics an OS must have, how it should be layered and how it should operate, was already reasonably established when Linux was created.

Also, as there are many Linux developers all around the world, with many of them shooting for glory, there is a kind of rush to implement new ideas and we see the result of this on how quickly things are created, rewritten and even substituted. You don't see, usually, this kind of vertiginous experience done on traditional Unix and I agree with you on that, it is more a kind of phenomenon you would meet on Linux side, not necessarily a bad thing, though.
"
Well, the entire Linux kernel is a copy of Unix. What do you expect? Original tech from Linux? I dont know of any original tech coming from Linux? What does Linux have invented that every OS wants and have copied or ported?

OTOH, everybody wants ZFS and DTrace and Containers and Crossbow and SMF, and...

For instance, take DTrace:
IBM AIX has a copy called ProbeVue
FreeBSD has ported it
Mac OS X has ported it
VMware has a copy called vProbes
QNX has ported it
Linux has copied it badly, and call it Systemtap.
Every major OS have copied or ported ZFS.

BTRFS is a linux copy of ZFS and it failed too.

systemd is a copy of SMF, Lennart talked about Solaris SMF all the time back then.

Dockers is a variant of Containers

Crossbow is copied and called... forgot the name

etc etc. What does Linux have that every OS lust and desire after? Any software?

Reply Score: 2

RE[3]: Comment by Drumhellar
by oiaohm on Mon 31st Oct 2016 15:20 UTC in reply to "RE[2]: Comment by Drumhellar"
oiaohm Member since:
2009-05-30

https://www.usenix.org/legacy/events/lsf07/tech/rodeh.pdf

Both ZFS and Btrfs come out of the ideas of WALF
https://www.google.com/patents/US5819292
Yes NetApp patented the base idea that both ZFS and BTRFS use. ZFS does slab and BTRFS does b-tree implementation of the idea and I cannot remember what odd beast WALF does.

Dtrace and Systemtap is funny. Dtrace is started jan 2005 first prototypes of Systemtap starts as dprobes in jun 2004. So these two are parallel invention it possible Dtrace is copied the other way. Of course the idea of dprobes is based off extending strace into kernel space and strace was sun idea.

vprobes name is based of dprobes so that by vmware is not copy of Dtrace either.

Linux world does have habit of rename something and start over so the old interface can be deprecated.

What is be-compared as the Linux replacement to dtrace is http://www.brendangregg.com/blog/2016-03-05/linux-bpf-superpowers.h...
bpf trace tools. This in it self is interesting. Both systemtap and dtrace are are based around using a special language for tracing converted to native by user-space complier. Where bpf trace tools is recycling the Berkeley packet filter engine byte code to have kernel Berkeley packet filter jit in kernel turn it to native code.

So Linux developers as normal like renaming stuff. Solaris developers love extending stuff and not renaming it.

This kinda shows how it gets muddy. Sun developers helped porting Linux to sparc so they did take a few idea from Linux as well. With the Linux word renaming stuff so often its very easy to mix up who invented what.

Solaris/Sun has had some fine developers. Even that systemtap with dprobes and Dtrace started around the same time Sun got Dtrace production ready first.

Reply Score: 2

RE[4]: Comment by Drumhellar
by Kebabbert on Mon 31st Oct 2016 16:18 UTC in reply to "RE[3]: Comment by Drumhellar"
Kebabbert Member since:
2007-07-27

Both ZFS and Btrfs come out of the ideas of WALF
https://www.google.com/patents/US5819292
Yes NetApp patented the base idea that both ZFS and BTRFS use. ZFS does slab and BTRFS does b-tree implementation of the idea and I cannot remember what odd beast WALF does.

NetApp was not known among ordinary users back then, NetApp was Enterprise and no ordinary user knew much about NetApp, i.e. Chris Mason never talked about NetApp which he knew nothing about, he only talked about ZFS.

BTRFS was created after ZFS gained traction. Chris Mason talked alot about ZFS back then. "ZFS does this and that etc". He looked a lot how ZFS was doing things and tried to copy lot of features.

I read an interview where Mason said he added checksums to BTRFS after the ZFS team explained why checksums is important against data corruption. He had no clue why checksum were important earlier. Solaris team had for decades stored large data on large servers, and knew that data corruption was common in the large server halls, so they built ZFS around checksums. Mason never has set his foot into a server hall and added checksums as an afterthought.

Today, people say BTRFS is broken:
https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-RAID-56-Is...
"Software that is designed/ intended to be reliable should not go through large periods of instability only to be written off as "prepubescence". BTRFS been in development for almost a decade and it STILL isn't ready. Sorry, we're not talking about a mission to the moon (which btw was done in less time)."

Kent Overstreet (author of BcacheFS, a promising next gen filesystem for Linux) explains the rationale why he creates BcacheFS (no filesystem for Linux is good):
https://www.patreon.com/bcachefs
"btrfs, which was supposed to be Linux's next generation COW filesystem - Linux's answer to zfs. Unfortunately, too much code was written too quickly without focusing on getting the core design correct first, and now it has too many design mistakes baked into the on disk format and an enormous, messy codebase - bigger that xfs. It's taken far too long to stabilize as well - poisoning the well for future filesystems because too many people were burned on btrfs, repeatedly (e.g. Fedora's tried to switch to btrfs multiple times and had to switch at the last minute, and server vendors who years ago hoped to one day roll out btrfs are now quietly migrating to xfs instead)."


.


Dtrace and Systemtap is funny. Dtrace is started jan 2005 first prototypes of Systemtap starts as dprobes in jun 2004. So these two are parallel invention it possible Dtrace is copied the other way.

Systemtap team looked alot on DTrace. After a while, Systemtap team deleted all talk about DTrace in their logs. DTrace creator writes:
https://blogs.oracle.com/ahl/entry/dtrace_knockoffs
"...Amusingly, in an apparent attempt to salvage their self-respect, the SystemTap team later renounced their inspiration. Despite frequent mentions of DTrace in their early meetings and email, it turns out, DTrace didn't actually inspire them much at all:

CVSROOT: /cvs/systemtap
Module name: src
Changes by: kenistoj@sourceware.org 2006-11-02 23:03:09

Log message: Removed refs to dtrace, to which we were giving undue credit in terms of "inspiration..."


.

vprobes name is based of dprobes so that by vmware is not copy of Dtrace either.

Wrong again. VMware developer explains that VProbes is a copy of DTrace:
http://x86vmm.blogspot.se/2008/07/what-is-vprobes-relationship-to-d...
"...Before breaking ground on what would eventually become VProbes, I investigated a straight port of DTrace. I went so far as meeting with VMware legal to figure out whether the CDDL said what it appeared to say... VProbes has a similar set of design goals to DTrace: to be safe-yet-programmable, and be of global scope. DTrace is permissively licensed open source, that pre-dates the existence of VProbes, so why in heaven's name don't they share code?"

.

Linux world does have habit of rename something and start over so the old interface can be deprecated.

Yes, Linux copies something and the copy is bad. BTRFS is bad, ZFS is better. systemtap is bad, SMF is better. Systemtap is bad, DTrace is better, etc etc etc. Linux devs are quite bad actually, and their code is of low quality. Even Linus Torvalds say this:
https://en.wikipedia.org/wiki/Criticism_of_Linux#Kernel_performance

.

What is be-compared as the Linux replacement to dtrace is http://www.brendangregg.com/blog/2016-03-05/linux-bpf-superpowers.h...
bpf trace tools. This in it self is interesting.

This blogg by Brendan Gregg is a long time Solaris kernel developer. Then he switched to Linux recently. He has written books on Solaris DTrace. It is thanks to him and other guys, Linux has the foundation to finally port DTrace now.

.

This kinda shows how it gets muddy. Sun developers helped porting Linux to sparc so they did take a few idea from Linux as well. With the Linux word renaming stuff so often its very easy to mix up who invented what.

I know that Linux has copied lot of Solaris stuff, but I have never seen Solaris copied Linux tech? What ideas of Linux have Solaris taken?

Reply Score: 2

RE[5]: Comment by Drumhellar
by acobar on Mon 31st Oct 2016 22:01 UTC in reply to "RE[4]: Comment by Drumhellar"
acobar Member since:
2005-11-15

First of all, I too would like if the Linux folks would be less "happy" to start to code, specially on important aspects of the OS. When things are a bit more thought-out there is a bigger chance that most of the post implementation will end up on smaller corrections and incremental improvements instead of rewrites. As I said on an earlier post, there is a culture in Linux folks circle to be trigger-happy. As many already noted, Linux is created with engineering compromises in mind, i.e., to make things work, preferentially as soon as possible, instead of the more traditional computer science approach, i.e., get your "model" perfected first.

Anyway, all this effervescence around Linux created a phenomenal, fast pacing, OS development push around it that made it possible to support a lot of hardware much faster than any *BSD or Solaris on x86 hardware. It also pushed a lot of folks to implement the missing parts of the OS, like graphical interfaces, file management and lots of tools. Many of them end up finding their way to *BSD, Solaris and other Unix systems. It also brought the need to make things more manageable and so DEB and RPM were created and this is very relevant because package management sucked on Unix (understandable, as the focus of them were way stricter and the pressure to have a good system package management was less pressuring) and suddenly we had a "better" system on x86 hardware that were easier to manage and put to work on a large range of scenarios.

Chris Mason never talked about NetApp which he knew nothing about, he only talked about ZFS.


Don't want to be pick but it seems to me that the argument of the original post is that the fundamental idea did not originate on Solaris. If that is true, it does not matter if the BTRFS guy knew it or not.

Also, because of the environment around Linux, many things got implemented in it, if not first, usually before others besides the OS where it started, the biggest known exceptions being Dtrace and ZFS.

See "https://en.wikipedia.org/wiki/Comparison_of_operating_system_kernels... or Google to the difference between Linux, *BSD and Solaris kernels. You will be surprised to find out the Linux may very well be the most complete/flexible OS of all.

Reply Score: 2

RE[5]: Comment by Drumhellar
by oiaohm on Mon 31st Oct 2016 23:09 UTC in reply to "RE[4]: Comment by Drumhellar"
oiaohm Member since:
2009-05-30

Ohad Rodeh of IBM is the original Author of Btrfs and wrote the feature list Btrfs. Chris Mason currently has still not added a single feature to the feature list for Btrfs. Chris Mason is a person paid to implement Btrfs not its master designer. Ohad Rodeh knew of WALF and the link I gave was to Ohad Rodeh first presentation. Chris Mason take over development of Btrfs and due to Btrfs and ZFS being descendent of the WALF of course Chris Mason is comparing to it. Chris Mason is incorrectly credited with Btrfs invention because it was Chris Mason who got Btrfs good enough to merge.

So both Ohad Rodeh and Chris Mason play very important parts in Btrfs history.
https://www.diva-portal.org/smash/get/diva2:822493/FULLTEXT01.pdf
Also there is performance differences.

Systemtap can perform traces that Dtrace cannot by design.

BPF tracing do watch the video here
http://www.brendangregg.com/blog/2016-03-05/linux-bpf-superpowers.h...
He demos a few traces that neither Systemtap or Dtrace can perform.

In fact VProbes mentions the same problem performance. Dtrace design is not that high in the performance department. The reason why systemtap was removing references to Dtrace is Dtrace has a problem.


Brendan Gregg is not really making Dtrace part of Linux. Instead taking inspiration from Dtrace and making something superior to it.

The issue with Brendan Gregg heavy Dtrace background is most people are making the incorrect mistake he is porting Dtrace. Instead he is taking systemtap attempt to make something that performs better and can perform traces Dtrace cannot in fact work very well.

Calling stuff knockoffs as Dtrace developers have means they have not looked at what problems those making so call knockoffs were attempting to fix. The license issue was only 1 of many issues why you don't want dtrace as is. The big one is performance.

ZFS design also appears to have a problem from performance side.

Please note ZFS is slab, Btrfs is b-tree and bcachefs is b+tree. So even that all 3 file systems are attempting to-do the same thing at the core all 3 are major different.

Linus might complain about Linux performance but that is mistake when you get solaris vs linux on a lot of things.

Reply Score: 2

RE[6]: Comment by Drumhellar
by Kebabbert on Tue 1st Nov 2016 14:42 UTC in reply to "RE[5]: Comment by Drumhellar"
Kebabbert Member since:
2007-07-27

Ohad Rodeh of IBM is the original Author of Btrfs and wrote the feature list Btrfs. Chris Mason currently has still not added a single feature to the feature list for Btrfs....Chris Mason is incorrectly credited with Btrfs invention because it was Chris Mason who got Btrfs good enough to merge.

The Linux developers really dont understand why ZFS is designed as it is. They just copy ZFS without understanding the design decisions. This is because back then large Linux servers did not exist. The largest Linux servers back then were 2-4 sockets, maximum 8-sockets. And Linux scaled bad on anything above 1 socket. The large Enterprise server halls had large Unix servers and Mainframes, with up to 64-sockets. There were no such large Linux servers then, no one sold them, you could not buy such large Linux servers, they did not exist.

So Linux developers did not have experience of large server workloads, storage of large data, etc. They did not understand that data corruption is common among large servers storing large data. The Linux kernel developers only had experience of desktop PCs and small 1-2 socket servers.

That is the reason BTRFS checksums where only added after Mason read about the ZFS team talking about data corruption. If the ZFS team had not talked about checksums, Mason would not have added it. It is like Apple, their new APFS filesystem has no checksums even though data corruption is more common than people think. Mason did not understand the point of checksums first, because he had no first hand experience of data corruption in large servers.

Another proof that Linux developers dont understand ZFS is that they called ZFS a "rampant layering violation":
http://arstechnica.com/staff/2007/05/rampant-layering-syndrome/
ZFS is monolithic, one large piece of code. It has integrated raid controller, filesystem, volume manager, etc etc. Linux had separated layers of software (separate LVM manager, separate raid software, separate filesystem, etc), and when the Linux developers saw that ZFS was one piece of code they mocked ZFS because ZFS had bad design. Well, they did not understand why ZFS is monolithic. Let me explain why.

ZFS has checksums to protect against data corruption. The problem is, it is very difficult to get data integrity, just adding checksums is not enough. Hard disks have lot of checksums (CRC codes) to detect and correct data corruption and still hard disks get errors (check spec sheet of the latest enterprise Fibre channel disks). So no, you can not just add checksums and be safe. No, you need special kind of checksums, you need end-to-end checksums to be safe. Data within a domain is checksummed and safe, but when data is handed over to another domain, say from raid controller to filesystem, the data might get corrupt, because checksum is not handed over. Only the data is passed to another domain, not the checksum. So, you need to pass data and checksum to another layer for data to be safe. But Linux does not do that, because there are separate layers, raid level to filesystem layer does not pass over checksum, so data might corrupt. However, ZFS is one single layer, so ZFS knows the checksum all the time, there are no layers in ZFS. So ZFS data is protected all the time. That is why ZFS is single layer. Linux developers did not understand that and said that ZFS sucked: it used to much RAM, it was badly designed, it was slow, etc.

At the same time, BTRFS was also monolithic, but the BTRFS developers did not understand why they created it monolithic. They just copied ZFS.

.


Jesus. That was one bad student thesis. He concludes that BTRFS is faster than ZFS. But he writes that Hegel saw the opposite, and wonders why (p32):
"The results from the experiment with the single disk setup conducted by Heger (2009) differs drastically from the results found in this thesis. In Heger's experiment, ZFS outperformed Btrfs in all scenarios except for the sequential write operation where Btrfs had slightly higher throughput."

Why is that Heger saw ZFS being faster than BTRFS? He did not know. Well, let me tell you why. The reason ZFS was faster is because Heger tested ZFS on Solaris vs BTRFS on Linux. This thesis tests ZFS on Linux vs BTRFS on Linux. And ZFS on Linux is not mature.

Also, thesis tests one disk, and four disks. Well, ZFS is built for scale, large servers. It scales to hundreds of disks. On large Petabyte ZFS installations, ZFS gets very high throughput. BTRFS is not built for such large servers, BTRFS is a desktop filesystem. BTRFS might be faster on a single disk, but when you scale up, ZFS sky rockets. For instance IBM Sequioa supercomputer has a large 55 Petabyte ZFS installation with 1 TB/sec read write throughput. And because BTRFS is broken, the largest BTRFS installation is probably one single disk or two disks. I dont think there are any petabyte BTRFS raids out there. BTRFS might be faster on a single disk, but ZFS is faster on many disks. ZFS scales. BTRFS does not. And BTRFS corrupts your data, ZFS protects your data.

Reply Score: 2

RE[7]: Comment by Drumhellar
by phoenix on Tue 1st Nov 2016 16:45 UTC in reply to "RE[6]: Comment by Drumhellar"
phoenix Member since:
2005-07-11

ZFS has multiple layers internally, they are just ordered differently to "the common layering methodology" used in Linux. There's two major layers (DMU, and DSL). Inside of each are also multiple layers, although they are treated the same from the outside.

The big difference is that Linux has a bazillion layers (physical, md, crypt, lvm, filesystem, to name a few) and umpteen different ways to layer them. ZFS hides all that internally, and just shows you a data management layer (how you arrange the physical disks into vdevs in a pool) and a filesystem layer. Everything else is handled internally.

Yes, it's a single monolithic codebase; but it is clearly split into layers internally and data is passed between the layers using strictly defined interfaces. Same as the Linux storage stack ... just ordered in a very different, and static, way.

Reply Score: 2

RE[6]: Comment by Drumhellar
by Kebabbert on Tue 1st Nov 2016 14:44 UTC in reply to "RE[5]: Comment by Drumhellar"
Kebabbert Member since:
2007-07-27

Systemtap can perform traces that Dtrace cannot by design.
BPF tracing do watch the video here
http://www.brendangregg.com/blog/2016-03-05/linux-bpf-superpowers.h...
He demos a few traces that neither Systemtap or Dtrace can perform. In fact VProbes mentions the same problem performance. Dtrace design is not that high in the performance department.

If anything important needs to be traced, you can do it with DTrace. Maybe you can trace some not important stuff with Systemtap that no one cares of.

Performance is not important when tracing production servers. It does not matter at all. What is important, is that servers do not crash. DTrace is safe, so you can trace business servers without problems. Linux has no tracers that can do that, systemtap crashes servers. So you can not use systemtap in practice.

Normally, you need to build up a copy of your problematic Linux server and examine the copy. It can take weeks. With DTrace, you just examine the server right on and find the problem immediately. Systemtap might be faster - but it crashes servers. DTrace is safe, it protects your server.

.

The reason why systemtap was removing references to Dtrace is Dtrace has a problem.

No, wrong again. It says that they removed references because they did not want to give credit to DTrace. It says so. Read it again.

.

Brendan Gregg is not really making Dtrace part of Linux. Instead taking inspiration from Dtrace and making something superior to it.

So, Gregg is making something better than DTrace? Just like BTRFS is better than ZFS? Or systemd is better than Solaris SMF? Are you serious? When has Linux developers created some tech better than Solaris tech? Never.

.

The issue with Brendan Gregg heavy Dtrace background is most people are making the incorrect mistake he is porting Dtrace. Instead he is taking systemtap attempt to make something that performs better and can perform traces Dtrace cannot in fact work very well.

Calling stuff knockoffs as Dtrace developers have means they have not looked at what problems those making so call knockoffs were attempting to fix. The license issue was only 1 of many issues why you don't want dtrace as is. The big one is performance....ZFS design also appears to have a problem from performance side.

Wrong again. Look, these large business servers handles millions of dollars every second. You must not let the server crash, because there are so much money involved in these large servers. If the server is down 1 hour, the company looses millions. That is why they never touch these large business servers, and they never upgrade them. They only want LTS with long support cycles. Some servers run for decades. The only important thing about these Enterprise servers, is stability. DTrace is safe to run on them. Systemtap is not. It does not matter if Systemtap is 10% faster than DTrace because systemtap can crash the server.

You dont have much experience of large servers handling millions of dollars every second. But I have. I know what is important. Linux people don't. They think performance is the only important thing, but no, stability is the most important. Performance is desktop users focus. Stability is Enterprise server focus.

.

Linus might complain about Linux performance but that is mistake when you get solaris vs linux on a lot of things.

Have you read the rest? The Linux developers say the code quality is bad. One of them even compares Solaris kernel code to Linux kernel code, and says that Solaris is very stable and superior. Linux is fast, but might crash any second. He says he is ashamed of the Linux code.

Reply Score: 2

RE[7]: Comment by Drumhellar
by oiaohm on Wed 2nd Nov 2016 01:08 UTC in reply to "RE[6]: Comment by Drumhellar"
oiaohm Member since:
2009-05-30

Performance is not important when tracing production servers. It does not matter at all. What is important, is that servers do not crash.


This is fatally wrong. It is correct that tracing causing crashes is not on with production servers. Losing large percentage of performance while performing trace is not all either. In fact trace engine costing too much resources can make the trace results worthless. Why the application that you tracing may no longer be loaded enough to trip over itself and malfunction after you enabled tracing.

[p]With DTrace, you just examine the server right on and find the problem immediately. Systemtap might be faster - but it crashes servers. DTrace is safe, it protects your server. [/p]

People saying this have not worked with Dtrace and Systemtap enough. The reality is at times Dtrace will be unable to find the problem. This was kinda leaving you between rock and hardplace. Please do remember under Linux you do have Dtrace as a loadable module its not like Linux does not have Dtrace. So you have been seeing workflows of attempt Dtrace when that fails drop to systemtap while praying it did not break the system to get the information to solve problem. Now with BPF trace you have the stability of Dtrace and almost the same level performance as doing a systemtap and has most of the extra features switching to systemtap use to give you without taking the risks.

In from my point of view BPF trace is the first time trace system has been designed correctly where a person has not put performance/features ahead of stability as Systemtap did or stability ahead of performance/features like Dtrace did. BPF trace put features, performance and stability on importance that none can be lacking.

It says that they removed references because they did not want to give credit to DTrace. It says so.


If you read the complete mailing list by giving credit to Dtrace different people were suggesting dropping different features from systemtap because Dtrace did not have them so Systemtap did not need them. Those features were the very reason Systemtap was being used. So people thinking of Systemtap as a copy of Dtrace was causing problems.

So, Gregg is making something better than DTrace? Just like BTRFS is better than ZFS? Or systemd is better than Solaris SMF? Are you serious? When has Linux developers created some tech better than Solaris tech? Never.


You don't find better tech by just coping. BTRFS is testing out if b-tree will work out better and bcachefs if b+tree will work out better. The reality is until the filesystem are truly complete we won't know. ZFS might turn out to be way worse than either bcachefs or btrfs.

This is the problem solaris people have you should copy solaris as systemtap to BPF trace shows that following Solaris path is not always the right one. BPF gets rid of using systemtap only language like Dtrace D language and replaces with the generic language of BPF that is secure and is maintained optimised to be used in networking.

So systemtap coping the Dtrace idea of a language just for tracing was in fact complete wrong. Now this also says everyone who has just copied Dtrace into their system has also screwed up. What is the problem if a language is only used for tracing people don't put the required hours in to optimise the language so it performs well. Systemtap allowing you to drop to C an unsafe language was a megahack around this problem.


The Linux developers say the code quality is bad. One of them even compares Solaris kernel code to Linux kernel code, and says that Solaris is very stable and superior. Linux is fast, but might crash any second. He says he is ashamed of the Linux code.


Its not that simple. That is Con Kolivas who said scheduler should not support cgroups that is solaris equal to zones. The range of hardware Linux supports compared to Solaris does bring in more different companies/developers with different coding styles into the source tree.

There is a price for everything.
https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project
The reality is the Linux world is more open to change that does cause problems but it also gives advantages. The Linux Kernel Self Protection Project is implementing feature that Solaris does not have to limit the damage drivers and kernel parts can do. So yes Linux source code might be worse to read but runtime protections are already ahead of Solaris.

Edited 2016-11-02 01:12 UTC

Reply Score: 2

RE[6]: Comment by Drumhellar
by Kebabbert on Tue 1st Nov 2016 14:48 UTC in reply to "RE[5]: Comment by Drumhellar"
Kebabbert Member since:
2007-07-27

And dont get me started on systemd. The Linux devs have not understood anything about Solaris SMF. systemd is a piece of junk, it has it's own terminal and other stuff too.

Reply Score: 2

RE: Comment by Drumhellar
by zdzichu on Fri 28th Oct 2016 12:35 UTC in reply to "Comment by Drumhellar"
zdzichu Member since:
2006-11-07

If you had read the article, you would find info about SystemTap gaining eBPF backend - https://lkml.org/lkml/2016/6/14/749

Reply Score: 2

Solaris Legacy
by ventejuy on Fri 28th Oct 2016 12:14 UTC
ventejuy
Member since:
2009-12-29

Looking back, Solaris 10 was one of that OSs that set a technical reference.

Reply Score: 2