Adam Leventhal (one of the DTrace developers) was in volved in making DTrace work for BrandZ. He’s written a blog entry on using this combination to have a look at the “top issue” (why is it that whenever you run top, the top process is top?). The blog entry shows how using DTrace and OpenSolaris (the only way that you can currently d BrandZ), can be of benefit to Linux.
DTracing Linux Userspace Using BrandZ
About The Author
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
2005-12-19 5:25 ambcantrill
I find it surprising that, since Adam’s blog entry, some have taken exception to my description of the so-called “top problem” — that when running top, the top process is top itself. (You aren’t the first to have disagreed with this, nor the first to attribute it to Solaris.) I find it surprising because I have talked about the “top problem” to hundreds of audiences — many of them with a strong Linux bias — and when I ask “when you run top, what’s the top process?”, audiences often practically shout in unison: “top!” Or, as one admin told me (and as I have been known to recount), when they run top they see that the top seven processes are seven different top processes from seven different admins logged into the box trying to figure out why the server is so slow. (!)
So I suspect that those for whom the top process isn’t top are simply not looking at a moderately loaded system. Once you get hundreds of users and thousands of processes, top consumes significant resources — be it on Solaris, Ubuntu or otherwise. Indeed, it was because of the poor load scalability of top that we (or rather, Andrei Dorofeev) wrote and shipped prstat(1) back in Solaris 8. And fortunately, we haven’t yet heard of anyone describing the “prstat problem”…
The point being made is that it is now possible to look at Linux user space code using DTrace using this methodology.
Sure, it’s quite possible that Solaris’ munmap takes longer.
The thing is though, you can now look for performance bottlenecks in application code that only runs in Linux operating environments with a tool that would be otherwise unavailable.
Having been physically in the same room as Bryan when he has asked this, I can definitely vouch for the response.
Does this mean I can run Red Hat binaries on Solaris now? Do I have to pay for the Solaris license?
2005-12-19 7:51 amaleventhal
Yes, you can run Red Hat binaries. No, you don’t have to pay for a license. <a href=”http://www.opensolaris.org/os/community/brandz/“>BrandZ</a>… lets you run Red Hat binaries on OpenSolaris, and Solaris itself has been free for about a year now and open source since June — no need to pay for a license
2005-12-19 12:40 pmmanmist
“Does this mean I can run Red Hat binaries on Solaris now?”
If you mean RHEL binaries, then yes you have to pay since it only comes with a subscription. If you mean binaries that run on RHEL then maybe not
The observer changes the observered.
In this is case, the observer is dtrace hooked into the the solaris kernel and an emulated linux environment.
How can Dtrace be used to measure linux system call times when the system calls aren’t executed in an actual linux environment?
The profiler imparts some of it’s own effect on the system, so it’s wise not to change even more variables (ie, the entire kernel) when profiling.
2005-12-19 8:06 amaleventhal
It’s certainly possible that changing the system can skew the results, but that doesn’t necessarily invalidate the conclusions. Ideally, one would be able to perform the analysis on a native system, but without examples of how one would examine user-land code it’s not clear how useful that tool you referenced would be. Was any attempt made to drive from the specific results to a root cause or was the exercise just to see if you could reproduce my results?
I suppose I should have more carefully documented my environment (hardware details, software load and version, top version and invocation details) so that it would be easier to duplicate my results, but it seems a bit hasty to assert the invalidity of my results because of an inability to reproduce them in what may have been a completely different environment.
The point was, after all, that BrandZ makes possible a sort of analysis previously unavailable for Linux applications.
Top sometimes is on top in linux, however the real problem i have is on the mac… Freaks me out..
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE
13798 top 11.7% 0:04.40 1 21 22 480K 428K 936K 27.0M
2005-12-20 12:24 amchekr
This off an iMac G5 with 10.4.3
2587 top 21.4% 0:09.59 1 18 22 508K 312K 2.53M 27.0M
The 21.4% is CPU Util.
System is fairly idle too
poor old top…
This is actually still a win for the Open Source community. Even though a Linux binary emulated on Solaris will not run completely native, dtrace will still allow software engineers to find bottlenecks in code.
Until systemtap ( http://sourceware.org/systemtap/ ) is featurecomplete and integrated into the mainline kernel, something like this is very useful
2005-12-19 4:12 pmAnonymous
“Until systemtap ( http://sourceware.org/systemtap/ ) is featurecomplete and integrated into the mainline kernel, something like this is very useful”
System Tap is already part of RHEL 4 U2 release and is also in Fedora development tree and FC 5 test 1 release.
Adam will be presenting on this at SCALE 4x in Los Angeles on Feb 11-12, 2006. http://www.socallinuxexpo.org/speakers/speakers_leventhal.php
SystemTap is still under development and evolving—do not use it on production systems and expect that things will change. However, SystemTap in its current state can still be a useful tool for developers.
2005-12-20 9:19 ammanmist
“SystemTap is still under development and evolving—do not use it on production systems and expect that things will change. However, SystemTap in its current state can still be a useful tool for developers. ”
Article is old. Software evolves. Whats your point?
2005-12-20 10:33 amAnonymous
Article is old. Software evolves. Whats your point?
His point was that his choice of OS is better than yours. Apparently that quote was the best he could come up with in the way of supporting evidence.
2005-12-20 11:03 ammanmist
If a sun employee cannot use better than outdated quotes in an article to support his “choice” of an operating system…. *shrug*. code in Linux is not tied to whims and fantasies of any particular vendor and a result moves forward faster. Hate Debian go SUSE. Hate SUSE go Fedora. Hate sun and you are stuck with no support for open solaris. The GPL license enables a code commons which CDDL effectively reduces for its own loss by being more lax. While this was done for Sun to stick away its own encumbered code it couldnt open source it has to live with the effects until it realises the drastic effect of the underlying licensing framework which is unfortunate IMO.
Sun did a great job with providing accessibility support on GNOME by working as a contributor along with the community and try and provide support now for its ABI specifications. If it had choosen to work in the same manner with Linux or just try to enable sharing across Linux and within the “open” solaris community it has a fabalous chance of thriving well by immediately getting the entirely OSS community on board. The walls between these two will have to live if and until GPL 3 provides a compatible patent claus.
2005-12-20 4:53 pmAnonymous
You are quite naive about open source software in large scale enterprises.
Hate Red Hat? Feeling fleeced by their licensing and proprietary code?
OOOPS, can’t move, because my ISV’s only support Red Hat! You are stuck, independent of GPL licenses.
At least with Sun, I can go to any number of enterprise vendors to get support – HP, IBM, EDS or Sun.
2005-12-20 3:39 pmlibray
That article was very easy to find. Its the very first link under the “Documentation” heading from http://sourceware.org/systemtap. Have any evidence to refute the old article? Good. Tell it to the systemtap maintainers.
Edited 2005-12-20 15:40
“why is it that whenever you run top, the top process is top?” [sic!]
Maybe it is a Solaris thing, because this does not happen under Ubuntu.