“With the introduction of Solaris 10 and the dynamic tracing facility (DTRACE), the ability to dynamically instrument a system and solve complex problems is a D script (D is the language used by DTRACE) away. Several extremely useful D scripts have emerged,
so I thought I would share my ten favorite DTRACE scripts with the OSNews community.”
Each of these scripts solves a real problem in a sysadmin’s life, somewhere. I’ve certainly been frustrated by tracking down which processes are accessing certain files or generating certain network traffic. The traditional UNIX tools just fall short, and I’d be forced into manually searching /proc or truss output or worse for hints (talk about a time sink 🙁 ).
It really seems the sysadmin’s imagination is the limit with these scripts.
Rarely does a sys admin have to trace an binary to see how it behaves. This is the work of the developers that work for the vendor that provides the software.
A few of those scripts can be potentially useful. It interesting to note that most of those scripts are parsed through other programs such as perl. You would think that dtrace would be a little more user friendly to make it more useful.
Read the information page about DTrace — then try to justify how such a tool should be “user friendly”. If you’re a sys-admin/developer, it’s not beyond you to figure out how it works. If you’re not, WTF are you doing with DTrace anyway?
Rarely does a sys admin have to trace an binary to see how it behaves. This is the work of the developers that work for the vendor that provides the software.
Rare, true, though handy when you need it. For example: during an initial security audit or to increase system stability. Knowing what uses what gives me the chance to determine what can be disabled or removed without causing problems. Package managers under *nix do a good but not perfect job with this automatically, though MS Windows does not. Even though that is the case, I still do this under *nix.
Side note: For MS Windows services and apps alone, I do this to determine what is crashing the system; the apps under other operating systems tend to be much better behaved.
Nice Troll 😉
We are two months from deploying Solaris 10 in a production environment. I expect that in three months, due only to DTrace, our understanding of our hardware utilization an software performance will dramatically increase. We will save, through time savings and consultant costs, thousands of dollars a year by quickly and accurately identifying problems, bottlenecks, resource limitations, etc… Additionally, our hardware expenditures will be far more focused due to having a better understand of exactly what resources our servers consume.
For people who use resource-hungry software, DTrace is the most important feature to arrive in the UNIX world in years. If you don’t already understand its importance, then you probably don’t need to (yet).
We are two months from deploying Solaris 10 in a production environment.
You’re in trouble then.
I expect that in three months, due only to DTrace, our understanding of our hardware utilization an software performance will dramatically increase.
It isn’t up to you to find out how your hardware is being utilised, how your software performs or to debug software, and if you’re doing that two months from a go-live you need a cat scan – fast.
Software that you buy off the shelf or as part of an operating system should never, ever need to be touched by a system administrator in that way or a company at any stage. You know all that money you pay to Sun? That’s what it’s for ;-).
DTrace is a software developer’s tool, and should prove to be great for open source developers in particular, but if you’re already deploying you’re past the point of no return. If I do need DTrace, that’s a problem for Sun’s engineers and developers to work out, not me or my organisation.
Many are called but few are chosen, too bad you’ve never been one because it’s the toughest job you’ll ever love. Sysadmin *is* a programming position, if you don’t know how and rely on the vendors then you’re just another operator posing as one.
PERL is user friendly and dtrace incredibly useful but only to a real Sysadmin.
Many are called but few are chosen, too bad you’ve never been one because it’s the toughest job you’ll ever love. Sysadmin *is* a programming position, if you don’t know how and rely on the vendors then you’re just another operator posing as one.
Whatever, true Sysadmins are neither DBA’s nor are they programmers. If you placed yourself in that bastardized role that’s your own fault. I don’t have time to debug some vendors crappy code. Its simple really if our company payed a vendor a couple million for the software it better well work and when it doesn’t the people responsible(Vendor developers) for the crappy code need to fix it. Perhaps you only have to worry about a single application in your environment which in that case your statement would be somewhat accurate. In contrast the environments that i worked involved many applications with 1000’s of users. Handling user application needs, server build outs and general Sys admin tasks doesn’t leave much time to fiddle around writing C.
PERL is user friendly and dtrace incredibly useful but only to a real Sysadmin.
A real sys admin does everything he/she needs using shell programming. Dtrace may be great for when you are working on a Solaris 10 machine but what do you do when you have the same issues on an HP-UX machine? AIX? Windows? Free advice being platform limited is not very marketable in todays job market.
Go back to your helpdesk seat and let the grown ups talk
Your definition of being a Sysadmin sounds incredibly limited and boring.
Your definition of being a Sysadmin sounds incredibly limited and boring.
Providing a organized and sucessful computing environment for a companies users is a fufilling job for me. I would rather havea root canal then to sit and write code day in and day out(i tried it not my thing).
A true sysadmin embraces and extends the tools available, since structure and automation is the key to managing complexity. IMHO shell scripting is not the answer, since it is remarkably hard to maintain shell scripts. I had a choice whether to make massive changes to an ~8 year old, 2000+ line ksh script or rebuild it in perl, and I decided not to put my eventual successor through the same thing. Perl in particular gets a bad reputation for being a write-only language, but shell scripting is far worse in that regard. It is easy to write unintelligible perl, but it is almost as easy to write elegant and readable perl. All non-trivial shell scripting is practically unreadable, even if you yourself wrote it a a couple years ago.
If your conception of a sysadmin’s job is to dispatch the proper vendor to solve any problems that might arise, then you are hardly useful to your company. A sysadmin should have and use the skills to drill down into problems. Programmers don’t want to hear, “its not working right, please fix it.” It’s the sysadmin’s job to report issues in a way that points the developer to the root of the problem.
Periodically I attend customer presentations where IT people from companies come in and talk about how they use our systems and what they would like to see improved. Sometimes I am blown away by the demanding and technical nature of their issues, and how well they articulate what is going on and why. These guys may be sysadmins, but it is clear that they spend a great deal of time examining the kinds of things that DTrace seems to make easier. These guys will have jobs while your’s goes oversees.
A true sysadmin embraces and extends the tools available,
You sir, are a real Sysadmin and a credit to your noble profession.
You mean the helpdesk you go crying to every time your server so much as farts? Your job may have the sysadmin title but the job description is at the level of an operator or a technician at best. A true Sysadmin is a DBA and programmer both. From the sound of it if your SLA went from platinum down to silver or less you would just have to quit and go find another job where Sun can hold your hand. Real Sysadmins are capable of supporting themselves.
I disagree with your assessment. UNIX is built on building blocks, and using other, well known tools for parsing and such is the norm. I prefer having one new tool do 1 new job and not reinventing the wheel.
Also, when you support a development shop and have to run home-grown binaries, it is essential that you can do something like this as it may not always be possible to work with the DEV’s, especailly when they tell you your box is mis-configured.
YMMV.
I liked the article, but it kept cracking me up how he ended every program:
Golden!
Tight!
Shibby!
Niiiiiiiiiiice!
Jeah!
Smooooooooooth!
If only Dorothy could see this!
Now this is some serious bling!
Shizam!
DTrace seems like it delivers good functionality. However, is there a GUI to help make sense of the results? Some sort of management tool to build some graphs?
DTrace seems like it delivers good functionality. However, is there a GUI to help make sense of the results? Some sort of management tool to build some graphs?
Write one!!
throw it to XML, ahem, then use XSLT to chew it to SVG or Flash maybe XHTML with AJAX that communicate with DTrace daemon webservice over SOAP .. err.. may be too overkilled
If I wanted to write a GUI for all the UNIX tools there are… well, that would be working at Apple on OS X, wouldn’t it? 😉
I don’t remember which vendor made the buzzword “autonomic computing”, but I would think automatic graphics would have to be a feature of that. Sorting through a lot of text is error prone and time consuming.
One of the things that made the original VTune fun was the graphics. I hope DTrace gets graphics in an upcoming revision.
Heh…that was a good response. Funny
I don’t see why you have such a negative reaction to DTrace? It might be true that a level 1,2, or 3 system administrator probably will not need it, but going up the escalation ladder, someone will have to eventually figure out what’s wrong. I think it’s a very necessary tool to have, especially as you go higher and higher into enterprise class system. Why is your Java program slowing down? Why is Oracle chocking? Why are things not happening like they should bE? These are all very valid concerns that are extremely difficult to answer using traditional tools.
Using the tools Microsoft provides, we are never able to figure out why our servers sometimes slows down. We get hit by our manager whenever the system slows down. In most likely hood, it is most probably some poorly written Stored Procedure, Trigger, WebService, infinite loop, but we just can’t find out. We would tell Microsoft to fix it but it really isn’t their fault either. It is most likely the internal application developer’s fault, but it always falls on us because we can’t prove it. It would be nice if we were able to contact some kind of high paid consultant, get a few DTrace type scripts, run them on our server when it has problems, and have him analyze the problem and tell us what the computer is doing at that time. I’m not saying DTrace won’t be for everyone, but if someone specializes in it, he will be able to give you a world of knowledge.
I think DTrace is really a good revolutionary idea. It sits somewhere between kernel dumps/ debugging, which are relatively useless on a live system unless it completely craps out and between a system can’t tell you anything about what’s going on. DTrace will be extremely usefull on production systems. Hopefully, Linux and Windows will implement similar capabilities too.
you’re kidding but in my former place, it was what they did to justify their overpriced bill to their client….
some guys want some stuff and the only solution was :
let’s build a webapps running j2ee with some servlets and a bunch of dedicated developer : it will be done in 3 months and we can garantee after 6 months…
seriously, I couldn’t imagine it worked that way in the ITT industry….
j2ee, jsp, tomcat and 36 man/months was the answer to every solution……..
I hope they’ll burn in hell
D is the language used by DTRACE
It is? Doesn’t look like D to me.
http://www.digitalmars.com/d/
“DTrace is a software developer’s tool,”
No, it’s both a floor wax _and_ a dessert topping!
Seriously, in any _real_ production environment, unexpected things will happen, even things beyond the control of Sun’s engineers. I’ve also seen situations where the ‘software developers’ couldn’t find their own ass in the dark with respect to operating system-level details. Somehow, anything below a high-level API is like an abyss of the unknowable to them.
Operating systems are very complex, and asking any software developer to both master something like J2EE on top of knowing all the internal details of a UNIX kernel is pushing the limits of any sane human.
Seriously, in any _real_ production environment, unexpected things will happen, even things beyond the control of Sun’s engineers. I’ve also seen situations where the ‘software developers’ couldn’t find their own ass in the dark with respect to operating system-level details. Somehow, anything below a high-level API is like an abyss of the unknowable to them.
No doubt. I’ve even had to train a developer to use a diff tool. How did this person graduate? How did they get a job????
If you (segedunum) have never had to do any deep troubleshooting, that’s great. However I have and the better the tools that I can throw at a problem, the faster I get the problem solved. Believe me I wish DTrace ran on Solaris 8, I could have used it a few times. Also Sun is not going to immediately escalate a case to a Tier 3 engineer without some serious background material so if nothing else DTrace allows an administrator to gather detailed information on the current state of the system.
And why does DTrace have to be graphical (pravda)? Work in an environment where X is not allowed or over a serial connection, I do so I need tools that don’t depend on graphics (most Sun admins would agree with me on this). Graphical tools are fine if that’s what you need, I just don’t want the CLI tools to go away in order to put more developers on graphical programs.
Pictures work so much better than text. The data itself can stay text/xml, but there should be a graphical client that can suck in the data and then make some sense of it.
A nice GUI client also would allow you to save a picture of config#1, get some more data for another config, and then you could easily see a visual for config#1 vs. config#2.
It’s that old adage of a picture is worth a thousand words.
The CLI should stay around. Every system should have CLI *and* GUI, not one or the other. Haven’t we learned this… yet???
The best administrative tool I ever used was smitty, the CLI version of IBM AIX’ smit. Very fast, no graphical overhead, and was able to do the same things as the graphical version. Unfortuantely what I am seeing is way too much emphasis on graphical tools at the expense of the CLI equivalent (or not developing a CLI tool at all).
The only time I use a GUI (other than Windows) is to administer the HIGH side machines using a Sun Ray, and even then I use multiple dtterm windows.
To each their own.
I believe that tools should always be available in three versions — pure text, web/html, and fancy graphics.
There are strong value propositions for all three.
Yet I am unaware of any OS that delivers all three.
AIX (WebSm, smit, smitty) and HP-UX (SAM CLI, GUI and web based).
fully agree ! like for example the kernel configuration tool :
make config: plain text
make menuconfig: curses
make xconfig: Qt
I developped a distributed tool (http://yadp.sourceforge.net) and i tried to fill these basics expectations.
The best administrative tool I ever used was smitty, the CLI version of IBM AIX’ smit.
I agree smitty is awesome if i remember right it also lists the command that is executed at the CLI so once you find out what you need to do via the gui you can skip it next time. SAM in HP-UX is also very good i use it mainly to tweak the kernel makes life so much easier. I think the balance is somewhere in between why do a 10 minute process using the CLI when it take 2 minutes using a GUI? If dtrace was similiar in functionality and presentation as say glance plus for HP-UX. I feel it would be much more useful.
You’re talking about the smit.log, that was good. Especially if you ever tried to use the installp command (and the options from Hell).
Never had the opportunity to use SAM, it is has been a few years since I administered a HP-UX box and until you mentioned it I have never heard of Glance Plus.
Whatever Sun does I hope they have equal efforts in GUI and CLI environments.
I don’t see why you have such a negative reaction to DTrace? It might be true that a level 1,2, or 3 system administrator probably will not need it, but going up the escalation ladder, someone will have to eventually figure out what’s wrong.
Perhaps it was the example given by SUN how they couln’t figure out network problem they were having with a server. They used dtrace to try and undertand what was going on turns out they were missing /etc/defaultrouter! So the were bit by sloppy SA work. Developers thinking they are Sysadmins <rolls eyes>. I will echo what others have said that dtrace is a great tool for developers not that useful for SA’s
Operating systems are very complex, and asking any software developer to both master something like J2EE on top of knowing all the internal details of a UNIX kernel is pushing the limits of any sane human.
Then those developers should not be developing for any given platform. Who throws code together without a unique understanding of the underlying Operating System.
If your conception of a sysadmin’s job is to dispatch the proper vendor to solve any problems that might arise, then you are hardly useful to your company. A sysadmin should have and use the skills to drill down into problems.
Did it twice last week and once this week already. Like i said i do not need dtrace to tell me that there is something borked with a very small relativly low usage database on a 6×16 server where the CPU utilization is averaging 78%. sar coupled with mrtg identified the problem getting a DBA to configure the database properly solved the problem.
My argument is i should not have to do the DBA’s job of keeping the database functioning properly nor should i do the job of the developers by debugging their code. I make a clear seperation between the entirely different careers. Assuming everyone does their job properly it all runs like a well oiled engine.