Why Linux people lust after DTrace

By David Stewart (Intel) (71 posts) on May 15, 2007 at 12:25 pm

Back in 2002, I was managing the team that worked with Oracle on the engineering side.  This was when they were transitioning from developing the database on Solaris to Linux, and we were helping them with this transition.  Of the various requests their devs made, one of the first was "when can we get DTrace on Linux?" 

I was a bit confused, since I had never heard of DTrace! 

Fast forward to 2007, and I sat listening to Bryan Cantrill, the inventor of DTrace show some examples of the power of this little tool which comes with Solaris. 

> The kernel has been instrumented with probes which cause no impact if they are not activated, but which allow the DTrace command to program them to fire.

> The command can be used to count probe firings as events and display these counts as histograms for example.

> Besides the kernel, one of the speakers I heard had instrumented the javascript runtime engine to enable DTrace usage, so you can identify where your time is going in a web site.

> It has absolutely no GUI.  Someone experienced in the art can simply write DTrace scripts on the fly and get amazing results. 

The recent example they used was Twitter.  Remember my recent comments about this new micro-blogging application?  Well it turns out that the gang at Twitter implemented their system using Ruby on Rails, the new and sexy rapid website programming system.  After the article in the New York Times, Twitter has grown in popularity to be the biggest application of Ruby on Rails.  And of course, they were having performance problems. 

Fortunately for Twitter, they implemented it on Solaris, and thus DTrace to the rescue.  Within a few hours, they had identified an issue with how often deep stack back traces were being taken, they get a fix from the Ruby guys and had a 30% performance boost.  Pretty snaz, yeah? 

No wonder I saw DTrace listed as one of the top 10 reasons Solaris is better than Linux.

Categories: Open Source

Comments (37) Comments RSS Feed

By Josh Bancroft’s TinyScreenfuls.com » How Twitter got a 30% performance boost from DTrace on May 15th, 2007 at 1:32 pm
[...] of the service know, that has resulted in some stability issues. Growing pains. I just came across this post by one of my cohorts, David Steward, over at the Intel Software Network blog on DTrace, a Solaris feature that Linux people drool over, and how the Twitter devs were able to [...]

By Josh Bancroft on May 15th, 2007 at 1:33 pm
Great story, David. I'm an avid Twitter user, and anything that helps them improve performance and uptime is great in my book. ;-)

By ISN Open Source Feed on May 15th, 2007 at 8:25 pm
links from Technoratian issue with how often deep stack back traces were being taken, they get a fix from the Ruby guys and had a 30% performance boost.  Pretty snaz, yeah?  No wonder I saw DTrace listed as one of the top 10 reasons Solaris is better than Linux. (viaISN - Posts in the Open Source Category)

By My soup on May 15th, 2007 at 9:30 pm
links from TechnoratiTwitter has experienced meteoric growth in the last few months, and as most users of the service know, that has resulted in some stability issues. Growing pains. I just came acrossthis post by one of my cohorts, David Steward, over at the Intel Software Network blogon DTrace, a Solaris feature that Linux people drool over, and how the Twitter devs were able to squeeze a 30% performance boost out of Ruby on Rails using it: The recent example they used was Twitter. Remember my recent comments about this new

By blogstar on May 16th, 2007 at 6:18 am
Of course if you actually have your 5 nines, "improving uptime" hardly merits a mention.

At this point, I'm not sure if twitter has 2 nines, let alone more, and has been plagued for a fortnight with 503's, offline messages, and null pages.

By z on May 16th, 2007 at 6:18 am
Isn't there KTrace for Linux?

By Brian Moyles on May 16th, 2007 at 6:35 am
Linux has its own DTrace-alike coming along, and some would argue that it's better than DTrace in some ways.
http://sourceware.org/systemtap/

By S on May 16th, 2007 at 7:25 am
Solaris has had very slow startup times for a process. Historically thats been one reason to avoid it.

By dazza on May 16th, 2007 at 8:39 am
Efforts are being made to port DTrace to FreeBSD. I think it's scheduled for release with FreeBSD 7 http://people.freebsd.org/~jb/dtrace/

By Josh Bancroft on May 16th, 2007 at 9:47 am
Oh, cool! It looks like DTrace is coming to Mac OS X Leopard, with a cool GUI. It's called Xray, and it's part of the XCode 3 IDE that's coming when Leopard launches.

http://www.apple.com/macosx/leopard/xcode.html

Can't wait to try it out! :-)

By David Stewart on May 16th, 2007 at 9:53 am
Yes, well of course if you LIKE the GUI interface, it's good to have one. I think I heard about one that's Solaris-based as well.

Lack of a GUI doesn't phase most *nix developers. In fact they prefer the command line interface for development when they can.

I could write a bunch about the GUI vs non-GUI approach for development. A lot seems to be based on where you got your start as a developer.

By David on May 16th, 2007 at 10:21 am
Try SystemTap: http://sourceware.org/systemtap/

By reddiggulo.us - The Most Recent Popular Stories on May 16th, 2007 at 12:32 pm
links from TechnoratiWhy Linux people lust after DTrace

By BloggerFodder.net on May 16th, 2007 at 3:23 pm
links from TechnoratiRon Paul vs. Giuliani on the Root Causes of Terrorism - "It's American foreign policy that enrages Osama and al-Qaida, not American culture and society." * We must attack Iran before it gets the bomb *Why Linux people lust after DTrace

By Oliver Herold on May 17th, 2007 at 1:36 am
http://people.freebsd.org/~jb/dtrace/

MacOS, FreeBSD ... where is the problem with Linux? Some silly license-war?

By Tet on May 17th, 2007 at 4:47 am
SystemTap is the Linux equivalent of DTrace. True, it's not yet quite as mature as DTrace, but it gives you 90% of what most people want, and it's getting better all the time...

By teki blog» le/fel/megjegyezni ezt meg azt on May 17th, 2007 at 5:50 am
links from TechnoratiWhy Linux people lust after DTrace

By David Stewart on May 17th, 2007 at 7:41 am
Comment from Oliver Herold: "MacOS, FreeBSD ... where is the problem with Linux? Some silly license-war?"

*I* don't have any problem with Linux. I think Linux is great and my hat is off to Linus and the Community for pulling it off.

Sun did choose to license the Solaris kernel under CDDL rather than GPL.

In brief: they felt that GPL wouldn't allow them to protect the IP rights of some of the code suppliers to the kernel and BSD wasn't protective enough of the rights of the Community. So they compromised between the two.

So actually there are some advantages of Solaris over Linux. Dtrace is one of them.

By mangoo on May 17th, 2007 at 10:46 pm
Have you checked SystemTap on Linux? It's more or less an equivalent of DTrace (some will argue it's less, some - it's more).

SystemTap website: http://sourceware.org/systemtap/

Some usage examples:

http://sourceware.org/systemtap/wiki/WarStories
http://sourceware.org/systemtap/wiki/ScriptsTools

By Alok on May 18th, 2007 at 3:17 am
Just use /usr/bin/strace and grok the system calls etc. Write a perl script to parse the log output to get required details. More work, but good enough in most cases.

By DZone - fresh links for developers on May 18th, 2007 at 8:41 am
links from TechnoratiWhy Linux people lust after DTracebloid via softwareblogs.intel.com Published: May 18 / 04:21. Views: 51, Clicks: 42 Back in 2002, I was managing the team that worked with Oracle on the engineering side. This was when they were transitioning from

By Bryan Cantrill on May 18th, 2007 at 12:30 pm
Glad that you enjoyed the talk, Dave. In retrospect, I got a bit fooled by the "CommunityOne" moniker into thinking that the presentations were more informal than they in fact were. So if you had never seen DTrace before I started demo'ing some of our newer features, it might have been a bit of a wild ride. ;)

To address some of the issues that have come up in the comments here: yes, DTrace has been ported to FreeBSD and MacOS X Leopard -- which we're naturally very excited about. (The thought of potentially having DTrace on my phone is almost too much excitement to bear.)

In terms of SystemTap and DTrace, I would refer interested parties to Stephen O'Grady's blog entry on the topic.

As for the license issues, I will only comment that the issue here is with the GPL, not with the CDDL -- as far as CDDL is concerned, you can drop DTrace into Linux and off you go...

By Bryan Cantrill on May 18th, 2007 at 12:32 pm
Whoops -- no HTML syntax apparently; here's the DTrace/SystemTap link:

http://redmonk.com/sogrady/2006/04/07/linux-responds-to-dtrace-systemtap-on-tap

By Running in the Rain on May 19th, 2007 at 9:11 pm
links from TechnoratiMy posts are not particularly popular - I might get 100 or so views of an average post, and that would be over several weeks. Imagine my surprise when one of my posts this week got over 10,000 views in just two days. The post wasnot that impressive

By Steve on May 20th, 2007 at 5:07 am
The problem with DTrace that it only help to the developers of crap applications who tries to push the debugging to the system administrators.
Poor sysadmins then have to use DTrace to figure out why that crap application doesn't work as expected.
It's not a cool application, it only something that a well designed system like Linux should avoid to implement.

No wonder, Solaris needs to have such a tool..

By David Stewart on May 21st, 2007 at 7:49 am
Hmmm ... methinks someone is trying to bait me! :-)

Having been in the real world a few years, I can say for sure that people's intentions often don' reflect reality. The reality is that there are plenty of applications which started life as a prototype, but which went on to become the production system without being rewritten.

Most often, these systems go into production and later on as more load comes in, there are strange performance problems or bugs, and there is no good way to instrument live code on a production system to see what's up.

This is where DTrace, built into Solaris, really saves some people's bacon.

By Michael Shadle on May 21st, 2007 at 1:47 pm
ZFS is another big advantage Solaris has. A couple BSD's are working on implementing that as well, as well as OS X (supposedly) - so it looks like as far as licensing is concerned, the BSD types go quite well with the CDDL types.

By David Stewart on May 21st, 2007 at 2:01 pm
Yes, BSD is compatible with CDDL licensing.

We could probably do a whole 'nother thread on ZFS and it's pros and cons. I have heard nothing but high praise for ZFS from those who are familiar with it, though I have also heard from some Linux people that it violates some architectural principle for them in terms of layering of abstractions.

By Michael Shadle on May 21st, 2007 at 3:13 pm
I think that might be just another excuse to get away from trying to port a ZFS clone (since right now it looks like ZFS won't be supported in Linux officially due to licensing [perhaps] and definately due to how it handles some kernel-level stuff)

At this point I could care less about who has a better license, etc. I want more ZFS style options out there, having inline data redundancy/checksumming and corruption tolerance with the ability to scale and be device agnostic is fantastic. It would almost be enough by itself for me to switch to OS X as my desktop OS. Worrying about data corruption (which has happened to me a lot lately it seems) is something that I would like to do without.

By tecosystems » links for 2007-05-24 on May 23rd, 2007 at 9:33 pm
[...] Intel® Software Network Blogs » Blog Archive » Why Linux people lust after DTrace some thoughts from an Intel dev on the value of the observability DTrace provides (tags: DTrace Solaris Linux observability Intel) [...]

By James Dickens on August 2nd, 2007 at 1:10 pm
http://uadmin.blogspot.com/2006/05/what-is-dtrace.html is a document I wrote over a year ago in May of 2006. It has numerous examples of what DTrace is and how it has been used to solve problems by both DTrace designers and its users on production and non-production machines.

If Systemtap is so great, where is a similar document showing how Systemtap has been used to solve problems in both production and non-production systems?

At this point, the people saying that Systemtap is equal or even better would be the equivalent getting advice on which beef is the best from vegans and the sales people trying to sell you a cow.

In short "Where's the Beef!"

By Adam Leventhal's Weblog on August 6th, 2007 at 7:55 pm
links from Technorati, and they, in turn, helped DTrace by growing the community and through direct contributions[2]. We love to see DTrace on other operating systems, and we're happy to help. So to the pretenders: enough already with the knockoffs.Your users want DTrace, you obviously want what DTrace offers, and the entire DTrace team and community are eager to help. I'm sure there's been some FUD about license incompatibilities, but it's certainly Sun's position (as

By links for 2007-09-12 at ???? - V12N on September 11th, 2007 at 4:42 pm
[...] Intel® Software Network Blogs » Blog Archive » Why Linux people lust after DTrace (tags: dtrace) [...]

By parga on October 28th, 2007 at 12:07 pm
i am looking for carrier in linux but after linux i want some thing which give me good post in MNC so please just reply me that after working in the linux then which course is better to get my carrier better and safe

By Donghai Ma on December 14th, 2007 at 4:17 am
links from TechnoratiIntel® Software Network Blogs » Why Linux people lust after DTrace

By Adam on April 2nd, 2008 at 8:15 pm
Good story, David :) !!! I'm realy think that Solaris feature that Linux....

By paul fox on May 4th, 2008 at 2:24 am
dtrace for linux does exist - i am busy getting it to work. i have the kernel drivers and userland dtrace binary compiled and loaded. i am slowly working my way through the features and mechanisms, and hope to announce more widely when i feel its ready for prime time (at the moment its like a car jacked-on-wheels!)


What do you think?

Name (required)

Email (required; will not be displayed on this page)

Your URL (optional)

Comments (required)