Here's also a chance to meet Ashok Raj, one of the Intel senior kernel guys working on OpenSolaris.

Original video source

I love reading case studies of teams that shake up "business as usual" and reinvent their world.  It inspires me to be open to the possibilities of such change.

If you are involved at all in software engineering, check out this blog entry by the Songbird development team as it shows their shift from the traditional waterfall model to Agile methods.

There are some good ideas here, some of which might be adoptable to your own work.

The traditional waterfall model has taken a huge number of deadly arrows over the course of my little lifetime in tech:

• The book Microsoft Secrets shows that in the early 90s, Microsoft couldn't manage their humongous Office projects with the traditional model and went to an "synch and stabilize" model to get things under control.
• Practices on the Internet have created an implicit demand that every downloadable software project will have both a stable version and the latest development version available for download. This doesn't work with the traditional waterfall model, because development builds for the waterfall model are not guaranteed to work.
• Back in my old Sequent days, working on the OS, we always had the standard that "you never break the build" but we never put our weekly builds out on the Internet for just anyone to download. I'm incredibly impressed with the stability and quality of bi-weekly builds of huge projects like OpenSolaris. For most purposes, you could probably take one of those builds and chances are, be quite happy with it.
• Yet, many famous and hugely profitable software enterprises seem to run on the waterfall model from all external evidence. I don't need to mention these.

By the way, Songbird is a fine open source music service / desktop media player. It's like a marriage of iTunes and Firefox. Worth a look. Screenshot below.

I gave my keynote this morning to kick off the last day of the conference. Now that I can relax a little, I'm listening to a couple of talks on DTrace.

One comment I heard from the current speaker is that SystemTap in Linux works by compiling a script into C code, compiling the C code and then linking it into the kernel.

This is in contrast to DTrace, which takes the script written in the "D" language into an intermediate code (DIF) which is then interpreted by the in-kernel DIF virtual machine.

So, the SystemTap method seems really … um … unsafe. Particularly if you are linking into a running production system to trace it.

Now I suppose you can do Really Bad Things in DTrace as well. So maybe this is a difference without a distinction. (I was just told that there is a "allow destructive behavior" option in the DTrace implementation which is off by default.) Is a virtual machine interpreter any safer than generated C code?

Does anyone have experience with both, who can comment?

I'm listening to Alexander Eremin, who is the originator of MilaX, a distro of OpenSolaris designed to be as small as possible.

MilaX 0.1 was announced in February 2008, and there was a storm of interest as a result.

• This was intended to be a very small distro of OpenSolaris which would have all of the Solaris features like ZFS, DTrace, and all of the drivers.
• You can boot this on old hardware with few resources and don't require the whole system included with Solaris.
• There are two versions of MilaX now, one oriented for desktops and one for servers.  The server version is optimized to get an Apache web server going quickly.
• It looks like today, you need 512MB of memory to run MilaX, of which 105MB is for the RAM disk image of MilaX.
• MilaX boots from USB sticks or other devices.
• Currently, MilaX doesn't use the new Image Package System, so future work includes making MilaX available as a pkg(1) package.  Alex wants to make booting faster as well.
• http://www.milax.org to get involved.

The opening keynote at the conference is by Jim Grisanzio, who is a sr. program manager at Sun working on building the community.  Jim's slides were 100% photos.  A far cry from my boring slides!  Here are some highlights from his talk.

Jim started out in the construction industry 20 years ago, and there are interesting parallels with starting up a community!

There are criticisms that OpenSolaris is not organic like Linux but is managed because Sun is trying to direct it.  This is probably a good criticism, but there are certainly organic things going on, like this OSDevCon, which was not done by Sun at all.  It came out of the community itself.

Jim is strongly involved in governence, being on the OpenSolaris Governing Board (OGB). The vast majority of people involved in OpenSolaris couldn't care less about governence, which is a problem for Jim.  So he is working on reorganizing the governence model to make it easier to contribute and get involved.

Three mistakes that in Jim's opinion were made in launching the community:

1. They began with governance.    In Jim's opinion, they should probably should have let the community start and let governance come later.
2. The CAB grew into the OGB. When they launched the project, they announced that there would be an OGB.  But the board was given no authority or resources or charter really.  It's more of an advisory board rather than a governing board.
3. The misjudged the interest.  They thought the community would help design the governance, but there was very little feedback on what they were doing until they had a vote on the constitution.  Later on of course, there have been controversies, but very little interest in governance.

Jim touched on the Roy T. Fielding early involvement in the project and the board and feels that the board should act more like shephards of the flock to keep interest alive.

Jim was complementary about the new engineering management for OpenSolaris at Sun, particularly the new direction being offered by Tim Cramer.  I am similarly positive about the work Tim is doing and he is making Sun management a lot more visible and transparent to the community.

Kudos also to Derek Cicero who is the project web master.

Jim closed with several key quotes from members of the community, emphasizing that code is king and those who produce code should have the loudest voice.

Good talk, Jim!

What is the most ideal platform to use for development? I attended a "tutorial" on this subject, which really turned into a mini-conference really on C++ development.

• The first talk was given by Roman Shaposhnik on Solaris and Sun Studio and how he thinks this makes for a perfect development environment. Much of his talk went through the rich suite of tools available for the developer – DTrace, D-Light for visualizing counters and microstate accounting, mdb and kmdb and dbx for debugging, and all of the *stat tools.
• Adriaan de Groot from the KDE project stood up then. Apparently Sun Studio uses an older C++ standard library, which makes it less interesting, but when this library was updated, Studio now became very interesting from a C++ purity standpoint. In fact, Studio will complain about things that gcc doesn't, and if you clean these things up you get much cleaner code. And since Studio generates both Linux and Solaris code, it becomes a nice development environment. It has taken many months to clean up these coding issues. The bottom line is that KDE4 is being developed on Sun Studio because you get better quality C++ code.
• Dennis was up next from a company called tBricks, which does algorithmic code targeted at the financial market. They use both Linux and Solaris for their server backend.

Dennis had some really interesting comments on developing on Solaris:

• He finds that build times are 2x faster on Linux than on Solaris on similar hardware. Hmmm … probably bears further analysis. (I suspect that since they are running the build over NFS that the problem is with the e1000g driver.)
• Solaris is a lot more stable for them than Linux, so it doesn't matter if it's slower
• None of their developers use Solaris on the desktop.
• They love using DTrace for tracking down customer problems. They even have customers collect data off of live running systems. But Dennis had some good comments about some things which he would like to see in DTrace.
• Dennis talked about libUmem in Solaris, which he really likes, vs. using leaks in MacOS and valgrind in Linux.
• Stable scheduler on Solaris which over multiple weeks shows more stability

So they deploy on Solaris but they develop on Linux mostly due to the build speed issue, though a few develop on Mac OS and their desktop client is being developed on Windows.

In retrospect, this would have been a good venue to introduce TBB to the audience!

But this time, downloading and installing the driver didn't work for me. Here is the kind advice from James Liu, who is always quite helpful:

>> Marvell has a binary-only distro of a driver that may be
>> useful in your case. I managed to get this working on a
>> number of laptops in the past.
>>
>> http://www.marvell.com/drivers/search.do
>>
>> Select Network Controller -> Solaris 8, 9, 10 x64 or your
>> architecture.


So far so good, but it didn't work on OpenSolaris 2008.05. I had a little time today, and dug into what's going on.

It turns out that the install scripts for this driver assume that you have a number of commands in /usr/ucb which are not populated in OpenSolaris 2008.05. Things like /usr/ucb/echo ! So, a few minutes twiddling with things and everything works now.

So beware if you install pre-IPS packages. They might include hard-coded paths which don't work any more by default!

Another note, I was in Menlo Park for some meetings yesterday and ran into Bart Smaalders. Bart is a really nice guy and one of the wizards behind a number of packaging technologies at SUN. According to him, there is a lot of hygiene work going on in the IPS packaging system. For example, my one big complaint about pkg image-update when the update is really huge is that if your update halts before it's over (say, due to a networking timeout), then you have to start all over again! According to Bart, they are working on improvements in this area which should be available "in weeks." Apparently this is all being discussed in package-discuss@opensolaris.org, a good place to go if you want to know more!

(On a side note, I just put the latest Nevada install image on a little server in my office and uploaded it to the server we use to host our FTPs internally. All 3.5GB of it! Because of our internal network is temporarily misconfigured, someone was complaining that it would take 7 hours to download the whole image!)

With the recent announcement of Indiana package updates to build 90, I decided it was time to check out the reality versus the promise!

First, I fired up my trusty Sony VAIO (a Centrino Pro based system) got on the wireless network and did


$ pkg refresh
$ pfexec pkg image-update

And that was it! I also ran the GRUB menu fixup, as recommended in the release notes. And now I'm up to b90!

The total size of the update was calculated to be around 1.1GB, so I'm thinking nearly every file must have gotten touched. But anyway, that's a lot better than a 3.5GB download!

Now to be fair, it didn't exactly work that easily the first time. My wireless connection is a bit flaky and I think the Monday after the packages were released was a pretty busy day on the server, and I got a timeout, even exporting PKG_CLIENT_TIMEOUT=200. In the face of a network timeout, there is a lot of messy complaining from the pkg(1) command that didn't seem too relevant.

Also, when I restarted the image-update, it seemed to start over from scratch. Would be nice to have it pick up where it left off, particularly with such a large update, but I can understand that this is a good 1.0 effort.

Finally, I'm not exactly sure what to do with the old BE's. I suppose it means reading the beadm man page... another time!

Anyway, all seems great in package land. I checked the 64 bit version of libc, and the new version optimized for Intel's Core2 processors are in place. Nice!

I find myself writing prose that makes me sound smarter than I really am, and I get worried!  Here is my simple analysis of the three practical advantages of open source.  <b>How am I doing here?</b> Does this correctly represent the main advantages of open source?  Comments, please.

• The ability for programmers to see clearly how functions work.  The experience of programming on an operating system or application stack where the source code is available is much more productive than if the source is not available.  Without source code, developers must depend on manual pages, documents and sample code, all of which may be out of date or harder to understand than the source itself.  The author’s own experience as a manager helping engineers program on a UNIX-like system without the source of the system calls or library calls being available shows this major advantage of open source.
• Beyond making programmers more productive, some open source projects enable developers to enhance the code that they are using.  If a function is not available, for example the support for a particular hardware device or optimized alignment of memory, the system can be changed through the use of open source.  Another example would be to make a boot sequence less “noisy” in terms of informative messages or to fix bugs.  The ability to use open source to build an improved product with new or changed features is a major value to some open source projects.
• Finally, some open source projects allow innovations developed to be fed back to maintainers of the project upstream, to raise the overall value of the project as it is experienced by all users.  Clearly work done on a bug fix or a new feature will have greater impact if it is available broadly.  Of course, this now introduces the added burden of assuring that the overall architecture, quality, reliability and compatibility of the change matches the standards set in the project.  When there are differences of opinion between innovators and maintainers, there is often the pressure to “fork” projects or create derivative distributions.

(I was having problems viewing this clip while I was in China this week. Not sure if they block blip.tv there or whether it was some other snafu. In any case, here is the YouTube link if you like, lower quality video but maybe more accessible?) http://www.youtube.com/watch?v=6313NNwWzGs