For those of you left reading this paragraph, let me clue you into a little secret.  That serial code that still needs to execute is dragging down your overall performance.  Sure, when you first ran a profile analysis, it only accounted for 5% of your execution time and the other 95% could be made parallel.  Since you knew this was going to be the case at the outset of development, you probably didn't even try to write an optimal algorithm for that piece.  However, now that you've addressed that 95% of the application, the untouched serial code time has become 30% of the execution time.  When you get more cores and can spawn more parallelism, that percentage is only going to get bigger.

Since it can't be made parallel (or you would have already done that), the only option to get more performance out of the application is to look into serial tuning.  I know you ran through all the serial optimizations you could in the portions of code that are now parallel before the transformation.  Now it's time to go back to the quick and easy code you tossed in just to get the application done and running.

Even though Berkeley professor David Patterson says "Serial computing is dead," I want to be sure that you're not throwing the baby out with the bath water.  In fact, I want to make sure you aren't even throwing out the bath water.  Even if we were to make all programming parallel, we still need to ensure that students on their way to being software professionals still understand the importance of things like loop unrolling, data alignment, loop ordering for memory access, and branch misprediction.

Even as things strive to go fully parallel, can we ever get beyond the need for serial optimization?  Probably not.  Even if we're able to compute with quantum states or DNA molecules or on abacuci because we've run out of energy, there will still be portions of algorithms that must be sequential.  We will need to be sure to use the best algorithms and program in the most efficient way for the processing medium.

Specifically, the comment that "Sequential programming is really hard" got me to thinking about thousands of talking Barbie dolls that once caused an uproar with her comment that "Math is hard."  I thought up comments about checking Prof. Tanenbaum for a drawstring between his shoulder blades.  (Shame on me; no, really.)  Then, I read the rest of his comments about how software is still pretty fragile and causes system crashes with an alarming frequency.  The second part of Tanenbaum's sentence about parallel programming being a step beyond that was sobering.

I'm not sure how I feel about sequential programming as a difficult activity.  We've moved beyond methods and technologies that had their productivity drawbacks, like patch cords, assembly language, punch cards, and rigidly formatted source lines.  Even so, I often think that in our quest to make programming easier or more reliable, we've heaped new methods and technologies on ourselves that can easily complicate things more than previous things have been.

If our current programming methods, models, and environments are complex enough that we can't write software to be much more reliable than a broken watch, then what will it be like when we try to wedge parallel programming and all those attendant complexities into those same methods, models, and environments?  This is what I took to be Prof. Tanenbaum's point in his quoted comments.  (And I renew my humility at his sagacity.)

What about chucking out all the flotsam and detritus we've accumulated over the last 40 years of programming and start with something new, something that starts at square [0] with parallel thought processes?  First: if you thought this, please hit yourself with a rolled up newspaper for me.  Second: realize that starting fresh may ultimately be a good choice and allow us to focus on the problems of parallelism and overcome the failings of current methods, but it's tough to get entrenched methods replaced by untried technologies.  Besides, who wants to take all the time required to completely rewire our modes of thinking and teaching, when they can get by with the mostly status quo?  Or the status quo with some parallelism wedged in a few slots?

Prof. Patterson summed it up well with, "In order for parallelism to succeed, it has to result in better productivity, efficiency, and accuracy."  For better or for worse, "market forces" will drive a winner in the parallel programming arena.  Will it be the best solution?  Maybe, but I don't think Prof. Tanenbaum will be holding his breath.  It certainly won't be the only solution we see in our near future.

Part 1 - Recognizing Parallelism. This lecture points out why parallelism is important in today's software development environments and how to recognize 2 types of opportunities for parallelism (task and data decomposition) within application code. Examples could start with real-world examples of processes that are "executed" in parallel and then provide a programming example.

Part 2 - Shared Memory and Threads. This lecture give more ideas about how the models covered in the previous part are found within serial code, how threads would be used to express that parallelism, and the major concerns programmers need to be aware of when working in shared memory with threads (i.e., data races).

Part 3 - OpenMP Programming.This lecture gives a quick introduction on how to parallelize loops with independent iterations using OpenMP and how OpenMP deals with some of the concerns expressed in the previous lecture. This is intended to give students a chance to actually do some simple parallel programming and see the performance benefits of parallel execution.  

Of course there are many more than three things that I would want to put into a university curriculum about parallel programming. If nothing else is ever taught on the topic in a student's academic career, this minimal set prepares for their own explorations of alternate threading models (e.g., Windows Threads, Pthreads, Intel(R) TBB) or alternates to shared-memory parallel programming (e.g., message-passing, functional languages).

Proposals for adding parallel programming to a Computer Science curriculum frequently gets pushback asking "What do we eliminate from our already over-stuffed curriculum in order to add parallel programming?" The half hour limit is intended to address this concern. I see these three lectures dropping into a first programming class (CS1) that used a language supporting OpenMP, right after loops and loop constructs were discussed. Ninety minutes out of a semester of contact (60 minutes, 3 times a week, for 15 weeks = 2700 minutes) in such a course could be set aside for these topics.

To paraphrase Dennis Miller, "...of course, this is just my opinion. I could be wrong." Does this list cover the minimum set of parallel programming knowledge? Could it be done with only two topics or is there a fourth (or fifth) vital element that I've overlooked? Is shared-memory programming on multi-core processors the way to introduce/practice these topics or would this just scar future programmers? If you were to answer the original question at the top of this post, how would you respond?

I found an Apple press release on-line about the new Snow Leopard OS that also describes Grand Central as technology to "[make] it easy for developers to create programs that take full advantage of the power of multi-core Macs." Thinking about this for a few moments, I realized that this could be anything. I mean, the on/off button makes it easier for developers to create parallel programs. That was too simple, so I went back to my search results.

After all the sites about Grand Central Station in The Big Apple, I came to a NY Times blog post by John Markoff.  This article reports Jobs saying that Apple has made a "parallel-programming breakthrough," but offers no more details than that.  Strike Two. 

Then I thought, surely someone from Intel must have made some kind of mention of the Snow Leopard OS announcement. I found Josh Bancroft's post dissecting points from the announcement. Looking for what was written about Grand Central, I found more general talk about this and that and better support for better performance blah blah blah.

*sigh*

Jobs has always been a cagey fella, so I'm thinking that he's probably playing this close to the vest. More details will follow when the time is right, I'm sure. Part of the fun of Christmas is the anticipation, so let's hope the payoff of Grand Central will have been worth all the mystery.

Karmella Haynes, biologist at Davidson College, is quoted as saying "Imagine having the parallel processing power of a million computers all in the space of a drop of water."  Sounds cool, but I've always wondered how we would interface with a drop of water to get data in and out.  You can have the fastest and most parallel computer, but it doesn't do anyone any good if we can't get the answer out.  Perhaps, in anticipation of this problem, the researchers are also working on a bacteria that will light up when it has a solution.

Haynes does qualify the above by stating that the systems currently under development will only be capable of solving mathematical puzzles like how to flip pancakes such that they end up in sorted order with all the burned sides down.  As she says, we won't be using bacteria to run our iPods, for a while anyway.  Sounds to me to be more like an ickPod when that day arrives.

The interesting feature is how the system is designed to work.  The input spoken word is first translated to text with Automatic Speech Recognition (ASR) software, the text is translated with Spoken Language Translation (SLT) software to the target language, and this text is used to generate spoken output with Text to Speech (TTS) software.  Over the course of the TC-STAR project, translation performance has improved from 40 percent to 60 percent. 

A quad-core processor should be able to handle three applications running in cooperation.  Use of a single processor might even give a performance boost (over three single-core chips) by being able to pipeline results more efficiently.  No details were given in the article on what kind of processors were utilized, but there was mention of using several ASR and SLT systems to improve the translations.  This combination by itself would be a good use of multiple cores.

It will require the cooperation of aliens offering us their SLT and TTS pieces if we're going to be able to build a universal translator based on this technique.  I'm skeptical that dolphins would be able to give us any transcription software.  However, even without opposable thumbs, if we could give them access to water-proof laptops and C++ manuals, who knows what they might come up with.

It's an intriguing notion (and something that will require lots of processing power from manycore chips).  The classic SF chestnut of using mathematics to gloss over how two alien cultures begin communicating has some believability.  In "Contact," aliens initiate communication with humans through prime numbers.  Is it too hard to imagine that, even though planets might vary widely, there are enough environmental commonalities to be able to find correlating "words" between us and them.  We have rocks, aliens have rocks.  We have hands, aliens have tentacles (but when they say "g9l%Bi*df" and wave them, we'll figure out what they mean).

I'm not convinced that we could be able to translate smells (Priest-Kings of Gor) or light emissions (Toshi in Pandora's Star) in truly alien races.  I'll be impressed enough when we can get such a device to work between different Earth languages (Opish, Ubbi Dubbi, or Pig Latin, anyone?).  When we can talk to dolphins, then I'll really think we might actually communicate with aliens using a universal translator device.  (I just hope "Klaatu Borada Nicktoe" doesn't end up being "Don't forget to pick up Klaatu's dry cleaning.")

But, can you really treat 50 disks as if it were RAM?  Kunkle and Coopperman point out that the disks were distributed across their cluster, so access should tend toward accessing the local disk rather than disks across the network. Also, because of their physical restrictions and associated latencies, disks aren't really random-access devices. These two points need to be taken into account when developing the parallel algorithms that will attempt to make the most efficient use of multiple disks as if they were a very large RAM.

It's a pretty keen idea and published none too soon if the April 2008 issue of IEEE Computer is any indication. The focus of that publication is "Data-Intensive Computing."

Using multiple disks on a cluster, as Kunkle and Coopperman did, is all well and good. I'm still not sure how or if this idea might be able to translate down to a single manycore chip. The cluster's multiple nodes all have separate connections for disks; a single chip only has so many paths onto it. We might be able to fill up memory from multiple disks, but we are still going to need some way to push that data into the chip for all the cores to compute on.

Using gestures, rather than a mouse and keyboard, would be a natural method for folks to interact with the keyboard. Researchers from two Fraunhofer Institutes demonstrated how hand waving and finger pointing could be used to operate a computer. Cameras watch the user's gestures and interpret the intention. Here again, certain gestures don't translate across all cultural boundaries, so we might see a system that needs to first learn a user's intentions.

All this is really cool work. It is certainly more useful (and less scary) than having Braille keys on a drive-up ATM. However, as I got to thinking about it more, I began to wonder what someone that couldn't read or write might do if they had easy access to a computer. I thought about all writing and working with numbers I did every day. That would be out. 

I decided that it would have to be something visual. Something to do with pictures or movies. My mind quickly ran to the trailer for the latest Bollywood feature, slide shows of paintings by van Gogh or Renoir, and travelogues of distant countries. Unfortunately, my mind just as quickly ran to examples of still and moving pictures on the Internet that tend to be rated by a letter between "W" and "Y".

Since I can't seem to get my mind out of the gutter, what kinds of content would you think anyone that can't read or write would want to "use" on a computer? And what kind of icon (picture) would you use to represent that content?

Both articles report on significant, newsworthy subjects.  But, after my St. Patrick's Day imbibements (luckily I retain my ability to do long division), one of the last things I wanted to read was the phrase "Intel's...chips, kind of the brain of a computer" [my emphasis] while crunching down my morning granola.  Later in the week, I see "Brain" right in the WSJ headline.

Here in Central Illinois I can understand the local paper feeling the need to bring a technical story down to the level of the soybean and corn farmers in the area, even with one of the premier CS and EE schools in the country located right down the street from the paper's offices. However, what is the lowest common denominator reader that the Wall Street Journal is aiming for?  Surely their average reader would consider herself at least as technically savvy as someone that raises tofu.

I don't get it.  I've seen this terminology used before, so this wasn't an isolated incident. Maybe I'm expecting too much, having been insulated within technical communities for almost 30 years, or I'm giving my fellow Man more credit than he deserves.  I hope not.