Opening TBB 2.0 code and providing it as free software gave us an excellent source of new ideas for improvements and enhancements in the library, in addition to the feedback from customers. The pipeline implementation also got a few. All the proposals were reviewed by the team; and the following were implemented:

Fixed idle spinning of worker threads in case of blocked input
Threads spinning idle always were the area of concern for TBB customers; I can recall a few places where we had to minimize idle spinning. The issue in pipeline was missed for some time however, because we did not expect that a thread may block inside input stage. It was obviously an oversight; pipeline users wanted it to read the data coming dynamically at runtime, e.g. from network, therefore waiting for data inside the input filter is very possible. The fix for the issue required significant overhaul of the implementation; however it was so important that we released it as part of TBB 2.0 Update 3. And the new implementation allowed the next improvement.

Pipeline input stage allowed to be parallel
In TBB 2.0 and before, the pipeline input stage was always serialized, even if the corresponding filter was constructed as parallel. After the aforementioned fix, it became possible to allow input stages run in parallel; and we implemented this improvement.
Be aware that if you used a “parallel” input stage with TBB 2.0, switching to TBB 2.1 may reveal some hidden pitfalls. In order to work correctly, the parallel input filter should conform to some non-formalized requirements:
- it should be thread-safe, as it will be called by different threads simultaneously;
- it should be tolerant to latecomers, because it can be called even after one of the previous calls returned NULL;
- it should better not block, otherwise every thread could block there.
Interestingly, this change has some affect on serial stages. As you might know, serial stages of TBB pipeline are in fact strictly ordered. In TBB 2.0, the order was defined exactly by the input stage; and all subsequent serial stages processed items (tokens) in this order. But now, the input stage can be parallel; in this case, the token order is defined by the first serial stage encountered in the pipeline. It happened to be easier than thread-safe token number assignment for parallel input stage; and it is better this way, because the first serial stage runs as "first come first served" that might improve total throughput.

Enumeration is used to distinguish filter types
It’s an example of community influence on TBB: we were requested at the forum to change Boolean parameter to an enumeration for filter type specification. Even if we only have serial vs. parallel filters, using enumeration improves code readability and allows compile-time check of the argument. So now you could write MyFilter::MyFilter() : tbb::filter(serial) {} instead of MyFilter::MyFilter() : tbb::filter(/*is_serial*/ true) {}. It really is easier to read, isn’t it? Thanks to the community for pointing the developers to these small but handy improvements.

Automatic removal of filters
This improvement in how the pipeline object should be destroyed was the last of the changes accepted for the next version of TBB. In TBB 2.0, one needs to call clear() for the pipeline prior to filter deletion; it is necessary for safe pipeline destruction, and it is documented. But why can’t filter destructor remove the filter from its pipeline? Well, now it can. But doing this in the way that is compatible with the previous implementation was somewhat tricky. We needed to change the single-linked list of filters to become double-linked, and add a pointer to the pipeline. Thus the layout of the filter class was changed, but user-defined filters inherit this class – so how we keep it backward compatible? Check the code to find the answer. :)

You might check the improved pipeline functionality with all the above features in the recent open-source stable release (at the moment of writing, it was tbb21_20080605oss).

There also were a couple of internal proposals we declined, and there remains a list of things still to be done or under consideration, such as:
- We added general support for cancellation and handling of exceptions to TBB (see Andrey Marochko’s blog for interesting details); now the pipeline should be improved to work correctly in case of cancellation.
- Type-safe filters were also proposed at the forum together with the implementation. We like the idea of compile-time type checks and operator>> semantics to build a pipeline, and would be glad to elaborate on it and improve the implementation; but unfortunately the author did not contribute it officially to the project.
- Out-of-order serial filters were considered and postponed because we did not see a really good motivation case. A case was recently proposed at the forum where unordered serial filter might improve output latency.
- The idea of automatic selection of the maximal number of tokens in flight was proposed, but it’s hard to find heuristics that would work well enough for every use case.
- And some others.
So we are not yet at the end of tbb::pipeline evolution.

And by the way, have you noticed how many times I mentioned the TBB community and the forum? If you have a great idea for TBB, you are very welcome with it. We listen to you; leaving a comment in this blog is a simple way to make your contribution. :)

 With multicore processors becoming widespread, parallel programming moves to mainstream. As indirect evidence, recently there are seen many attempts to develop a simple parallel benchmarking test to see performance benefits from multicore and multithreading. And with no doubt this is good, but…

 I have observed a few such attempts where parallel code used the Threading Building Blocks library (TBB). Much to experimenters’ astonishment, not only their simple parallel programs sometimes expose no reasonable speedup but even those can be slower than sequential counterparts! And unfortunately, not every program author could (and wants to) understand why.

 Being one of the TBB developers at Intel, in some cases I had to analyze what was wrong with these codes. And since we were told that TBB developers should blog more, I eventually decided to try it out, and I will start with writing about these cases. Those on the one hand are simple and comprehensible; most developers do not start their parallel experiments with complex algorithms but with something simple like a loop to sum all array elements. On the other hand, these examples can as well be interesting.

 The array summation is a good example to start :). The question about why so simple a program became slower with TBB had arisen at the TBB forum (see “performance of parallel_for” there). Here is relevant part of the code:

#define REAL double
#define MAX 10000000
class SumFoo {
    REAL *a;
    REAL sum;
    ...
    void operator()(const blocked_range<size_t>& r) {
        REAL *arr = a;
        for(size_t i=r.begin(); i!=r.end(); i++) sum+=arr[i];
    }
};
void ParallelSumFoo(REAL a[], size_t n, size_t gs) {
    SumFoo sf(a,n);
    parallel_reduce(blocked_range<size_t>(0,n,gs), sf);
    ...
}
class SerialApplyFoo {
    REAL *a;
    size_t len;
    double sum;
public :
    SerialApplyFoo(REAL *array, size_t length) : a(array), len(length){
        sum = 0.0;
        for(size_t i=0; i<len; i++)
            sum += a[i];
    }
    …
};
int main() {
    task_scheduler_init init;
    REAL *array = new REAL[MAX];
    ...
    SerialApplyFoo sf(array, MAX);
    ParallelSumFoo(array, MAX, GRAINSIZE);
    ...
}

On a system with Intel® Core® 2 Duo processor, instead of expected 2x speedup, ParallelSumFoo was almost twice slower than SerialApplyFoo, e.g. on my laptop it ran for 0.08 sec vs 0.044 in serial. Let’s see what was bad there.

 First, let’s examine what time the TBBfied code takes to run sequentially. There are two ways for that; one is to explicitly specify the number of threads when creating task_scheduler_init object for library initialization, and the other is to set the grain size of parallel_reduce equal to the array size. I used the first way:

    task_scheduler_init init(1);

And the result was kind of shocking: 0.21 sec, which is almost 5 times slower than the original serial code! Now no wonder that on two cores it was also slow; and TBB can’t be the direct reason of such slowdown.

 The root of the issue is in optimization by compilers. The serial loop is very simple to optimize; the compiler knows all about it, from the constant length to the fact that sum should only be visible after the constructor of SerialApplyFoo completes. So it can generate very efficient machine code for this loop; for example, Visual* C++ 2005 compiler produced the following:

            fldz
            lea      eax, DWORD PTR [edi+010h]
            mov      ecx, 0x2625A0h
main+0x105: fadd     QWORD PTR [eax-16]
            add      eax, 0x20h
            sub      ecx, 0x1h
            fadd     QWORD PTR [eax-40]
            fadd     QWORD PTR [eax-32]
            fadd     QWORD PTR [eax-24]
            jnz      main+0x105

The magic 0x2625A0 constant is 2500000, one fourth of the array length. The compiler applied loop enrolling optimization and replaced four iterations by one, thus decreasing overhead in the loop. The intermediate sum is accumulated in a register and only stored to memory once at the end (not shown). And there is exactly one memory load (of the needed array element) per iteration of the original loop.

 The TBBfied code is not that simple for the compiler. Start and end of the loop are specified in data fields of an object passed by reference; loop length is unknown; the result is stored in a data field of an active object. The compiler has to be more conservative:

execute+0x57: fld      QWORD PTR [edi]
              add      ecx, 0x1h
              fadd     QWORD PTR [edx+08h]
              add      edi, 0x8h
              fstp     QWORD PTR [edx+08h]
              cmp      ecx, DWORD PTR [esi+08h]
              jnz      execute+0x57

In this code, there are 3 memory loads (!) and 1 store to process just one array element. Besides the element itself, the loop end value is loaded every time, as well as the intermediate sum which is then stored back only to be reloaded at the next iteration.

 I wrote the above considerations in the forum, but to bad luck I said that the benchmark “favors” the serial case; by that I probably pointed the author to wrong direction. He tried to improve the test by reading the array length from console (which is good because it makes the test more realistic) and replaced SerialApplyFoo object to the following function:

REAL SerialSum(REAL *a_, unsigned long int start, unsigned long int end) {
    REAL sum = 0.0;
    for(unsigned long int i=start; i<end; i++)
        sum += a_[i];
    return sum;
}

But this change has next to no effect because now the intermediate sum is accumulated in a local variable and compiler uses a register for it with the same ease as before, and this is good. Slowing down the serial code shouldn't be the goal; to solve the problem, the parallel code should be improved to simplify optimizations for the compiler; namely, local variables should be used in the loop in operator():

    void operator()(blocked_range<size_t> &r) {
        REAL local_sum = 0;
        const size_t end = r.end();
        for(size_t i=r.begin(); i!=end; i++)
            local_sum+=a[i];
        sum += local_sum;
    }

 This is the corresponding machine code I got:

              fldz
execute+0x58: fadd        qword ptr [edx]
              add         edx,8
              sub         ecx,1
              jne         execute+58h
              fadd        qword ptr [edi+8]
              fstp        qword ptr [edi+8]

Like in the serial code, there is just one memory load per iteration, the intermediate sum is accumulated in a register and then added to the sum field of SumFoo (see the last two instructions). And it significantly improved the speed: now ParallelForSum completes in 0.033 sec if both cores are used, so the speedup of 1.33 is achieved. But why not 2x, you might ask. Well, it might be a subject of another post.

 Conclusion: when developing programs with TBB, you should take into account that using TBB classes and functions may impact compiler optimizations, which has especially bad impact on simple algorithms with small amount of work per iteration. Proper use of local variables helps optimization and improves parallel speedup.