Distributed Shared Memory Programming

I spent my summer doing some kind of research on Distributed Shared Memory languages. And as a parallel programming fan, I must say I am impressed with the idea.

Assuming that you know some basic stuff about parallel programming, there are two different and more common models; distributed memory and shared memory model. As can be understood from the names they are used in different architectures, and have advantages and disadvantages accordingly.

Shared memory programming is used for smaller workstations where processing elements can reach any place at the memory with almost same latency. Distributed memory, on the other hand is used where total memory of the computer is distributed over multiple nodes, implying that some PEs can access some parts of the memory faster than the others.

Most common implementation of shared memory model is OpenMP. Programmer can define some block of the code as parallel and give some hints about parallelization and OpenMP handles the rest of the work. As memory is shared there is no need for explicit message passing for data transfer. This leads to a very simple parallelization process. However as this model requires a monolithic memory, computers you can program with this paradigm is relatively small.

Distributed memory, on the other hand does not have any theoretical limit for the number of cores you have or the amount of memory. However, since distributed memory systems are large scale systems which have memory modules at different locations, explicit message passing is required. Programmer must manage the data distribution and arrange the messaging between the nodes as well as the algortihm itself. In that sense, most common implementation of distributed memory, MPI, presents the programmer a local view of the code. Meaning that a program can directly reach only some part of the memory which is specific to some thread.

As can be understood from the other model, DSM tries to find the midpoint between the two most common parallel programming paradigms. To do that DSM presents programmer a global view, where he does not have to deal with message passing and can focus on the algorithm more. However, locality-aware code is required to get the best performance. That means, even though you are not responsible with handling the message passing, you need to know how to distribute the data among nodes and must access that data accordingly.

After you deal with some DSM languages, you get the feeling that previous programming methods you used are like assembly language where you need to fine tune every single detail of the code. However, there is a big drawback for DSM programming and that is of course performance.

Main aim of any type of parallel programming is without a doubt performance. Even though their performance shows serious increase all the time, DSM languages are still far away from being the fastest model. the reason is obvious I believe. Programmer does not have the full control over the code. The bottleneck of most parallel programs, memory access, is done by compiler and all DSM compilers still lack enough optimizations to beat MPI.

Parallel programming is still in its childhood period, and for large scale parallel programming it will still be in childhood for a while, because it is not easy to find resources. However, with the easy concepts presented by DSM model, the process of getting into adulthood will be easier for parallel programming as it makes writing parallel code easier thus reaching a wider community.