Cluster-enabled Omni OpenMP
on a software distributed shared memory system SCASH.
Implementation Notes.

Introduction

One way to support OpenMP in a distributed memory environment is to use a software distributed shared memory system (SDSM) as an underlying runtime system for OpenMP. Cluster-enabled Omni OpenMP on SCASH is an implementation of Omni OpenMP compiler for a software distributed shared memory system SCASH running under SCore Cluster System Software.

In the most SDSMs, only part of the address space is shared. In SCASH, the address space allocated by a shared memory allocation primitive can be shared among the processors. Variables declared in global scope are private in the processor. We call this memory model, ``shmem memory model''. Parallel programs using the Unix ``shmem'' system calls use this memory model. In this model, all shared variables must be allocated at run-time at the beginning of execution. We have implemented the OpenMP compiler for shmem memory model using Omni OpenMP compiler system. The compiler detects references to a shared data object, and rewrite it to the reference into objects re-allocated in shared memory area.

Software Distributed Shared Memory System SCASH

SCASH is a page-based software distributed shared memory system using the PM low-latency and high bandwidth communication library PM for a Myrinet gigabit network and memory management functions, such as memory protection, supported by an operating system kernel. The consistency of shared memory is maintained on a per-page basis. SCASH supports two page consistency protocols, invalidate and update . The home node of a page is a node that keeps the latest data of the page. In the invalidate protocol, the home node sends an invalidation message to nodes which shares the page so that the page is invalidated on these node. The invalidate protocol is used as default.

SCASH is based on the Release Consistency (RC) memory model with the multiple writers protocol. The consistency of a shared memory area is maintained at a synchronization called the "memory barrier synchronization" point such as a barrier. At this point, only modified part is transferred to update pages. Explicit consistency management primitives are also supported for the lock operations.

Translation of OpenMP Programs to SCASH

• Transformation for "shmem memory model"

  In OpenMP programming model, global variables are shared as default. To compile OpenMP program into "shmem memory model" of SCASH, the compiler transforms code to allocate a global variable in shared address space at run time. All declarations of global variables are converted into the pointers which contain the address of the data in shared address space. The compiler rewrites all references to the global variables to the indirect references through the corresponding pointers.
  For example, a code:

   double x;  /* global variable declaration */
   double a[100];  /* global array declaration */
   ...
   a[10] = x;
  

  is transformed as follows:

   double *_G_x;   /* indirect pointer to "x" */
   double *_G_a;   /* indirect pointer to "a" */
   ...
   (_G_a)[10] = (*_G_x); /* reference through the pointers */
  

  The compiler generates the global data initialization function for each compilation unit. The initialization functions are linked and called at the beginning of execution to allocate the object in shared address space and store the address into the indirect pointer.

• OpenMP directive translation and Runtime library

  The OpenMP directives are transformed into a set of runtime functions which uses SCASH primitives to synchronize and communicate between processors. To translate a sequential program annotated with parallel directives into a fork-join parallel program, the compiler encapsulates each parallel region into a separate function. The master node calls the runtime function to invoke the slave threads which execute this function in parallel. All threads in each node are created at the beginning of execution, and waits for the fork operation on slave nodes. At the fork, pointers to shared variables with auto storage class are copied into shared memory heap and passed to slaves. No nested parallelism is supported.
  In SCASH, the consistency of all shared memory area is maintained at a barrier operation. This matches the OpenMP memory model. The lock and synchronization operations in OpenMP use the explicit consistency management primitives of SCASH on a specific object.

Concluding Remarks

We have implemented an OpenMP compiler for a full set of OpenMP on the software distributed shared memory system, SCASH. We found that the page home distribution is a key to achieve the good performance on the SDSM through our several experiments on RWCP PC clusters.

Currently, we are introducing OpenMP extensions to specify data mapping in a flexible way, which gives application specific knowledge to the compiler. The loop scheduling to exploit locality for data mapping is also an important issue to investigate. Recently, we extended the directives to specifiy the data mappging and loop scheudling. We expect that these extensions improve the performance of OpenMP programs on SCASH.