![[Contents]](TOC.GIF){kind=small}
![[Prev. Chapter]](REW.GIF){kind=small}
![[Next Section]](NEXT.GIF){kind=small}
![[Next Chapter]](FF.GIF){kind=small}
![[Index]](INDEX.GIF){kind=small}
![[Help]](HELP.GIF){kind=small}
The MEMORY CHANNEL Application Programming Interface (API) implements highly efficient memory sharing between MEMORY CHANNEL API cluster members, with automatic error-handling, locking, and UNIX style protections. This chapter contains information to help you develop applications based on the MEMORY CHANNEL API library. It explains the differences between MEMORY CHANNEL address space and traditional shared memory, and describes how programming using MEMORY CHANNEL as a transport differs from programming using shared memory as a transport.
This chapter also contains examples that show how to use the MEMORY CHANNEL API library
functions in programs.
You will find these code files in the
/usr/examples/cluster/
directory.
Each file contains compilation instructions.
The chapter discusses the following topics:
Initializing the MEMORY CHANNEL API library for a user program (Section 2.1)
Accessing MEMORY CHANNEL address space (Section 2.2)
Clusterwide locks (Section 2.3)
Cluster signals (Section 2.4)
Cluster information (Section 2.5)
Comparison of shared memory and message passing models (Section 2.6)
The
imc_api_init()
function is used to initialize the MEMORY CHANNEL API library in a user program.
Call the
imc_api_init()
function in a process before any
of the other MEMORY CHANNEL API functions are called.
If a process forks, the
imc_api_init()
function must be called before calling any other API
functions in the child process, or an undefined behavior will result.
![[Prev. Section]](PREV.GIF){kind=small}
The MEMORY CHANNEL interconnect provides a form of memory sharing between MEMORY CHANNEL API cluster members. The MEMORY CHANNEL API library is used to set up the memory sharing, allowing processes on different members of the cluster to exchange data using direct read and write operations to addresses in their virtual address space. When the memory sharing has been set up by the MEMORY CHANNEL API library, these direct read and write operations take place at hardware speeds without involving the operating system or the MEMORY CHANNEL API library software functions.
When
a system is configured with MEMORY CHANNEL, part of the physical address space of the
system is assigned to the MEMORY CHANNEL address space.
The size of the MEMORY CHANNEL address
space is specified by the
imc_init
command.
A process accesses
this MEMORY CHANNEL address space by using the MEMORY CHANNEL API to map a region of MEMORY CHANNEL address
space to its own virtual address space.
Applications that wish to
access the MEMORY CHANNEL address space on different cluster members can allocate part
of the address space for a particular purpose by calling the
imc_asalloc()
function.
The
key
parameter associates
a clusterwide key with the region.
Other processes that allocate the same
region also specify this key.
This allows processes to coordinate access to
the region.
To use an allocated
region of MEMORY CHANNEL address space, a process maps the region into its own process
virtual address space, using the
imc_asattach()
function
or the
imc_asattach_ptp()
function.
When a process attaches
to a MEMORY CHANNEL region, an area of virtual address space the same size as the MEMORY CHANNEL
region is added to the process virtual address space.
When attaching the region,
the process indicates whether the region is mapped to receive or transmit
data, as follows:
Transmit--Indicates that the region is to be used to transmit data on MEMORY CHANNEL. When a process writes to addresses in this virtual address region, the data is transmitted over the MEMORY CHANNEL interconnect to the other members of the MEMORY CHANNEL API cluster.
To map a region for transmit, specify the value
IMC_TRANSMIT
for the
dir
parameter to the the
imc_asattach()
function.
Receive--Indicates that the region is to be used to receive data from MEMORY CHANNEL. In this case, the address space that is mapped into the process virtual address space is backed by a region of physical memory on the system. When data is transmitted on MEMORY CHANNEL, it is written into the physical memory of any hosts that have mapped the region for receive, so that processes on that system read from the same area of physical memory. The process does not receive any data that is transmitted before the region is mapped.
To map a region for receive, use the value
IMC_RECEIVE
as the
dir
parameter for the
imc_asattach()
function.
A process can attach to a MEMORY CHANNEL region in broadcast mode, point-to-point mode, or loopback mode. These methods of attach are described in Section 2.2.1.
Memory sharing using the MEMORY CHANNEL interconnect is similar to conventional shared memory in that, once it is established, simple accesses to virtual address space allow two different processes to share data. However, there are two differences between these memory sharing mechanisms that you must allow for, as follows:
When conventional shared memory is created it is assigned a virtual address. In C programming terms, there is a pointer to the memory. This single pointer can be used both to read and write data to the shared memory. However, a MEMORY CHANNEL region can have two different virtual addresses assigned to it: a transmit virtual address and a receive virtual address. In C programming terms, there are two different pointers to manage; one pointer can only be used for write operations, the other pointer is used for read operations.
In conventional shared memory, write operations are made directly to memory and are immediately visible to other processes reading from the same memory. However, when a write operation is made to a MEMORY CHANNEL region, the write operation is not made directly to memory but to the I/O system and the MEMORY CHANNEL hardware. This means that there is a delay before the data appears in memory on the receiving system. This is described in more detail in Section 2.2.5.
The following subsections describe the ways in which a process can attach to MEMORY CHANNEL address space. There are three ways in which a process can attach to MEMORY CHANNEL address space, as follows:
Broadcast attach
Point-to-point attach
Loopback attach
This section also explains initial coherency, reading and writing MEMORY CHANNEL regions, latency related coherency, and error management, and includes some code examples.
When one process maps a region for transmit, and other processes map the same region for receive, the data that the transmit process writes to the region is transmitted on MEMORY CHANNEL to the receive memory of the other processes. Figure 2-1 shows a three-host MEMORY CHANNEL implementation that shows how the address spaces are mapped.
With the address spaces mapped as shown in Figure 2-1, note the following:
Process A allocates a region of MEMORY CHANNEL address space.
Process
A then maps the allocated region to its virtual address space when it attaches
the region for transmit using the
imc_asattach()
function.
Process B and Process C both allocate the same region of MEMORY CHANNEL address space as Process A. However, unlike Process A, Process B and Process C both attach the region to receive data.
When data is written to the virtual address space of Process A, the data is transmitted on MEMORY CHANNEL.
When the data from Process A appears on MEMORY CHANNEL, it is written to the physical memory that backs the virtual address spaces of Process B and Process C.
An
allocated region of MEMORY CHANNEL address space can be attached for transmit in
point-to-point mode to the virtual address space of a process on another node.
This is done by calling the
imc_asattach_ptp()
function with a specified host as a parameter.
This means that
writes to the region are sent only to the host specified in the parameter,
and not to all hosts in the cluster.
Regions attached using the
imc_asattach_ptp()
function
are always attached in transmit mode, and are write-only.
Figure 2-2
shows a two-host MEMORY CHANNEL implementation that shows point-to-point address
space mapping.
With the address spaces mapped as shown in Figure 2-2, note the following:
Process 1 allocates a region of MEMORY CHANNEL address space.
Process
A then maps the allocated region to its virtual address space when it attaches
the region point-to-point to Host B using the
imc_asattach_ptp()
function.
Process 2 allocates the region and then attaches it for receive,
using the
imc_asattach()
function.
When data is written to the virtual address space of Process 1, the data is transmitted on MEMORY CHANNEL.
When the data from Process 1 appears on MEMORY CHANNEL, it is written to the physical memory that backs the virtual address space of Process 2 on Host B.
A region
can be attached for both transmit and receive by processes on a host.
Data
written by the host is written to other hosts that have attached the region
for receive.
However, by default, data written by the host is not also written
to the receive memory on that host; it is written only to other hosts.
If
you want a host to see data that it writes, you must specify the
IMC_LOOPBACK
flag to the
imc_asattach()
function
when attaching for transmit.
The loopback attribute of a region is set up on a per-host basis, and is determined by the value of the flag parameter to the first transmit attach on that host.
If you specify the value
IMC_LOOPBACK
for the
flag
parameter, two MEMORY CHANNEL transactions occur for every write,
one to write the data and one to loop the data back.
Because of the nature of point-to-point attach mode, looped-back writes are not permitted.
Figure 2-3 shows a configuration in which a region of MEMORY CHANNEL address space is attached both for transmit with loopback and for receive.
When a MEMORY CHANNEL region is attached for receive, the initial contents are undefined. This situation can arise because a process that has mapped the same MEMORY CHANNEL region for transmit might update the contents of the region before other processes map the region for receive. This is referred to as the initial coherency problem. You can overcome this in two ways:
Write the application in a way that ensures that all processes attach the region for receive before any processes write to the region.
At allocation time, specify that the region is coherent, by
specifying the
IMC_COHERENT
flag when you allocate the
region using the
imc_asalloc()
function.
This ensures
that all processes will see every update to the region, regardless of when
the processes attach the region.
Coherent regions use the loopback feature. This means that two MEMORY CHANNEL transactions occur for every write, one to write the data and one to loop the data back; because of this, coherent regions have less available bandwidth than noncoherent regions.
Processes that attach a region of MEMORY CHANNEL address space can only write to a transmit pointer, and can only read from a receive pointer. Any attempt to read a transmit pointer will result in a segmentation violation.
Apart from explicit read operations on MEMORY CHANNEL transmit pointers, segmentation violations will also result from operations that cause the compiler to generate read-modify-write cycles; for example:
Postincrement and postdecrement operations.
Preincrement and predecrement operations.
Assignment to simple data types that are not an integral multiple of four bytes.
Use of the
bcopy(3)
library function where the length
parameter is not an integral multiple of eight bytes, or where the source
or destination arguments are not eight-byte aligned.
Assignment to structures that are not quadword-aligned (that
is, the value returned by the
sizeof()
function is not
an integral multiple of eight).
This refers only to unit assignment of the
whole structure; for example,
mystruct1 = mystruct2.
Example 2-1 shows how to initialize, allocate, and attach to a region of MEMORY CHANNEL address space, and also shows two of the differences between MEMORY CHANNEL address space and traditional shared memory:
Initial coherency, as described in Section 2.2.2
Asymmetry of receive and transmit regions, as described in Section 2.2.3
The sample program shown in Example 2-1 executes in master or slave mode, as specified by a command-line parameter. In master mode, the program writes its own process identifier (PID) to a data structure in the global MEMORY CHANNEL address space. In slave mode, the program polls a data structure in the MEMORY CHANNEL address space to determine the PID of the master process.
Note
Programs should be flexible in their use of keys, to prevent problems resulting from key clashes. The use of meaningful, application-specific keys is recommended.
/* /usr/examples/cluster/mc_ex1.c */
#include <c_asm.h>
#include <sys/types.h>
#include <sys/imc.h>
#define VALID 756
main (int argc, char *argv[])
{
imc_asid_t glob_id;
typedef struct {
pid_t pid;
volatile int valid; [1]
} clust_pid;
clust_pid *global_record;
caddr_t add_rx_ptr = 0, add_tx_ptr = 0;
int status;
int master;
int logical_rail=0;
/* check for correct number of arguments /*
if (argc != 2) {
printf("usage: mcpid 0|1\n");
exit(-1);
}
/* test if process is master or slave */
master = atoi(argv[1]); [2]
/* initialize MEMORY CHANNEL API library */
status = imc_api_init(NULL); [3]
if (status < 0) {
imc_perror("imc_api_init::",status); [4]
exit(-2);
}
imc_asalloc(123, 8192, IMC_URW, 0, &glob_id,
logical_rail); [5]
if (master) {
imc_asattach(glob_id, IMC_TRANSMIT, IMC_SHARED,
0, &add_tx_ptr); [6]
global_record = (clust_pid*)add_tx_ptr; [7]
global_record->pid = getpid();
mb(); [8]
global_record->valid = VALID;
mb();
}
else { /* secondary process */
imc_asattach(glob_id, IMC_RECEIVE, IMC_SHARED,
0, &add_rx_ptr); [9]
(char*)global_record = add_rx_ptr;
while ( global_record->valid != VALID)
; /* continue polling */ [10]
printf("pid of master process is %d\n",
global_record->pid);
}
imc_asdetach(glob_id);
imc_asdealloc(glob_id); [11]
}
The
valid
flag is declared
as volatile to prevent the compiler from performing any optimizations that
might prevent the code from reading the updated PID value from memory.
[Return to example]
The first argument on the command line indicates whether the process is a master (argument equal to 1) or a slave process (argument not equal to 1). [Return to example]
The
imc_api_init()
function
initializes the MEMORY CHANNEL API library, and should be called before calling any of the other MEMORY CHANNEL API library
functions.
[Return to example]
All MEMORY CHANNEL API library functions return a zero (0) status if successful.
The
imc_perror()
function decodes error status values.
For brevity, this example ignores the
status from all functions other than the
imc_api_init()
function.
[Return to example]
The
imc_asalloc()
function allocates a region of MEMORY CHANNEL address space with the following
characteristics:
key=123--The value identifies
the region of MEMORY CHANNEL address space.
Other applications that attach this region
will use the same key value.
size=8192--The size of the region
is 8192 bytes.
perm=IMC_URW--The access permission
on the region is user read and write.
id=glob_id--The
imc_asalloc()
function returns this value, which uniquely identifies the allocated
region.
The program uses this value in subsequent calls to other MEMORY CHANNEL
functions.
logical_rail=0--The region is allocated
using MEMORY CHANNEL logical rail zero (0).
The master process attaches
the region for transmit by calling the
imc_asattach()
function
and specifying the
glob_id
identifier, which was returned
by the call to the
imc_asalloc()
function.
The
imc_asattach()
function returns
add_tx_ptr, a
pointer to the address of the region in the process virtual address space.
The
IMC_SHARED
value signifies that the region is shareable,
so other processes on this host can also attach the region.
[Return to example]
The program overlays the global region with
the global record structure and writes the process PID in the
pid
field of the global record.
Note that the master process has attached
the region for transmit; therefore, it can only write data in the field.
An
attempt to read the field will result in a segmentation violation; for example:
(pid_t)x = global_record->pid;
The program uses memory barrier instructions
to ensure that the
pid
field is forced out of the Alpha
CPU write buffer before the
VALID
flag is set.
[Return to example]
The slave process attaches the region for receive
by calling the
imc_asattach()
function and specifying the
glob_id
identifier, which was returned by the call to the
imc_asalloc()
function.
The
imc_asattach()
function
returns
add_rx_ptr, a pointer to the address of the region
in the process virtual address space.
On mapping, the contents of the region
may not be consistent on all processes that map the region.
Therefore, start
the slave process before the master to ensure that all writes by the master
process appear in the virtual address space of the slave process.
[Return to example]
The slave process overlays the region with
the global record structure and polls the valid flag.
The earlier declaration
of the flag as volatile ensures that the flag is immune to compiler optimizations,
which might result in the field being stored in a register.
This ensures that
the loop will load a new value from memory at each iteration and will eventually
detect the transition to
VALID.
[Return to example]
At termination, the master and
slave processes explicitly detach and deallocate the region by calling the
imc_asdetach()
function and the
imc_asdealloc()
function.
In the case of abnormal termination, the allocated regions are automatically
freed when the processes exit.
[Return to example]
As described in Section 2.2.2, the initial coherency problem can be overcome by retransmitting the data after all mappings of the same region for receive have been completed, or by specifying at allocation time that the region is coherent. However, when a process writes to a transmit pointer, several microseconds can elapse before the update is reflected in the physical memory that corresponds to the receive pointer. If the process reads the receive pointer during that interval, the data it reads might be incorrect. This is known as the latency related coherency problem.
Latency problems do not arise in conventional shared memory systems. Memory and cache control ensure that store and load instructions are synchronized with data transfers.
Example 2-2 shows two versions of a program that decrements a global process count and detects the count reaching zero (0). The first program uses System V shared memory and interprocess communication. The second uses the MEMORY CHANNEL API library.
/* /usr/examples/cluster/mc_ex2.c */
/****************************************
********* System V IPC example *******
****************************************/
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
main()
{
typedef struct {
int proc_count;
int remainder[2047]
} global_page;
global_page *mypage;
int shmid;
shmid = shmget(123, 8192, IPC_CREAT | SHM_R | SHM_W);
(caddr_t)mypage = shmat(shmid, 0, 0); /* attach the
global region */
mypage->proc_count ++; /* increment process
count */
/* body of program goes here */
.
.
.
/* clean up */
mypage->proc_count --; /* decrement process
count */
if (mypage->proc_count == 0 )
printf("The last process is exiting\n");
.
.
.
}
/****************************************
******* MEMORY CHANNEL example *******
****************************************/
#include <sys/types.h>
#include <sys/imc.h>
main()
{
typedef struct {
int proc_count;
int remainder[2047]
} global_page;
global_page *mypage_rx, *mypage_tx; [1]
imc_asid_t glob_id;
int logical_rail=0;
int temp;
imc_api_init(NULL);
imc_asalloc(123, 8192, IMC_URW | IMC_GRW, 0, &glob_id,
logical_rail); [2]
imc_asattach(glob_id, IMC_TRANSMIT, IMC_SHARED,
IMC_LOOPBACK, &(caddr_t)mypage_tx); [3]
imc_asattach(glob_id, IMC_RECEIVE, IMC_SHARED,
0, &(caddr_t)mypage_rx); [4]
/* increment process count */
mypage_tx->proc_count = mypage_rx->proc_count + 1; [5]
/* body of program goes here */
.
.
.
/* clean up */
/* decrement process count
temp = mypage_rx->proc_count - 1 [6]
mypage_tx->proc_count = temp;
/* wait for MEMORY CHANNEL update to occur */
while (mypage_rx->proc_count != temp)
;
if (mypage_rx->proc_count == 0 )
printf("The last process is exiting\n");
.
.
.
}
The process must be able to read the data that it writes to the MEMORY CHANNEL global address space. Therefore, it declares two addresses, one for transmit and one for receive. [Return to example]
The
imc_asalloc()
function allocates a region of MEMORY CHANNEL address space.
The characteristics
of the region are as follows:
key=123--This value identifies the
region of MEMORY CHANNEL address space.
Other applications that attach this region will
use the same key value.
size=8192--The size of the region
is 8192 bytes.
perm=IMC_URW | IMC_GRW--The region
is allocated with user and group read and write permission.
id=glob_id--The
imc_asalloc()
function returns this value, which uniquely identifies the allocated
region.
The program uses this value in subsequent calls to other MEMORY CHANNEL API library
functions.
logical_rail=0--The region is allocated
using MEMORY CHANNEL logical rail zero (0).
This call to the
imc_asattach()
function attaches the region for transmit at the address
pointed to by the
mypage_tx
variable.
The value of the
flag
parameter is set to
IMC_LOOPBACK, so that
any time the process writes data to the region, the data is looped back to
the receive memory.
[Return to example]
This call to the
imc_asattach()
function attaches the region for receive at the address pointed
to by the
mypage_rx
variable.
[Return to example]
The program increments the global process count by adding 1 to the value in the receive pointer, and by assigning the result into the transmit pointer. When the program writes to the transmit pointer, it does not wait to ensure that the write instruction completes. [Return to example]
After the body of the program completes,
the program decrements the process count and tests that the decremented value
was transmitted to the other hosts in the cluster.
To ensure that it examines
the decremented count (rather than some transient value), the program stores
the decremented count in a local variable,
temp.
It writes
the decremented count to the transmit region, and then waits for the value
in the receive region to match the value in
temp.
When
the match occurs, the program knows that the decremented process count has
been written to the MEMORY CHANNEL address space.
[Return to example]
In this example, the use of the local variable ensures that the program compares the value in the receive memory with the value that was transmitted. An attempt to use the value in the receive memory before ensuring that the value had been updated may result in erroneous data being read.
In a shared memory system, the process of reading and writing to memory is assumed to be error-free. In a MEMORY CHANNEL system, the error rate is of the order of three errors per year. This is much lower than the error rates of standard networks and I/O subsystems.
The MEMORY CHANNEL hardware reports detected errors to the MEMORY CHANNEL software. The MEMORY CHANNEL hardware provides two guarantees that make it possible to develop applications that can cope with errors:
It does not write corrupt data to host systems.
It delivers data to the host systems in the sequence in which the data is written to the MEMORY CHANNEL hardware.
These guarantees simplify the process of developing reliable and efficient messaging systems.
The MEMORY CHANNEL API library provides the following functions to help applications implement error management:
imc_ckerrcnt_mr()--The
imc_ckerrcnt_mr()
function checks for the existence of errors on
a specified logical rail on MEMORY CHANNEL hosts.
This allows transmitting processes
to check whether or not errors occur when they send messages.
imc_rderrcnt_mr()--The
imc_rderrcnt_mr()
function reads the clusterwide error count for
the specified logical rail and returns the value to the calling program.
This
allows receiving processes to check the error status of messages that they
receive.
The operating system maintains a count of the number of errors that occur on the cluster. The system increments the value whenever it detects a MEMORY CHANNEL hardware error in the cluster, and when a host joins or leaves the cluster.
The task of detecting and processing an error takes a small, but finite,
amount of time.
This means that the count returned by the
imc_rderrcnt_mr()
function might not be up to date with respect to an error that
has just occurred on another host in the cluster.
On the local host, the count
is always up to date.
Use the
imc_rderrcnt_mr()
function to implement a
simple and effective error-detection mechanism by reading the error count
before transmitting a message, and including the count in the message.
The
receiving process compares the error count in the message body with the local
value determined after the message arrives.
The local value is guaranteed
to be up to date, so if this value is the same as the transmitted value, then
it is certain that no intervening errors occurred.
Example 2-3
shows this technique.
/* /usr/examples/cluster/mc_ex3.c */
/*****************************************
********* Transmitting Process *********
******************************************/
#include <sys/imc.h>
#include <c_asm.h>
main()
{
typedef struct {
volatile int msg_arrived;
int send_count;
int remainder[2046];
} global_page;
global_page *mypage_rx, *mypage_tx;
imc_asid_t glob_id;
int i;
volatile int err_count;
imc_api_init(NULL);
imc_asalloc (1234, 8192, IMC_URW, 0, &glob_id,0);
imc_asattach (glob_id, IMC_TRANSMIT, IMC_SHARED, IMC_LOOPBACK,
&(caddr_t)mypage_tx);
imc_asattach (glob_id, IMC_RECEIVE, IMC_SHARED, 0,
&(caddr_t)mypage_rx);
/* save the error count */
while ( (err_count = imc_rderrcnt_mr(0) ) < 0 )
;
mypage_tx->send_count = err_count;
/* store message data */
for (i = 0; i < 2046; i++)
mypage_tx->remainder[i] = i;
/* now mark as valid */
mb();
do {
mypage_tx->msg_arrived = 1;
} while (mypage_rx->msg_arrived != 1); /* ensure no error on
valid flag */
}
/*****************************************
*********** Receiving Process **********
******************************************/
#include <sys/imc.h>
main()
{
typedef struct {
volatile int msg_arrived;
int send_count;
int remainder[2046];
} global_page;
global_page *mypage_rx, *mypage_tx;
imc_asid_t glob_id;
int i;
volatile int err_count;
imc_api_init(NULL);
imc_asalloc (1234, 8192, IMC_URW, 0, &glob_id,0);
imc_asattach (glob_id, IMC_RECEIVE, IMC_SHARED, 0,
&(caddr_t)mypage_rx);
/* wait for message arrival */
while ( mypage_rx->msg_arrived == 0 )
;
/* get this systems error count */
while ( (err_count = imc_rderrcnt_mr(0) ) < 0 )
;
if (err_count == mypage_rx->send_count) {
/* no error, process the body */
.....
}
else {
/* do error processing */
......
}
}
As shown in
Example 2-3,
the
imc_rderrcnt_mr()
function can be safely used to detect
errors at the receiving end of a message.
However, it cannot be guaranteed
to detect errors at the transmitting end.
This is because there is a small,
but finite, possibility that the transmitting process will read the error
count before the transmitting host has been notified of an error occurring
on the receiving host.
In
Example 2-3, the program must
rely on a higher-level protocol informing the transmitting host of the error.
The
imc_ckerrcnt_mr()
function provides guaranteed error detection for a specified logical
rail.
This function takes a user-supplied local error count and a logical
rail number as parameters, and returns an error in the following circumstances:
An outstanding error is detected on the specified logical rail
Error processing is in progress
The error count is higher than the supplied parameter
If the function returns successfully, no errors have been detected between
when the local error count was stored and the
imc_ckerrcnt_mr()
function was called.
The
imc_ckerrcnt_mr()
function reads the MEMORY CHANNEL adapter
hardware error status for the specified logical rail; this is a hardware operation
that takes several microseconds.
Therefore, the
imc_ckerrcnt_mr()
function takes longer to execute than the
imc_rderrcnt_mr()
function, which reads only a memory location.
Example 2-4 shows an amended version of the send sequence shown in Example 2-3. In Example 2-4, the transmitting process performs error detection.
/* /usr/examples/cluster/mc_ex4.c */
/**********************************************/
/* Transmitting Process With Error Detection */
/**********************************************/
#include <c_asm.h>
#define mb() asm("mb")
#include <sys/imc.h>
main()
{
typedef struct {
volatile int msg_arrived;
int send_count;
int remainder[2046];
} global_page;
global_page *mypage_rx, *mypage_tx;
imc_asid_t glob_id;
int i, status;
volatile int err_count;
imc_api_init(NULL);
imc_asalloc (1234, 8192, IMC_URW, 0, &glob_id,0);
imc_asattach (glob_id, IMC_TRANSMIT, IMC_SHARED, IMC_LOOPBACK,
&(caddr_t)mypage_tx);
imc_asattach (glob_id, IMC_RECEIVE, IMC_SHARED, 0,
&(caddr_t)mypage_rx);
/* save the error count */
while ( (err_count = imc_rderrcnt_mr(0) ) < 0 )
;
do {
mypage_tx->send_count = err_count;
/* store message data */
for (i = 0; i < 2046; i++)
mypage_tx->remainder[i] = i;
/* now mark as valid */
mb();
mypage_tx->msg_arrived = 1;
/* if error occurs, retransmit */
} while ( (status = imc_ckerrcnt_mr(&err_count,0)) != IMC_SUCCESS);
}
In a MEMORY CHANNEL system, the processes communicate by reading and writing regions of the MEMORY CHANNEL address space. The preceding sections contain sample programs that show arbitrary reading and writing of regions. In practice, however, a locking mechanism is sometimes needed to provide controlled access to regions and to other clusterwide resources. The MEMORY CHANNEL API library provides a set of lock functions that enable applications to implement access control on resources.
The MEMORY CHANNEL API library implements locks by using mapped pages of the global MEMORY CHANNEL
address space.
For efficiency reasons, locks are allocated in sets rather
than individually.
The
imc_lkalloc()
function allows you
to allocate a lock set.
For example, if you want to use 20 locks, it is more
efficient to create one set with 20 locks than five sets with four locks each,
and so on.
To facilitate the initial coordination of distributed applications,
the
imc_lkalloc()
function allows a process to atomically
(that is, in a single operation) allocate the lock set and acquire the first
lock in the set.
This feature allows the process to determine whether or not
it is the first process to allocate the lock set.
If it is, the process is
guaranteed access to the lock and can safely initialize the resource.
Instead of allocating the lock set and acquiring the first lock
atomically, a process could call the
imc_lkalloc()
function
and then the
imc_lkacquire()
function.
In that case, however,
there is a risk that another process might acquire the lock between the two
function calls, and the first process would not be guaranteed access to the
lock.
Example 2-5 shows a program in which the first process to lock a region of MEMORY CHANNEL address space initializes the region, and the processes that subsequently access the region simply update the process count.
/* /usr/examples/cluster/mc_ex5.c */
#include <sys/types.h>
#include <sys/imc.h>
main ( )
{
imc_asid_t glob_id;
imc_lkid_t lock_id;
int locks = 4;
int status;
typedef struct {
int proc_count;
int pattern[2047];
} clust_rec;
clust_rec *global_record_tx, *global_record_rx; [1]
caddr_t add_rx_ptr = 0, add_tx_ptr = 0;
int j;
status = imc_api_init(NULL);
imc_asalloc(123, 8192, IMC_URW, 0, &glob_id, 0);
imc_asattach(glob_id, IMC_TRANSMIT, IMC_SHARED,
IMC_LOOPBACK, &add_tx_ptr);
imc_asattach(glob_id, IMC_RECEIVE, IMC_SHARED,
0, &add_rx_ptr);
global_record_tx = (clust_rec*) add_tx_ptr; [2]
global_record_rx = (clust_rec*) add_rx_ptr;
status = imc_lkalloc(456, &locks, IMC_LKU, IMC_CREATOR,
&lock_id); [3]
if (status == IMC_SUCCESS)
{
/* This is the first process. Initialize the global region */
global_record_tx->proc_count = 0; [4]
for (j = 0; j < 2047; j++)
global_record_tx->pattern[j] = j;
/* release the lock */
imc_lkrelease(lock_id, 0); [5]
}
/* This is a secondary process */
else if (status == IMC_EXISTS)
{
imc_lkalloc(456, &locks, IMC_LKU, 0, &lock_id); [6]
imc_lkacquire(lock_id, 0, 0, IMC_LOCKWAIT); [7]
/* wait for access to region */
global_record_tx->proc_count = global_record_rx->proc_count+1; [8]
/* release the lock */
imc_lkrelease(lock_id, 0);
}
/* body of program goes here */
.
.
.
/* clean up */
imc_lkdealloc(lock_id); [9]
imc_asdetach(glob_id);
imc_asdealloc(glob_id);
}
The process, in order to read the data that it writes to the MEMORY CHANNEL global address space, maps the region for transmit and for receive. See Example 2-2 for a detailed description of this procedure. [Return to example]
The program overlays the transmit and receive pointers with the global record structure. [Return to example]
The process tries
to create a lock set that contains four locks and a
key
value of
456.
The call to the
imc_lkalloc()
function also specifies the
IMC_CREATOR
flag.
Therefore,
if the lock set is not already allocated, the function will automatically
acquire lock zero (0).
If the lock set
already exists, the
imc_lkalloc()
function fails to allocate
the lock set and returns the value
IMC_EXISTS.
[Return to example]
The process that creates the lock set (and consequently holds lock zero (0)) initializes the global region. [Return to example]
When the process finishes initializing
the region, it calls the
imc_lkrelease()
function to release
the lock.
[Return to example]
Secondary processes that execute after
the region has been initialized, having failed in the first call to the
imc_lkalloc()
function, now call the function again, without the
IMC_CREATOR
flag.
Because the value of the
key
parameter is the same (456), this call allocates the same
lock set.
[Return to example]
The secondary process calls the
imc_lkacquire()
function to acquire lock zero (0) from the lock set.
[Return to example]
The secondary process updates the process count and writes it to the transmit region. [Return to example]
At the end of the program, the processes release all MEMORY CHANNEL resources. [Return to example]
When a process acquires a lock, other processes executing on the cluster cannot acquire that lock.
Waiting for locks to become free entails busy spinning and has a significant effect on performance. Therefore, in the interest of overall system performance, applications should acquire locks only as they are needed and release them promptly.
The MEMORY CHANNEL API library provides the
imc_kill()
function to allow processes to send signals to specified
processes executing on a remote host in a cluster.
This function is similar
to the UNIX
kill(2)
function.
The main difference is that the
imc_kill()
function does not support the sending of signals to multiple
processes.
The following sections discuss how to use the MEMORY CHANNEL API functions to access cluster information, and how to access status information from the command line.
The MEMORY CHANNEL API library provides the
imc_getclusterinfo()
function,
which allows processes to get information about the hosts in a MEMORY CHANNEL API
cluster.
The function returns one or more of the following:
A count of the number of hosts in the cluster, and the name of each host.
The number of logical rails in the cluster.
The active MEMORY CHANNEL logical rails bitmask, with a bit set for each active logical rail.
The function does not return information about a host unless the MEMORY CHANNEL API library is initialized on the host.
The MEMORY CHANNEL API library provides the
imc_wait_cluster_event()
function to block a calling thread until a specified cluster event
occurs.
The following MEMORY CHANNEL API cluster events are valid:
A host joins or leaves the cluster.
The logical rail configuration of the cluster changes.
The
imc_wait_cluster_event()
function checks
the current representation of the MEMORY CHANNEL API cluster configuration item
being monitored and returns the new MEMORY CHANNEL API cluster configuration.
Example 2-6
shows how you can use the
imc_getclusterinfo()
function with the
imc_wait_cluster_event()
function to request the names of the members of the MEMORY CHANNEL API cluster
and the active MEMORY CHANNEL logical rails bitmask, and then wait for an event
change on either.
/* /usr/examples/cluster/mc_ex6.c */
#include <sys/imc.h>
main ( )
{
imc_railinfo mask;
imc_hostinfo hostinfo;
int status;
imc_infoType items[3];
imc_eventType events[3];
items[0] = IMC_GET_ACTIVERAILS;
items[1] = IMC_GET_HOSTS;
items[2] = 0;
events[0] = IMC_CC_EVENT_RAIL;
events[1] = IMC_CC_EVENT_HOST;
events[2] = 0;
imc_api_init(NULL);
status = imc_getclusterinfo(items,2,mask,sizeof(imc_railinfo),
&hostinfo,sizeof(imc_hostinfo));
if (status != IMC_SUCCESS)
imc_perror("imc_getclusterinfo:",status);
status = imc_wait_cluster_event(events, 2, 0,
mask, sizeof(imc_railinfo),
&hostinfo, sizeof(imc_hostinfo));
if ((status != IMC_HOST_CHANGE) && (status != IMC_RAIL_CHANGE))
imc_perror("imc_wait_cluster_event didn't complete:",status);
} /*main*/
The MEMORY CHANNEL API library
provides the
imcs
command to report on MEMORY CHANNEL status.
The
imcs
command writes information to the standard output
about currently active MEMORY CHANNEL facilities.
The output is displayed as a
list of regions or lock sets, and includes the following information:
The type of subsystem that created the region or lock set (possible values are IMC or PVM)
An identifier for the MEMORY CHANNEL region
An application-specific key that refers to the MEMORY CHANNEL region or lock set
The size, in bytes, of the region
The access mode of the region or lock set
The username of the owner of the region or lock set
The group of the owner of the region or lock set
The MEMORY CHANNEL logical rail used for the region
A flag specifying the coherency of the region
The number of locks available in the lock set
There are two models that you can use to develop applications based on the MEMORY CHANNEL API library:
Shared memory
Message passing
At first, the shared memory approach might seem more suited to the MEMORY CHANNEL features. However, developers who use this model must deal with the latency, coherency, and error-detection problems described in this chapter. In some cases, it might be more appropriate to develop a simple message-passing library that hides these problems from applications. The data transfer functions in such a library could be implemented completely in user space. Therefore, they would operate as efficiently as implementations based on the shared memory model.