Mac OS X Beowulf Cluster Deployment Notes

Introduction

Installing open source cluster software on Mac OS X systems is no more difficult than on other UNIX systems.  The Mac OS X system includes the following tools and system layers which make building Beowulf clusters a standard routine:

What follows are our deployment notes for building a Beowulf cluster on a Mac OS X system.  The overall setup assumes:

  1. Mac OS X Server is used to handle user accounts.
  2. File services are done on a dual-homed machine (can be a separate Mac OS X client or could be the Mac OS X Server).
  3. Our naming convention for our cluster nodes is node-01, node-02, node-03, etc.
  4. Our cluster nodes are on a private network with addresses on subnet 10.0.0.
  5. The dual-homed machine is the gateway for our private and public networks.

These assumptions are a consequence of the decisions we made building our first cluster.  Techniques other than those outlined below can be used.  For example, ssh can be used instead of rsh as a means of having secure inter-nodal communication.  We solved our security problem by creating a private network for the cluster instead of using ssh.  Others may not want to do it this way.  In the future other techniques will be documented here as well.

Deployment

Compiling LAM in Mac OS X 10.1.x

LAM/MPI compiles fairly smoothly on Mac OS X 10.1.x.  There are only a few modifications necessary to the source:

The configure flags we used are:

Compiling and installing was straight forward using the commands  make and  make install respectively.

Compiling PVM in Mac OS X 10.1.x

Compiling PVM is pretty straightforward, just a few modifications:

Set the PVM_ROOT environment variable then just run make and make install

Setting up Password-less Login

This does not apply if you're using SSH.  Because our cluster is running on a private subnet we use rsh to establish a system wide password-less login.  Setting it up you need to:

Setting Up DNS

If you do not have DNS entries for each machine on your cluster you will need to setup some way of reverse lookup, Mac OS X Server needs to be able to resolve client ip addresses.  There are various way of doing this, through experience we have found that the best way to do this (without having to setup a DNS server) is to change nslookupd's lookup order to use the entries in /etc/hosts (normally /etc/hosts is only used while starting up in single user mode).

To change the default behavior of lookupd you need to:

NFS Mounting

The settings for NFS mounts can be maintained in the /etc/mounts file or in the NetInfo database, Mac OS X's default location.  We use the NetInfo database.  There are several ways of editing the NetInfo database.  In any case it is prudent to backup your NetInfo data before making changes to it, especially when you are new to the procedure.  The database files themselves are kept in /var/db/netinfo and can be backed up simply by copying the directory.

Tools for editing NetInfo are:

We used a combination of NFSManager and NetInfo Manager, in other words, the easier interactive GUI tools.  The other tools, with the exception of nicl, are non-interactive and better suited for batch processing situations.  nicl is new and we haven't had a chance to work with it.

We used NFSManager to inspect the settings and used NetInfo Manager to enter them.  NFSManager could be used to write to NetInfo though.

We have both the home directories and the binaries mounted across all nodes through NFS.  Homes are on node-00:/Volumes/craid/Home.  With NetInfo create the following entries in the  mounts directory of the root domain (/):

Key Value
name node-00:/Volumes/craid/Home
vfstype nfs
dir /Network/Servers/
opts net

Commit the settings when prompted by NetInfo.  A check using "nidump -r /mounts /" should produce the following output:

#nidump -r /mounts /
{
   "name" = ( "mounts" );
   CHILDREN = (
   {
      "name" = ( "node-00:/Volumes/craid/Home" );
      "vfstype" = ( "nfs" );
      "dir" = ( "/Network/Servers/" );
      "opts" = ( "net" );
   }
   )
}

The export settings are created in the exports directory of the local domain of the server (.).

Key Value
clients  
name /Volumes/craid/Home
opts maproot=root
  network=10.0.0.0
  mask=255.255.255.0

Commit the settings when prompted by NetInfo.  A check using "nidump -r /exports ." should produce the following output:

#nidump -r /exports .
{
   "name" = ( "exports" );
   CHILDREN = (
   {
      "clients" = ( )
      "name" = ( "/Volumes/craid/Home" );
      "opts" = ( "maproot=root", "network=10.0.0.0", "mask=255.255.255.0" );
   }
}

Statically Mounting the Binaries

Note: It's not necessary to cross mount your binaries, you can of course have all the binaries reside on the local hard drive of each node and use rsync or scp to keep them all up to date.  Cross mounting them is less of an administrative hassle but you sacrifice some performance.

On the server the export settings are created in the exports directory of the local domain (.):

Key Value
clients  
name /usr/local/bin
opts maproot=root
  network=10.0.0.0
  mask=255.255.255.0

#nidump -r /exports .
{
   "name" = ( "exports" );
   CHILDREN = (
   {
      "clients" = ( );
      "name" = ( "/usr/local/bin" );
      "opts" = ( "maproot=root", "network=10.0.0.0", "mask=255.255.255.0" );
   }
   )
}

On each node the settings are created in the mounts directory of the local domain (.):

Key Value
vfstype nfs
dir /usr/local/bin
name node-00:/usr/local/bin
opts -b
  -c
  -r=8192
  -s
  -P
  -w=8192
  ro

#nidump -r /mounts .
{
   "name" = ( "mounts" );
   CHILDREN = (
   {
      "vfstype" = ( "nfs" );
      "dir" = ( "/usr/local/bin" );
      "name" = ( "node-00:/usr/local/bin" );
      "opts" = ( "-b", "-c", "-r=8192", "-s", "-P", "-w=8192", "ro" );
   }
   )
}

Maui Scheduler Molokini Edition

Having researched the various schedulers currently available for cluster computing (PBS, SGE etc) we came to the decision to use Maui ME as the scheduler for our G4 cluster. It has most of the functionality we were looking for and installing it on OS X was really quite straightforward.

Software

To succesfully compile the Maui scheduler you need to download and install javacc
The maui scheduler has a MySQL backend, you can get a binary installer for MySQL here
Download Maui from mauischeduler.sourceforge.net

Installation

*Note: For Maui 1.3.1 and 1.4.1 to compile on 10.2 (Jaguar) you need Java 1.4.1

Make sure the user mauid exists before you install Maui

The configure flags we used: Run make and make install and execute the following commands (as root): Make sure you have MySQL up and running before you do this and enter the list of GRANT statements that follow after running maui_grant_db into MySQL.
To make the binaries available to all nodes you can either nfs mount the maui directory or copy them over to all nodes. Also copy the /etc/maui.key file to all nodes.

Configuration

All configuration is done in the /usr/local/bin/maui/share/maui.properties file.
Some of the important properties are: Execute /usr/local/bin/maui/bin/mauictl start on the scheduling node.
Execute /usr/local/bin/maui/bin/nodectl start on the computation nodes.
Make sure you start the daemons at either root or mauid DO NOT use sudo!

Copy these startup items to /Library/StartupItems to automatically start the daemons:
Node startup item
Scheduler startup item

Some Issues with NFS on Mac OS X (at least on 10.1.5)


UCLA Department of Statistics
Last updated: 14-Nov-2002
Access count is: 140193, since 10-Nov-2002
Maintained by: Web Staff [webstaff@stat.ucla.edu]