Mac OS X Beowulf Cluster Deployment Notes

Introduction

Installing open source cluster software on Mac OS X systems is no more difficult than on other UNIX systems.  The Mac OS X system includes the following tools and system layers which make building Beowulf clusters a standard routine:

• Mach 3.0 kernel
• BSD 4.4
• FreeBSD 3.2 networking
• (Just enough) POSIX compliance
• GCC 3.3 compilers
• Java 2 SE v1.3.1

What follows are our deployment notes for building a Beowulf cluster on a Mac OS X system.  The overall setup assumes:

1. Mac OS X Server is used to handle user accounts.
2. File services are done on a dual-homed machine (can be a separate Mac OS X client or could be the Mac OS X Server).
3. Our naming convention for our cluster nodes is node-01, node-02, node-03, etc.
4. Our cluster nodes are on a private network with addresses on subnet 10.0.0.
5. The dual-homed machine is the gateway for our private and public networks.

These assumptions are a consequence of the decisions we made building our first cluster.  Techniques other than those outlined below can be used.  For example, ssh can be used instead of rsh as a means of having secure inter-nodal communication.  We solved our security problem by creating a private network for the cluster instead of using ssh.  Others may not want to do it this way.  In the future other techniques will be documented here as well.

Deployment

Compiling LAM in Mac OS X 10.1.x

LAM/MPI compiles fairly smoothly on Mac OS X 10.1.x.  There are only a few modifications necessary to the source:

• In tools/tkill.c, Line 367 - Change remove  to  rmdir
  
• In share/etc/registry.c - Change  #include <sys/sem.h>  to   #include <sys/semaphore.h> (This step is only needed when compiling on 10.1.*)

The configure flags we used are:

• --with-romio-flags="-cflags=-DNO_AIO"
  
• --prefix=/usr/local
  
• --with-trillium (if you want to use XMPI)
  
• --without-fc (optional if you don't have fortran available)

Compiling and installing was straight forward using the commands  make and  make install respectively.

Compiling PVM in Mac OS X 10.1.x

Compiling PVM is pretty straightforward, just a few modifications:

• In tracer/tracer.h, Line 44 - Change #include <malloc.h>  to   #include <sys/malloc.h>
  
• In tracer/trclib.h, Line 38 - Change #include <malloc.h>  to   #include <sys/malloc.h>
  
• In tracer/trcsort.c, Line 35 - Change #include <malloc.h>  to   #include <sys/malloc.h>

Setting up Password-less Login

This does not apply if you're using SSH.  Because our cluster is running on a private subnet we use rsh to establish a system wide password-less login.  Setting it up you need to:

• Uncomment in /etc/inetd.conf the following lines:
  
  login stream tcp nowait root /usr/libexec/tcpd rlogind
shell stream tcp nowait root /usr/libexec/tcpd rshd
  


• Then create a /etc/hosts.equiv file with an entry for each node in your cluster.  The file's entries are:

node-01
node-02
  etc...

Setting Up DNS

If you do not have DNS entries for each machine on your cluster you will need to setup some way of reverse lookup, Mac OS X Server needs to be able to resolve client ip addresses.  There are various way of doing this, through experience we have found that the best way to do this (without having to setup a DNS server) is to change nslookupd's lookup order to use the entries in /etc/hosts (normally /etc/hosts is only used while starting up in single user mode).

To change the default behavior of lookupd you need to:

• Create the directory /etc/lookupd and put a hosts file in this directory with the following lines:

LookupOrder = Cache FF DNS NI NIL
ValidateCache = NO



• Create /etc/hosts file.  The file's entries are:

10.0.0.1 node-01
10.0.0.2 node-02
10.0.0.3 node-03
etc...

NFS Mounting

The settings for NFS mounts can be maintained in the /etc/mounts file or in the NetInfo database, Mac OS X's default location.  We use the NetInfo database.  There are several ways of editing the NetInfo database.  In any case it is prudent to backup your NetInfo data before making changes to it, especially when you are new to the procedure.  The database files themselves are kept in /var/db/netinfo and can be backed up simply by copying the directory.

Tools for editing NetInfo are:

• /Applications/Utilities/NetInfo Manager

• /usr/bin/niutil

• /usr/bin/niload

• /usr/bin/nicl

• /Applications/Utilities/NFSManager (available from Bresink Software)

We used a combination of NFSManager and NetInfo Manager, in other words, the easier interactive GUI tools.  The other tools, with the exception of nicl, are non-interactive and better suited for batch processing situations.  nicl is new and we haven't had a chance to work with it.

We used NFSManager to inspect the settings and used NetInfo Manager to enter them.  NFSManager could be used to write to NetInfo though.

We have both the home directories and the binaries mounted across all nodes through NFS.  Homes are on node-00:/Volumes/craid/Home.  With NetInfo create the following entries in the  mounts directory of the root domain (/):

Commit the settings when prompted by NetInfo.  A check using "nidump -r /mounts /" should produce the following output:

#nidump -r /mounts /
{
   "name" = ( "mounts" );
   CHILDREN = (
   {
      "name" = ( "node-00:/Volumes/craid/Home" );
      "vfstype" = ( "nfs" );
      "dir" = ( "/Network/Servers/" );
      "opts" = ( "net" );
   }
   )
}

The export settings are created in the exports directory of the local domain of the server (.).

Commit the settings when prompted by NetInfo.  A check using "nidump -r /exports ." should produce the following output:

#nidump -r /exports .
{
   "name" = ( "exports" );
   CHILDREN = (
   {
      "clients" = ( )
      "name" = ( "/Volumes/craid/Home" );
      "opts" = ( "maproot=root", "network=10.0.0.0", "mask=255.255.255.0" );
   }
}

Statically Mounting the Binaries

Note: It's not necessary to cross mount your binaries, you can of course have all the binaries reside on the local hard drive of each node and use rsync or scp to keep them all up to date.  Cross mounting them is less of an administrative hassle but you sacrifice some performance.

On the server the export settings are created in the exports directory of the local domain (.):

#nidump -r /exports .
{
   "name" = ( "exports" );
   CHILDREN = (
   {
      "clients" = ( );
      "name" = ( "/usr/local/bin" );
      "opts" = ( "maproot=root", "network=10.0.0.0", "mask=255.255.255.0" );
   }
   )
}

On each node the settings are created in the mounts directory of the local domain (.):

#nidump -r /mounts .
{
   "name" = ( "mounts" );
   CHILDREN = (
   {
      "vfstype" = ( "nfs" );
      "dir" = ( "/usr/local/bin" );
      "name" = ( "node-00:/usr/local/bin" );
      "opts" = ( "-b", "-c", "-r=8192", "-s", "-P", "-w=8192", "ro" );
   }
   )
}

Maui Scheduler Molokini Edition

Software

Installation

• --prefix=/usr/local/bin/maui
• --sysconfdir=/etc

• /usr/local/bin/maui/sbin/maui_genkey
• /usr/local/bin/maui/sbin/maui_create_db
• /usr/local/bin/maui/sbin/maui_grant_db

Configuration

• maui.host (the machine the scheduler runs on)
• wiki.udp.multicast (set this to false)
• wiki.slots (number of cpu's per machine)
• mysql.host (the db host, in case this is not also the scheduling host make sure maui.host has permission to access the maui_db on the db hosts)
• logger.out (defaults to /var/log/maui/maui.out, either change this to /var/log/maui.out or create a maui directory under /var/log)

Some Issues with NFS on Mac OS X (at least on 10.1.5)

• Cannot mount an exported UFS partition onto HFS (does work the other way around)
  
• Cannot succesfully mount exported symlinks
  
• If you try to setup NFS mounted home directories through Server Admin you'll notice that somehow the NFS checkbox (under sharing) doesn't remain checked, it always reverts back to AFP. (This is a bug which has been addressed since 10.1.5.)