Grid Gurus

Life with a Repoman

I have recently begun learning to use kusu for cluster management. Kusu maintains what is called a repository for managing installation images, software packages, as well as other assorted information for distribution to cluster nodes. A repository is specific to a single operating system.

While kusu allows you to distribute packages at the RPM level, the best way to deliver a software system from the repository to the nodes is by using a “kit”. A kit consists of a component RPM, the software RPMs, and the kit RPM itself.

The steps for adding a kit to a repository are:

1. Build the repository and/or kit RPMs from source;
2. install the kit RPM. It will be called something like "my-kit-<name>", where name is the name of the kit (e.g. hpc) I use the yum command to install the kit. The kit will install all of the files necessary for the software system into a single directory path. Depending on the authors of the kit, it may come with an installation script called "install-kit-<name>" in the /opt/kusu/sbin directory;
3. If the kit came with the script, you can simply run:
       $ install-kit-<name> -r <repository name>
   where the repository name is specific to the operating system.  Without this type of script, you will need to execute the following steps by hand:
1. Create a new kit:
       $ kitops -add -m <path to kit files>;
2. Add the kit to the repository:
       $ repoman -a -k <kit name> -r <repository name>
   (I believe that the repository manager command, repoman, is a reference the movie of the same name);
3. Update the repository:
       $ repoman -u -r <repository name>
   A repository update deletes the existing files and creates new ones from scratch;
4. Synchronize the files from the installer node to the cluster node:
       $ cfmsync -f -p
4. Choose the node group(s) to which to add the package using the ngedit command.

Unfortunately, updating the repository requires a lot of patience.  The ten to twenty minutes I spend waiting each time I run an update is very frustrating. While much of this wait might be caused from using virtual machines for a test cluster, I can think of a number of other loads that could have equivalent, if not worse, impacts.

This situation is particularly bad when I am working on the creation of a new kit. Three or so small changes can easily take an hour.  Thus my productivity during those times approaches zero. If kusu is to be widely accepted by the grid community, the repository update process will need to be streamlined.

Automating UniCluster Express Compute Node Installation

Although installing UniCluster Express (UCE) is in principle a fairly straightforward task, the thought of running UCE installer script manually on a large number of compute nodes probably does not sound very exciting to most people. Recently I found myself in that situation, and decided that spending some time automating the compute node installation procedure would be very wisely spent. Fortunately, this was not a very difficult task, and I got things working in a couple of hours using expect to drive the installation process. My installation script is shown below. It assumes that configuring advanced UCE options is not needed, and that root account has passwordless access enabled to compute nodes. After customizing few variables at the top of the script (and possibly the script login part), its usage should be fairly simple, as it requires only the compute node name to be provided as argument.

$ cat install_uce_compute.exp 
#!/usr/bin/env expect

# Usage:
#   $0 <node name>
# Install script location.
set node [lindex $argv 0]
set root_password ""
set root_prompt "]# "

set install_script "/root/unicluster/install_unicluster.sh"
set owner "ucluster"
set install_path "/usr/local/unicluster"
set bootstrap_server "hp2node2"
set bootstrap_client "ucluster"
set bootstrap_password "123456"

# Check arguments.
if {$argc < 1} {
  send_user "Node name not provided.\n"
  exit
}

# Login.
spawn ssh root@$node 
#expect password:
#send "$root_password\r"

expect "$root_prompt"
send "$install_script\r"

# License agreement.
expect "yes|no]: "
send "yes\r"

# Node type.
expect "workstation]:"
send "compute\r"

# Owner.
expect "ucluster]: "
send "$owner\r"

# Installation path.
expect "/usr/local/unicluster]: "
send "$install_path\r"

# Bootstrap server.
expect "bootstrap server: "
send "$bootstrap_server\r"

# Bootstrap account.
expect "ucluster]: "
send "$bootstrap_client\r"

# Bootstrap password.
expect "password: "
send "$bootstrap_password\r"
expect "verification: "
send "$bootstrap_password\r"

# Advanced options.
expect "y|N]:"
send "\r"
expect "to continue."
send "\r"

# Timeout.
set timeout 600

# Done.
expect "Installation completed successfully"
exit

Once your master node is up and running and you have expect rpm installed on your machine, installing compute nodes using the above script should be fairly straightforward, and involve simple shell command similar to the following:

$ for node in hp2node2 hp2node3 ; do ./install_uce_compute.exp $node; done

Just to be complete, the corresponding uninstall script (which should work for any UCE node type) is given below:

$ cat uninstall_uce.exp
#!/usr/bin/env expect

# Usage:
#   $0 <node name>

# Install script location.
set node [lindex $argv 0]
set root_password ""
set root_prompt "]# "

set uninstall_script "/root/unicluster/uninstall_unicluster.sh"
set install_path "/usr/local/unicluster"

# Check arguments.
if {$argc < 1} {
  send_user "Node name not provided.\n"
  exit
}

# Login.
spawn ssh root@$node 
#expect password:
#send "$root_password\r"

expect "$root_prompt"
send "$uninstall_script\r"

# Install path.
expect "path: "
send "$install_path\r"

# Backup.
expect "path: "
send "\r"

# Timeout.
set timeout 600

# Done.
expect "Uninstallation completed successfully"
exit

Hopefully, if you are reading this blog and considering installing UCE on your cluster, these scripts will save you some time and typing.