Building a Beowulf Cluster/Cloning of Slaves

When master and one slave (the golden slave) are installed and configured and now we want to scale up this configuration to more slaves by replicating the exact configuration of the golden slave.

Installing and configuring the OS on each machine manually is cumbersome and prone to error. However, nodes are identical, so why not just copy everything we need? This process is called cloning. We first setup a so-called golden node or model node and then transfer the system to other slave machines. Each new node will come with one new entry in the head node's DHCP server file (/etc/dhcpd.conf) and /etc/hosts file.

For preparation, make sure that in /etc/fstab and in the /boot/grub/menu.lst, there are no physical addresses of hardware (e.g. a hard disk), as they will be different among the nodes. All hardware should be addressed by their subdirectory in /dev which you can see in the output when you type mount.

I used low level R/W with dd and piping to and from netcat, respectively on machine to clone from and machine to clone to, as described in a howto. We clone using convert and copy (dd) and netcat (nc).

On the golden slave (or an identical clone) you run:

node1# dd if=/dev/hda conv=sync,noerror bs=64k | nc -l 5000

On the completely blank soon-to-be-slave you run:

node2# nc 192.168.1.1 5000 | dd of=/dev/hda bs=64k

where 192.168.1.1 is the ip of the golden slave. This presupposes the disk of soon-to-be slave is at least as big as the disk in the golden slave.

This took several hours time (it said 158GB read/written), but it worked.