In this post, we will build a 20 node Beowulf cluster on Amazon EC2 and run some computations using both MPI and its Python wrapper pyMPI. This tutorial will only describe how to get the cluster running and show a few example computations. I'll save detailed benchmarking for a later write-up.
One way to build an MPI cluster on EC2 would be to customize something like Warewulf or rebundle one of the leading linux cluster distributions like Parallel Knoppix or the Rocks Cluster Distribution onto an Amazon AMI. Both of these distros have kernels which should work with EC2. To get things running quickly as a proof of concept, I implemented a "roll-your-own" style cluster based on a Fedora Core 6 AMI managed with some simple Python scripts. I've found this approach suitable for running occasional parallel computations on EC2 with 20 nodes and have been running a cluster off and on for several months without any major issues. If you need to run a much larger cluster or require more complex user management, I'd recommend modifying one of the standard distributions. This will save you from some maintenance headaches and give you the additional benefit of the user/developer base for those systems.
The main task I use the cluster for is distributing large matrix computations, which is a problem well suited to existing libraries based on MPI. Depending on your needs, another platform such as Hadoop, Rinda, or cow.py might make more sense. I use Hadoop for some other projects, including MapReduce style tasks with Jython, and highly recommend it. That said, lets start building the MPI cluster...
The only prerequisite we assume is that the tutorial on Amazon EC2 has been completed and all needed web service accounts, authorizations, and keypairs have been created.
The command blocks which begin with peter-skomorochs-computer:~ pskomoroch$ are run on my local laptop, the commands preceded by -bash-3.1# or [lamuser@domu-12-31-33-00-03-46 ~]$ are run on EC2.
Its looking like this will be a long tutorial, so I'll break it into three parts...
Update: March 5, 2007 - I'm in the process of publishing a public AMI, and have changed a few things in the tutorial. The steps describing copying over rsa keys have been moved from this post to part 2 of the tutorial. People interested in testing an MPI cluster on EC2 can skip all the installs and just use my example AMI with your own keys as described in part 2
Tutorial Contents:
Part 1 of 3
- Fire Up a Base Image
- Rebundle a Larger Base Image
- Uploading the AMI to Amazon S3
- Registering the Larger Base Image
- Modifying the Larger Image
- Rebundle the compute node image
- Upload node AMI to Amazon S3
- Register Compute Node Image
Part 2 of 3
- Launching the EC2 nodes
- Cluster Configuration and Booting MPI
- Testing the MPI Cluster
- Changing the Cluster Size
- Cluster Shutdown
Part 3 of 3
- Basic MPI Cluster Administration on EC2 with Python
- Example application: Parallel Distributed Matrix Multiplication with PyMPI and Numpy
- Benchmarking EC2 for MPI
Fire Up a Base Image
We will build our cluster on top of the Fedora Core 6 base image published by "marcin the cool". Navigate to your local bin directory holding the Amazon EC2 developer tools and fire up the public image
peter-skomorochs-computer:~ pskomoroch$ ec2-run-instances ami-78b15411 -k gsg-keypair
RESERVATION r-e264818b 027811143419 default
INSTANCE i-2b1efa42 ami-78b15411 pending gsg-keypair 0
To check on the status of the instance run the following:
peter-skomorochs-computer:~ pskomoroch$ ec2-describe-instances i-2b1efa42
RESERVATION r-e264818b 027811143419 default
INSTANCE i-2b1efa42 ami-78b15411 domU-12-31-33-00-03-46.usma1.compute.amazonaws.com running gsg-keypair 0
The status has changed from "pending" to "running", so we are ready to ssh into the instance as root:
peter-skomorochs-computer:~ pskomoroch$ ssh -i id_rsa-gsg-keypair root@domU-12-31-33-00-03-46.usma1.compute.amazonaws.com
The authenticity of host 'domu-12-31-33-00-03-46.usma1.compute.amazonaws.com (216.182.230.204)' can't be established.
RSA key fingerprint is ZZZZZZ
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'domu-12-31-33-00-03-46.usma1.compute.amazonaws.com,216.182.230.204' (RSA) to the list of known hosts
-bash-3.1#
Here are some basic stats on the EC2 machine:
$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 37
model name : AMD Opteron(tm) Processor 250
stepping : 1
cpu MHz : 2405.452
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm ts fid vid ttp
bogomips : 627.50
The first change we make will be to modify the ssh properties to avoid timeouts:
Edit /etc/ssh/sshd_config and add the following line:
ClientAliveInterval 120
This image boots up fast, but it is missing a lot of basics along with the MPI libraries and Amazon AMI packaging tools. The main partition is fairly small, so before we start our installs, we will need to rebundle a larger version.
In order to rebundle, we need the Amazon developer tools installed...
Amazon AWS AMI tools install
Install the Amazon AWS ami tools from the rpm:
yum -y install wget nano tar bzip2 unzip zip fileutils
yum -y install ruby
yum -y install rsync make
cd /usr/local/src
wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm
rpm -i ec2-ami-tools.noarch.rpm
Rebundle a Larger Base Image
Copy over the pk/cert files:
peter-skomorochs-computer:~ pskomoroch$ scp -i id_rsa-gsg-keypair ~/.ec2/pk-FOOXYZ.pem ~/.ec2/cert-BARXYZ.pem root@domU-12-31-33-00-03-46.usma1.compute.amazonaws.com:/mnt/
pk-FOOXYZ.pem 100% 721 0.7KB/s 00:00
cert-BARXYZ.pem 100% 689 0.7KB/s 00:00
peter-skomorochs-computer:~ pskomoroch$
Using the -s parameter we boost the trimmed down fedora core 6 image from 1.5 GB to 5.5 GB so we have room to install more packages (substitute own your cert and user option values from the Amazon tutorial).
-bash-3.1# ec2-bundle-vol -d /mnt -k /mnt/pk-FOOXYZ.pem -c /mnt/cert-BARXYZ.pem -u 99999ABC -s 5536
Copying / into the image file /mnt/image...
Excluding:
/sys
/proc
/proc/sys/fs/binfmt_misc
/dev
/media
/mnt
/proc
/sys
/mnt/image
/mnt/img-mnt
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.015051 seconds, 69.7 MB/s
mke2fs 1.39 (29-May-2006)
warning: 256 blocks unused.
Bundling image file...
Splitting /mnt/image.tar.gz.enc...
Created image.part.00
Created image.part.01
Created image.part.02
Created image.part.03
Created image.part.04
Created image.part.05
Created image.part.06
Created image.part.07
Created image.part.08
Created image.part.09
Created image.part.10
Created image.part.11
Created image.part.12
Created image.part.13
Created image.part.14
...
Created image.part.39
Created image.part.40
Created image.part.41
Generating digests for each part...
Digests generated.
Creating bundle manifest...
ec2-bundle-vol complete.
Uploading the AMI to Amazon S3
This step is identical to the Amazon tutorial, use you own Amazon assigned AWS Access Key ID (aws-access-key-id) and AWS Secret Access Key (aws-secret-access-key). I'll use the following values in the code examples:
- Access Key ID: 1AFOOBARTEST
- Secret Access Key: F0Bar/T3stId
bash-3.1# ec2-upload-bundle -b FC6_large_base_image -m /mnt/image.manifest.xml -a 1AFOOBARTEST -s F0Bar/T3stId
Setting bucket ACL to allow EC2 read access ...
Uploading bundled AMI parts to https://s3.amazonaws.com:443/FC6_large_base_image ...
Uploaded image.part.00 to https://s3.amazonaws.com:443/FC6_large_base_image/image.part.00.
Uploaded image.part.01 to https://s3.amazonaws.com:443/FC6_large_base_image/image.part.01.
...
Uploaded image.part.48 to https://s3.amazonaws.com:443/FC6_large_base_image/image.part.48.
Uploaded image.part.49 to https://s3.amazonaws.com:443/FC6_large_base_image/image.part.49.
Uploading manifest ...
Uploaded manifest to https://s3.amazonaws.com:443/FC6_large_base_image/image.manifest.xml.
ec2-upload-bundle complete
The upload will take several minutes...
Registering the Larger Base Image
To register the new image with Amazon EC2, we switch back to our local machine and run the following:
peter-skomorochs-computer:~/src/amazon_ec2 pskomoroch$ ec2-register FC6_large_base_image/image.manifest.xml
IMAGE ami-3cb85d55
Included in the output is an AMI identifier, (ami-3cb85d55 in the example above) which we will use as our base for building the compute nodes.
Modifying the Larger Image
We need to start an instance of the larger image we registered and install some needed libraries.
First, start the new image:
peter-skomorochs-computer:~ pskomoroch$ ec2-run-instances ami-3cb85d55 -k gsg-keypair
RESERVATION r-e264818b 027811143419 default
INSTANCE i-2z1efa32 ami-3cb85d55 pending gsg-keypair 0
Wait for a hostname so we can ssh into the instance...
peter-skomorochs-computer:~ pskomoroch$ ec2-describe-instances i-2b1efa42
RESERVATION r-e264818b 027811143419 default
INSTANCE i-2z1efa32 ami-3cb85d55 domU-12-31-33-00-03-57.usma1.compute.amazonaws.com running gsg-keypair 0
ssh in as root:
peter-skomorochs-computer:~ pskomoroch$ ssh -i id_rsa-gsg-keypair root@domU-12-31-33-00-03-57.usma1.compute.amazonaws.com
The authenticity of host 'domu-12-31-33-00-03-57.usma1.compute.amazonaws.com (216.182.238.167)' can't be established.
RSA key fingerprint is 23:XY:FO...
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'domu-12-31-33-00-03-57.usma1.compute.amazonaws.com,216.182.238.167' (RSA) to the list of known hosts.
-bash-3.1#
Yum Installs
Run the following yum installs to get some needed libraries:
yum -y install python-devel
yum -y install gcc
yum -y install gcc-c++
yum -y install subversion gcc-gfortran
yum -y install fftw-devel swig
yum -y install compat-gcc-34 compat-gcc-34-g77 compat-gcc-34-c++ compat-libstdc++-33 compat-db compat-readline43
yum -y install hdf5-devel
yum -y install readline-devel
yum -y install python-numeric python-numarray Pyrex
yum -y install python-psyco
yum -y install wxPython-devel zlib-devel freetype-devel tk-devel tkinter gtk2-devel pygtk2-devel libpng-devel
yum -y install octave
ACML Install
For improved performance in matrix operations, we will want to install processor specific math libraries. Since the Amazon machines run on AMD Opteron processors, we will install ACML instead of Intel MKL.
- Login into the AMD developer page
- Download acml-3-6-0-gnu-32bit.tgz , and scp the archive over to the EC2 instance.
peter-skomorochs-computer:~ pskomoroch$ scp acml-3-6-0-gnu-32bit.tgz root@domU-12-31-33-00-03-57.usma1.compute.amazonaws.com:/usr/local/src/
acml-3-6-0-gnu-32bit.tgz 100% 9648KB 88.5KB/s 01:49
- To install acml, decompress the files and run the install scripts and accept the license. Note where it installs acml (in my case /opt/acml3.6.0/)
- cd into the /opt/acml3.6.0/ directory and run the tests by issuing make.
-bash-3.1# chmod +x /usr/lib/gcc/i386-redhat-linux/3.4.6/libg2c.a
-bash-3.1# ln -s /usr/lib/gcc/i386-redhat-linux/3.4.6/libg2c.a /usr/lib/libg2c.a
-bash-3.1# cd /usr/local/src/
-bash-3.1# ls
acml-3-6-0-gnu-32bit.tgz ec2-ami-tools.noarch.rpm
-bash-3.1# tar -xzvf acml-3-6-0-gnu-32bit.tgz
contents-acml-3-6-0-gnu-32bit.tgz
install-acml-3-6-0-gnu-32bit.sh
README.32-bit
ACML-EULA.txt
-bash-3.1# bash install-acml-3-6-0-gnu-32bit.sh
Add the libraries to the default path by adding the following to /etc/profile:
LD_LIBRARY_PATH=/opt/acml3.6.0/gnu32/lib
export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE INPUTRC LD_LIBRARY_PATH
Example of running the ACML tests:
-bash-3.1# cd /opt/acml3.6.0/gnu32/examples/
-bash-3.1# make
Compiling program cdotu_c_example.c:
gcc -c -I/opt/acml3.6.0/gnu32/include -m32 cdotu_c_example.c -o cdotu_c_example.o
Linking program cdotu_c_example.exe:
gcc -m32 cdotu_c_example.o /opt/acml3.6.0/gnu32/lib/libacml.a -lg2c -lm -o cdotu_c_example.exe
Running program cdotu_c_example.exe:
(export LD_LIBRARY_PATH='/opt/acml3.6.0/gnu32/lib:/opt/acml3.6.0/gnu32/lib'; ./cdotu_c_example.exe > cdotu_c_example.res 2>&1)
ACML example: dot product of two complex vectors using cdotu
------------------------------------------------------------
Vector x: ( 1.0000, 2.0000)
( 2.0000, 1.0000)
( 1.0000, 3.0000)
Vector y: ( 3.0000, 1.0000)
( 1.0000, 4.0000)
( 1.0000, 2.0000)
r = x.y = ( -6.000, 21.000)
Compiling program cfft1d_c_example.c:
gcc -c -I/opt/acml3.6.0/gnu32/include -m32 cfft1d_c_example.c -o cfft1d_c_example.o
Linking program cfft1d_c_example.exe:
gcc -m32 cfft1d_c_example.o /opt/acml3.6.0/gnu32/lib/libacml.a -lg2c -lm -o cfft1d_c_example.exe
Running program cfft1d_c_example.exe:
(export LD_LIBRARY_PATH='/opt/acml3.6.0/gnu32/lib:/opt/acml3.6.0/gnu32/lib'; ./cfft1d_c_example.exe > cfft1d_c_example.res 2>&1)
ACML example: FFT of a complex sequence using cfft1d
----------------------------------------------------
Components of discrete Fourier transform:
Real Imag
0 ( 2.4836,-0.4710)
1 (-0.5518, 0.4968)
2 (-0.3671, 0.0976)
3 (-0.2877,-0.0586)
4 (-0.2251,-0.1748)
5 (-0.1483,-0.3084)
6 ( 0.0198,-0.5650)
Original sequence as restored by inverse transform:
Original Restored
Real Imag Real Imag
0 ( 0.3491,-0.3717) ( 0.3491,-0.3717)
1 ( 0.5489,-0.3567) ( 0.5489,-0.3567)
2 ( 0.7478,-0.3117) ( 0.7478,-0.3117)
3 ( 0.9446,-0.2370) ( 0.9446,-0.2370)
4 ( 1.1385,-0.1327) ( 1.1385,-0.1327)
5 ( 1.3285, 0.0007) ( 1.3285, 0.0007)
6 ( 1.5137, 0.1630) ( 1.5137, 0.1630)
......
ACML example: solution of linear equations using sgetrf/sgetrs
--------------------------------------------------------------
Matrix A:
1.8000 2.8800 2.0500 -0.8900
5.2500 -2.9500 -0.9500 -3.8000
1.5800 -2.6900 -2.9000 -1.0400
-1.1100 -0.6600 -0.5900 0.8000
Right-hand-side matrix B:
9.5200 18.4700
24.3500 2.2500
0.7700 -13.2800
-6.2200 -6.2100
Solution matrix X of equations A*X = B:
1.0000 3.0000
-1.0000 2.0000
3.0000 4.0000
-5.0000 1.0000
Testing: no example difference files were generated.
Test passed OK
-bash-3.1#
If everything checks out, the next step is to compile a version of cblas from source.
Cblas Install
See http://www.netlib.org/blas/ for more details
- Download the cblas source code and unzip into /usr/local/src
To compile we follow George Nurser's writeup (thanks for the help on this part George...). For the 32bit EC2 machines, we changed the compile flags in /usr/local/src/CBLAS/Makefile.LINUX to:
CFLAGS = -O3 -DADD_ -pthread -fno-strict-aliasing -m32 -msse2 -mfpmath=sse -march=opteron -fPIC
FFLAGS = -Wall -fno-second-underscore -fPIC -O3 -funroll-loops -march=opteron -mmmx -msse2 -msse -m3dnow
RANLIB = ranlib
BLLIB = /opt/acml3.6.0/gnu32/lib/libacml.so
CBDIR = /usr/local/src/CBLAS
Next we copy the Makefile.LINUX to Makefile.in and execute "make". The resulting cblas.a must then be copied to libcblas.a in the same directory as the libacml.so:
-bash-3.1# cd /usr/local/src/CBLAS
-bash-3.1# ln -s Makefile.LINUX Makefile.in
-bash-3.1# make all
-bash-3.1# cd/usr/local/src/CBLAS/lib/LINUX
-bash-3.1# cp cblas_LINUX.a /opt/acml3.6.0/gnu32/lib/libcblas.a
-bash-3.1# cd /opt/acml3.6.0/gnu32/lib/
-bash-3.1# chmod +x libcblas.a
This directory then needs to be added to the $LD_LIBRARY_PATH and $LD_RUN_PATH before we compile numpy.
export LD_LIBRARY_PATH=/opt/acml3.6.0/gnu32/lib
export LD_RUN_PATH=/opt/acml3.6.0/gnu32/lib
Compile Numpy
Compile numpy from source:
cd /usr/local/src
svn co http://svn.scipy.org/svn/numpy/trunk/ ./numpy-trunk
cd numpy-trunk
Before building scipy with setup.py, we need to configure a site.cfg file in both the numpy-trunk directory and the distutils subdirectory. This was overlooked the first time I did this which resulted in a slower default Numpy install that was missing the ACML optimized lapack and blas. If the install fails, make sure that you get rid of earlier tries with:
rm -rf /usr/lib/python2.4/site-packages/numpy
rm -rf usr/local/src/numpy-trunk/build
again, for more details see George Nurser's writeup
Contents of both site.cfg files for my install:
[DEFAULT]
library_dirs = /usr/local/lib
include_dirs = /usr/local/include
[blas]
blas_libs = cblas, acml
library_dirs = /opt/acml3.6.0/gnu32/lib
include_dirs = /usr/local/src/CBLAS/src
[lapack]
language = f77
lapack_libs = acml
library_dirs = /opt/acml3.6.0/gnu32/lib
include_dirs = /opt/acml3.6.0/gnu32/include
We execute the actual compile with the following:
python setup.py build
python setup.py install
cd ../
rm -R numpy-trunk
Scipy Install
Take a look at the instructions for the lapack and blas environment as described here:
http://www.scipy.org/Installing_SciPy/BuildingGeneral
I found that no modifications from the defaults were needed, the install should pick up the libraries built in the previous steps.
Install Scipy from source:
cd /usr/local/src
svn co http://svn.scipy.org/svn/scipy/trunk/ ./scipy-trunk
cd scipy-trunk
python setup.py build
python setup.py install
cd ../
rm -R scipy-trunk
Verify numpy and scipy work and are using the correct libraries:
-bash-3.1# python
Python 2.4.4 (#1, Oct 23 2006, 13:58:00)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy,scipy
>>> numpy.show_config()
>>> numpy.show_config()
blas_info:
libraries = ['cblas', 'acml']
library_dirs = ['/opt/acml3.6.0/gnu32/lib']
language = f77
lapack_info:
libraries = ['acml']
library_dirs = ['/opt/acml3.6.0/gnu32/lib']
language = f77
atlas_threads_info:
NOT AVAILABLE
blas_opt_info:
libraries = ['cblas', 'acml']
library_dirs = ['/opt/acml3.6.0/gnu32/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
atlas_blas_threads_info:
NOT AVAILABLE
lapack_opt_info:
libraries = ['acml', 'cblas', 'acml']
library_dirs = ['/opt/acml3.6.0/gnu32/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
atlas_info:
NOT AVAILABLE
lapack_mkl_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
atlas_blas_info:
NOT AVAILABLE
mkl_info:
NOT AVAILABLE
>>> scipy.show_config()
blas_info:
libraries = ['cblas', 'acml']
library_dirs = ['/opt/acml3.6.0/gnu32/lib']
language = f77
lapack_info:
libraries = ['acml']
library_dirs = ['/opt/acml3.6.0/gnu32/lib']
language = f77
atlas_threads_info:
NOT AVAILABLE
blas_opt_info:
libraries = ['cblas', 'acml']
library_dirs = ['/opt/acml3.6.0/gnu32/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
atlas_blas_threads_info:
NOT AVAILABLE
djbfft_info:
NOT AVAILABLE
lapack_opt_info:
libraries = ['acml', 'cblas', 'acml']
library_dirs = ['/opt/acml3.6.0/gnu32/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
fftw3_info:
libraries = ['fftw3']
library_dirs = ['/usr/lib']
define_macros = [('SCIPY_FFTW3_H', None)]
include_dirs = ['/usr/include']
umfpack_info:
NOT AVAILABLE
atlas_info:
NOT AVAILABLE
lapack_mkl_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
atlas_blas_info:
NOT AVAILABLE
mkl_info:
NOT AVAILABLE
>>>
Now that we have numpy and scipy, we can install matplotlib:
yum -y install python-matplotlib
We can benchmark the performance improvement from the ACML libraries using a script George Nurser provided:
EC2 image with Default Numpy:
-bash-3.1# python bench_blas2.py
Tests x.T*y x*y.T A*x A*B A.T*x half 2in2
Dimension: 5
Array 1.8900 0.4300 0.3900 0.4300 1.2600 1.4500 1.6000
Matrix 6.6100 2.0900 0.9100 0.9400 1.4200 3.1300 3.8100
Dimension: 50
Array 18.8300 2.1600 0.7000 12.8300 2.3100 1.7300 1.9000
Matrix 66.3900 3.9900 1.2200 13.4600 1.7500 3.4300 4.1100
Dimension: 500
Array 1.9800 5.1500 0.6600 125.9200 7.5600 0.3500 0.6700
Matrix 6.8400 5.2200 0.6700 125.9700 0.9000 0.4000 0.7300
EC2 image with Numpy built with ACML:
-bash-3.1# python bench_blas2.py
Tests x.T*y x*y.T A*x A*B A.T*x half 2in2
Dimension: 5
Array 2.0300 0.6500 0.3800 0.7100 1.2000 1.4400 1.5200
Matrix 6.7500 2.4100 0.8400 1.2400 1.3800 3.0300 3.5600
Dimension: 50
Array 20.4500 2.7500 0.5900 11.8300 2.2200 1.7300 1.8000
Matrix 68.2400 4.5900 1.1100 12.4200 1.7100 3.3600 3.9100
Dimension: 500
Array 2.1800 5.1900 0.5800 77.1200 7.4200 0.3300 0.6900
Matrix 6.9500 5.2800 0.5900 77.3400 0.6200 0.3800 0.7500
MPICH2 Install
Install mpich2 from source:
cd /usr/local/src
wget http://www-unix.mcs.anl.gov/mpi/mpich2/downloads/mpich2-1.0.5.tar.gz
tar -xzvf mpich2-1.0.5.tar.gz
cd mpich2-1.0.5
./configure
make
make install
PyMPI install
Build pyMPI from source:
see http://www.llnl.gov/computing/develop/python/pyMPI.pdf
cd /usr/local/src
wget http://downloads.sourceforge.net/pympi/pyMPI-2.4b2.tar.gz?modtime=1122458975&big_mirror=0
tar -xzvf pyMPI-2.4b2.tar.gz
cd pyMPI-2.4b2
The basic build and install is invoked with:
./configure --with-includes=-I/usr/local/include
make
make install
This will build a default version of pyMPI based on the python program the configure script finds in your path. It also tries to find mpcc, mpxlc, or mpicc to do the compiling and linking with the MPI libraries.
PyTables Install
Install PyTables from source (requires the previous yum install of hdf5-devel)
cd /usr/local/src
wget http://downloads.sourceforge.net/pytables/pytables-1.4.tar.gz
tar -xvzf pytables-1.4.tar.gz
cd pytables-1.4/
python setup.py build_ext --inplace
python setup.py install
Configuration and Cleanup
To help reduce the image size, lets remove the compressed source files we downloaded:
-bash-3.1# rm ec2-ami-tools.noarch.rpm mpich2-1.0.5.tar.gz pyMPI-2.4b2.tar.gz acml-3-6-0-gnu-32bit.tgz contents-acml-3-6-0-gnu-32bit.tgz pytables-1.4.tar.gz
For the mpich configuration we need to add a couple of additional files to the base install:
Create the file mpd.conf as follows (with your own password)
cd /etc
touch .mpd.conf
chmod 600 .mpd.conf
nano .mpd.conf
secretword=Myp@ssW0rD
Next we set the ssh variable "StrictHostKeyChecking" to "no". This is an evil hack to avoid the tedious adding of each compute node host... I'm assuming these EC2 nodes will only connect to eachother, please be careful.
See the following article for why this is risky: http://www.securityfocus.com/infocus/1806
edit the ssh_config file:
nano /etc/ssh/ssh_config
change the following line..
# StrictHostKeyChecking ask
StrictHostKeyChecking no
Changing this setting avoids having to manually accept each compute node later on:
The authenticity of host 'domu-12-31-34-00-00-3a.usma2.compute.amazonaws.com (216.182.236.94)' can't be established.
RSA key fingerprint is 58:ae:0b:e7:a6:d8:d0:00:4f:ca:22:53:42:d5:e5:22.
Are you sure you want to continue connecting (yes/no)? yes
Creating a non-root user
We should run the MPI process as a non-root user, so we will create a "lamuser" account on the instance (in another version of this tutorial, I used LAM instead of MPICH2). Substitute your own cert, keys, and passwords.
-bash-3.1# adduser lamuser
-bash-3.1# passwd lamuser
Changing password for user lamuser.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
Now configure the .bash_profile and .bashrc:
-bash-3.1# cd /home/lamuser/
-bash-3.1# ls
-bash-3.1# ls .
./ ../ .bash_logout .bash_profile .bashrc
-bash-3.1# nano .bash_profile
The contents of bash_profile should be as follows (uncomment the LAM settings if you want to use LAM MPI instead of MPICH2):
-bash-3.1# more .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
LAMRSH="ssh -x"
export LAMRSH
#LD_LIBRARY_PATH="/usr/local/lam-7.1.2/lib/"
#export LD_LIBRARY_PATH
MPICH_PORT_RANGE="2000:8000"
export MPICH_PORT_RANGE
PATH=$PATH:$HOME/bin
#PATH=/usr/local/lam-7.1.2/bin:$PATH
#MANPATH=/usr/local/lam-7.1.2/man:$MANPATH
export PATH
#export MANPATH
We need to give the lamuser the same MPI configuration we created for the root user in part 1...
Create the file .mpd.conf as follows (with your own password for the secretword):
cd /home/lamuser
touch .mpd.conf
chmod 600 .mpd.conf
nano .mpd.conf
secretword=Myp@ssW0rD
The last step is to set ownership on the directory contents to the user:
chown -R lamuser:lamuser /home/lamuser
Adding the S3 Libraries
Download the developer tools for S3 to the instance:
-bash-3.1# wget http://developer.amazonwebservices.com/connect/servlet/KbServlet/download/134-102-759/s3-example-python-library.zip
-bash-3.1# unzip s3-example-python-library.zip
Archive: s3-example-python-library.zip
creating: s3-example-libraries/python/
inflating: s3-example-libraries/python/README
inflating: s3-example-libraries/python/S3.py
inflating: s3-example-libraries/python/s3-driver.py
inflating: s3-example-libraries/python/s3-test.py
Rebundle the compute node image
We are going to make this a public AMI, so we need to clear out some data first.
Here's the advice from the Amazon EC2 Developer Guide:
Protect Yourself
We have looked at making shared AMIs safe, secure and useable for the users who launch them, but if you publish a shared AMI you should also take steps to protect yourself against the users of your AMI. This section looks at steps you can take to do this.
We recommend against storing sensitive data or software on any AMI that you share. Users who launch a shared AMI potentially have access to rebundle it and register it as their own. Follow these guidelines to help you to avoid some easily overlooked security risks:
- Always delete the shell history before bundling. If you attempt more than one bundle upload in the same image the shell history will contain your secret access key.
- Bundling a running instance requires your private key and X509 certificate. Put these and other credentials in a location that will not be bundled (such as the ephemeral store).
- Exclude the ssh authorized keys when bundling the image. The Amazon public images store the public key an instance was launched with in that instance's ssh authorized keys file.
ssh into the modified image and clean up:
rm -f /root/.ssh/authorized_keys
rm -f /home/lamuser/.ssh/authorized_keys
rm ~/.bash_history
rm /var/log/secure
rm /var/log/lastlog
The ec2-bundle-vol command has some optional parameters we will use:
-bash-3.1# ec2-bundle-vol --help
Usage: ec2-bundle-vol PARAMETERS
MANDATORY PARAMETERS
-c, --cert PATH The path to the user's PEM encoded RSA public key certificate file.
-k, --privatekey PATH The path to the user's PEM encoded RSA private key file.
-u, --user USER The user's EC2 user ID (Note: AWS account number, NOT Access Key ID).
OPTIONAL PARAMETERS
-e, --exclude DIR1,DIR2,... A list of absolute directory paths to exclude. E.g. "dir1,dir2,dir3". Overrides "--all".
-a, --all Include all directories, including those on remotely mounted filesystems.
-p, --prefix PREFIX The filename prefix for bundled AMI files. E.g. "my-image". Defaults to "image".
-s, --size MB The size, in MB (1024 * 1024 bytes), of the image file to create. The maximum size is 10240 MB.
-v, --volume PATH The absolute path to the mounted volume to create the bundle from. Defaults to "/".
-d, --destination PATH The directory to create the bundle in. Defaults to "/tmp".
--ec2cert PATH The path to the EC2 X509 public key certificate. Defaults to "/etc/aes/amiutil/cert-ec2.pem".
--debug Display debug messages.
-h, --help Display this help message and exit.
-m, --manual Display the user manual and exit.
Execute the same bundle command we ran previously, but give the image a prefix name:
-bash-3.1# ec2-bundle-vol -d /mnt -p fc6-python-mpi-node -k /mnt/pk-FOOXYZ.pem -c /mnt/cert-BARXYZ.pem -u 99999ABC -s 5536
Copying / into the image file /mnt/image...
Excluding:
/sys
/proc
/proc/sys/fs/binfmt_misc
/dev
/media
/mnt
/proc
/sys
/mnt/image
/mnt/img-mnt
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.015051 seconds, 69.7 MB/s
mke2fs 1.39 (29-May-2006)
warning: 256 blocks unused.
Bundling image file...
Splitting /mnt/image.tar.gz.enc...
Created fc6-python-mpi-node.part.00
Created fc6-python-mpi-node.part.01
Created fc6-python-mpi-node.part.02
Created fc6-python-mpi-node.part.03
Created fc6-python-mpi-node.part.04
Created fc6-python-mpi-node.part.05
Created fc6-python-mpi-node.part.06
Created fc6-python-mpi-node.part.07
Created fc6-python-mpi-node.part.08
Created fc6-python-mpi-node.part.09
Created fc6-python-mpi-node.part.10
Created fc6-python-mpi-node.part.11
Created fc6-python-mpi-node.part.12
Created fc6-python-mpi-node.part.13
Created fc6-python-mpi-node.part.14
...
Created fc6-python-mpi-node.part.39
Created fc6-python-mpi-node.part.40
Created fc6-python-mpi-node.part.41
Generating digests for each part...
Digests generated.
Creating bundle manifest...
ec2-bundle-vol complete.
Now remove the keys and delete the bash history:
-bash-3.1# rm /mnt/pk-*.pem /mnt/cert-*.pem
Upload the keyless node AMI to Amazon S3
bash-3.1# ec2-upload-bundle -b datawrangling-images -m /mnt/fc6-python-mpi-node.manifest.xml -a 1AFOOBARTEST -s F0Bar/T3stId
Setting bucket ACL to allow EC2 read access ...
Uploading bundled AMI parts to https://s3.amazonaws.com:443/datawrangling-images ...
Uploaded image.part.00 to https://s3.amazonaws.com:443/datawrangling-images/fc6-python-mpi-node.part.00.
Uploaded image.part.01 to https://s3.amazonaws.com:443/datawrangling-images/fc6-python-mpi-node.part.01.
...
Uploaded image.part.48 to https://s3.amazonaws.com:443/datawrangling-images/fc6-python-mpi-node.part.48.
Uploaded image.part.49 to https://s3.amazonaws.com:443/datawrangling-images/fc6-python-mpi-node.part.49.
Uploading manifest ...
Uploaded manifest to https://s3.amazonaws.com:443/datawrangling-images/fc6-python-mpi-node.manifest.xml .
ec2-upload-bundle complete
The upload will take several minutes...
Register Compute Node Image
To register the new image with Amazon EC2, we switch back to our local machine and run the following:
peter-skomorochs-computer:~ pskomoroch$ ec2-register datawrangling-images/fc6-python-mpi-node.manifest.xml
IMAGE ami-3e836657
Included in the output is an AMI identifier for our MPI compute node image (ami-4cb85d77 in the example above). In the next part of this tutorial, we will run some basic tests of MPI and pyMPI on EC2 using this image. In part 3, we will add some python scripts to automate routine cluster maintenance and show some computations which we can run with the cluster.