README for Linpack for QS22, version 1.0a
--------------------------------------

 0. Install Cell SDK according to the SDK installation instructions.

 1. Install Atlas BLAS.

      cd /usr/local/src
      wget http://downloads.sourceforge.net/math-atlas/atlas3.8.1.tar.bz2

      tar -xjvf atlas3.8.1.tar.bz2

      mkdir atlas_build ; cd atlas_build
      ../ATLAS/configure -b 64 -A 5 -V 1 -m 3200
      vi Make.inc
         # :0,$s/=970/=cell/g
         # :0,$s/= gcc/= ppu-gcc/g
      make 

      Note: the "make" takes a *long* time! (e.g. 12 hours)

      make install

      # The libraries are installed in /usr/local/atlas/lib

 2. Install OpenMPI.  Here are the steps to install OpenMPI with TCP/IP as the
    primary transport layer.  (Slightly different configure options are needed
    to use Infiniband as the primary transport layer.) As root:

      cd /usr/local/src
      wget http://www.open-mpi.org/software/ompi/v1.2/downloads/openmpi-1.2.8.tar.bz2
      tar -xjf openmpi-1.2.8.tar.bz2 
      cd openmpi-1.2.8
      mkdir obj64
      cd obj64
      ../configure --prefix=/usr/local/openmpi64-1.2.8 --disable-mpi-f90 \
                   --enable-ltdl-convenience --with-platform=optimized \
                   CC=ppu-gcc CPP=ppu-cpp CXX=ppu-g++ CXXPP=ppu-cpp \
                   CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 LDFLAGS=-m64 \
                   --with-wrapper-cflags=-m64 --with-wrapper-ldflags=-m64 --with-wrapper-fflags=-m64
      make -j2
      make install

      ln -s /usr/local/openmpi64-1.2.8 /usr/local/openmpi

      echo "/usr/local/openmpi/lib" > /etc/ld.so.conf.d/openmpi.conf 
      ldconfig

      # Create /etc/profile.d/openmpi.sh
      cat > /etc/profile.d/openmpi.sh << EOD
if ! echo \${PATH} | grep -q /usr/local/openmpi/bin ; then
    PATH=/usr/local/openmpi/bin:\${PATH}
fi
EOD
      chmod +x /etc/profile.d/openmpi.sh

      # Now verify the install ...

      # Log out and log back in, then
      echo $PATH        # should contain /usr/local/openmpi/bin

      file /usr/local/openmpi/lib/libmpi.so.0.0.0    # should say 64-bit PowerPC

      ompi_info         # should show gobs of stuff

      mpirun -np 2 --host $(hostname) hostname

 3. Grab the HPL 1.0a version of the hpl source from the web

      wget http://www.netlib.org/benchmark/hpl/hpl.tgz
      tar -xzvf hpl.tgz

 4. Patch hpl with changes to add qs22 support and build:

      cd hpl
      patch -p 0 < hpl_qs22.patch
      export CELL_TOP=/opt/cell/sdk
      make arch=qs22

 5. Configure huge pages.  By default, Linpack for QS22 uses huge pages to hold the 
    matrix and other large buffer areas, but this can be changed with settings in
    the makefile.  To configure huge pages.

      HUGE_TOT=448    # Pick an appropriate value here. Each huge page is 16MB.

      mkdir -p /huge
      /sbin/sysctl -w vm.nr_hugepages=$HUGE_TOT
      mount -t hugetlbfs nodev /huge
      chmod 777 /huge

    You can verify that the huge pages were configured as requested with

      cat /proc/meminfo

 6. Run. 

      cd <path>/hpl/bin/qs22

      # Set up parameters.  The HPL.dat file in hpl/bin/qs22 defines the parameters
      # for a set of tests to be executed.  Linpack for QS22 currently imposes some 
      # restrictions on the values for these settings.  In particular:
      #   * NB must be 128
      #   * L1 must be not transposed (1)
      #   * U must be transposed (0)
      #   * memory alignment must be 64 (or some multiple of 64)

      mpirun  --mca btl_openib_want_fork_support 0 \
              -np 1 numactl --physcpubind=0 ./xhpl : \
              -np 1 numactl --physcpubind=3 ./xhpl