24 July 2015

614. SIESTA with MPI and acml on debian jessie

One of my students might be using SIESTA for some simulations, and a first step towards that is to set it up on my cluster.

This isn't an optimised build -- right now I'm just looking at having a simple parallell build that runs.

I had a look at http://www.pa.msu.edu/people/tomanek/SIESTA-installation.html and http://pelios.csx.cam.ac.uk/~mc321/siesta.html.
 
NOTE: don't use the int64 acml or openblas BLAS libs, or you'll get SIGSEV due to invalid memory reference when running. NWChem is the complete opposite, and for some reason both the int64 and regulat acml libs have the same names. Not sure how that's supposed to work out on a system with nwchem, which needs the int64 libs.

See here for acml on debian. I've got /opt/acml/acml5.3.1/gfortran64_int64/lib in my /etc/ld.so.conf.d/acml.conf on behalf of nwchem.
 Being lazy, I opted for the debian scalapack and libblacs packages:
 
sudo apt-get install libscalapack-mpi-dev libblacs-mpi-dev libopenmpi-dev

To get the link to the SIESTA code, go to http://departments.icmab.es/leem/siesta/CodeAccess/selector.html

Then, if you're an academic, you can do:
sudo mkdir /opt/siesta
sudo chown $USER /opt/siesta
cd /opt/siesta
wget http://departments.icmab.es/leem/siesta/CodeAccess/Code/siesta-3.2-pl-5.tgz
tar xvf siesta-3.2-pl-5.tgz
cd siesta-3.2-pl-5/Obj
sh ../Src/obj_setup.sh
*** Compilation setup done. *** Remember to copy an arch.make file or run configure as: ../Src/configure [configure_options]
../Src/./configure --help
`configure' configures siesta 2.0 to adapt to many kinds of systems. Usage: ./configure [OPTION]... [VAR=VALUE]... [..] Installation directories: --prefix=PREFIX install architecture-independent files in PREFIX [/usr/local] --exec-prefix=EPREFIX install architecture-dependent files in EPREFIX [PREFIX] By default, `make install' will install all the files in `/usr/local/bin', `/usr/local/lib' etc. You can specify an installation prefix other than `/usr/local' using `--prefix', for instance `--prefix=$HOME'. [..] --enable-mpi Compile the parallel version of SIESTA --enable-debug Compile with debugging support --enable-fast Compile with best known optimization flags Optional Packages: --with-PACKAGE[=ARG] use PACKAGE [ARG=yes] --without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no) --with-netcdf=<lib> use NetCDF library --with-siesta-blas use BLAS library packaged with SIESTA --with-blas=<lib> use BLAS library --with-siesta-lapack use LAPACK library packaged with SIESTA --with-lapack=<lib> use LAPACK library --with-blacs=<lib> use BLACS library --with-scalapack=<lib> use ScaLAPACK library [..]
../Src/./configure --enable-mpi
checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu [..] checking for mpifc... no checking for mpxlf... no checking for mpif90... mpif90 checking for MPI_Init... no checking for MPI_Init in -lmpi... yes [..] checking for sgemm in /opt/openblas/lib/libopenblas.so... yes checking LAPACK already linked... yes checking LAPACK includes divide-and-conquer routines... yes configure: using DC_LAPACK routines packaged with SIESTA due to bug in library. Linker flag might be needed to avoid duplicate symbols configure: creating ./config.status config.status: creating arch.make
Edit arch.make:
# # This file is part of the SIESTA package. # # Copyright (c) Fundacion General Universidad Autonoma de Madrid: # E.Artacho, J.Gale, A.Garcia, J.Junquera, P.Ordejon, D.Sanchez-Portal # and J.M.Soler, 1996- . # # Use of this software constitutes agreement with the full conditions # given in the SIESTA license, as signed by all legitimate users. # .SUFFIXES: .SUFFIXES: .f .F .o .a .f90 .F90 SIESTA_ARCH=x86_64-unknown-linux-gnu--unknown FPP= FPP_OUTPUT= FC=mpif90 RANLIB=ranlib SYS=nag SP_KIND=4 DP_KIND=8 KINDS=$(SP_KIND) $(DP_KIND) FFLAGS=-g -O2 FPPFLAGS= -DMPI -DFC_HAVE_FLUSH -DFC_HAVE_ABORT LDFLAGS= ARFLAGS_EXTRA= FCFLAGS_fixed_f= FCFLAGS_free_f90= FPPFLAGS_fixed_F= FPPFLAGS_free_F90= BLAS_LIBS=-L/opt/acml/acml5.3.1/gfortran64/lib -lacml LAPACK_LIBS= BLACS_LIBS=-L/usr/lib -lblacs-openmpi -lblacsCinit-openmpi SCALAPACK_LIBS=-L/usr/lib -lscalapack-openmpi COMP_LIBS=dc_lapack.a NETCDF_LIBS= NETCDF_INTERFACE= MPI_LIBS= -L/usr/lib/openmpi/lib -lmpi -lmpi_f90 LIBS=$(SCALAPACK_LIBS) $(BLACS_LIBS) $(LAPACK_LIBS) $(BLAS_LIBS) $(NETCDF_LIBS) $(MPI_LIBS) -lpthread #SIESTA needs an F90 interface to MPI #This will give you SIESTA's own implementation #If your compiler vendor offers an alternative, you may change #to it here. MPI_INTERFACE=libmpi_f90.a MPI_INCLUDE=. #Dependency rules are created by autoconf according to whether #discrete preprocessing is necessary or not. .F.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_fixed_F) $< .F90.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_free_F90) $< .f.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_fixed_f) $< .f90.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_free_f90) $<
make
cd ../
ln -s Obj/siesta siesta

I added /opt/siesta/siesta-3.2-pl-5 to $PATH.

To test, edit /opt/siesta/siesta-3.2-pl-5/test.mk:
6 #SIESTA=../../../siesta 7 SIESTA=mpirun -n 2 ../../../siesta
Then
cd /opt/siesta/siesta-3.2-pl-5/Tests/h3po4_2
export LD_LIBRARY_CONFIG=/opt/acml/acml5.3.1/gfortran64/lib 
make
>>>> Running h3po4_2 test... ==> Copying pseudopotential file for H... ==> Copying pseudopotential file for O... ==> Copying pseudopotential file for P... ==> Running SIESTA as mpirun -n 2 ../../../siesta ===> SIESTA finished successfully

Also, look at work/h3po4_2.out:
* Running on    2 nodes in parallel
>> Start of run:  24-JUL-2015  21:58:13

                           ***********************       
                           *  WELCOME TO SIESTA  *       
                           ***********************       

reinit: Reading from standard input
[..]
elaps:  optical           1       0.000       0.000     0.00
  
>> End of run:  24-JUL-2015  21:58:20

No comments:

Post a Comment