Compile and execute CLaMS programs on JUROPA

Compiler

To use the Intel MPI compiler the following command will be necessary:

```
module load parastation/intel 
```

Compilation

All used libraries and module files are linked to:

/lustre/jhome4/jicg11/jicg1108/clams_lib
/lustre/jhome4/jicg11/jicg1108/clams_mod

The CLaMS makefiles are adjusted for compilation of CLaMS programs on JUROPA with the Intel compiler (mkincl/config/config.Linux_ifc).

The netCDF version 3.6.3 is used. For using netCDF 4 see Used Libraries (netCDF3/netCDF4).

The CLaMS programs can be compiled with:

```
   1 gmake useComp=ifc [useMPI=true] progname
```

Create batch script

Write a batch script including the mpiexec command for the just compiled executable:

   1 #!/bin/bash -x
   2 #MSUB -l nodes=<no of nodes>:ppn=<no of procs/node>
   3 #MSUB -l walltime=<hh:mm:ss>
   4 #MSUB -e <full path for error file>
   5 #MSUB -o <full path for output file>
   6 #MSUB -M <mail-address>
   7 #      official mail address
   8 #MSUB -m n|a|b|e
   9 #      send mail: never, on abort, beginning or end of job
  10 
  11 ### start of jobscript
  12 cd $PBS_O_WORKDIR
  13 echo "workdir: $PBS_O_WORKDIR"
  14 NP=$(cat $PBS_NODEFILE | wc -l) # nodes*ppn
  15 echo "running on $NP cpus ..."
  16 mpiexec -np $NP <executable>

Job Limits

Batch jobs

max. number of nodes: 256 (default: 1)
max. wall-clock time: 12 hours (default: 30 min)

Interactive jobs:

max. number of nodes: 2 (default: 1)
max. wall-clock time: 30 min (default: 10 min)

Submit job

Submit the job with:

```
msub <jobscript>
```

On success msub returns the job ID of the submitted job.

Useful commands

checkjob -v <jobid>
get detailed information about a job
showq [-r]
show status of all (running) jobs
canceljob <jobid>
cancel a job
mjobctl -q starttime <jobid>
show estimated starttime of specified job
mjobctl --help
show detailed information about this command

Problems with timeouts using MPI

Currently jobs requesting > 128 nodes ( >1024 cores) fail with PMI barrier timeouts. This is caused by a problem in the ParaStation process management which could not be fixed yet. As a workaround ParTec recommends the setting of the following environment variables in the job script of large MPI applications:

  export __PSI_LOGGER_TIMEOUT=3600  
  export PMI_BARRIER_TMOUT=1800

just before the 'mpiexec' call. Similar problems have been observed on workstations as well.

Interactive Session

Allocate interactive partition with X11 forwarding:
```
msub -I -X -l nodes=#:ppn=8
```
possible number of nodes: 1 or 2
Start your applications
Deallocate interactive partition with:
```
exit
```