Best Practice Guide – Generic x86

Vegard Eide


Nikos Anastopoulos


Henrik Nagel



1. Introduction

This guide provides an overview of best practices on using x86 HPC cluster systems. Topics discussed include programming environment, e.g. choice of compiler options and numerical libraries, runtime environment settings, debuggers, performance analysis tools and running jobs in batch systems.

The guide is not system-specific, i.e. it will not cover information that is targeted for one specific machine system but instead focus on topics that will be common to systems based on the x86 architecture. To get details about architectures and configurations for specific systems users are referred to system user guides, some of which can be found at

The guide is organised into these parts:

  • Chapter 2 gives a general overview of the x86 architecture and its properties.
  • Chapter 3 describes the environment setup through the modules system used when porting and developing codes. A selection of compilers and a list of common and useful options are described. The chapter discusses four of the available MPI implementations and their general usage when building parallel applications, and job launcher commands with options supported. Included is also information for compiling OpenMP codes, a section describing hybrid programming, and a coverage of math libraries used when building applications.
  • Chapter 4 describes ways of improving application performance using compiler optimisation flags, and runtime tuning through MPI and OpenMP environment variables.
  • Chapter 5 discusses the process of debugging codes using compiler debug flags and environment variables, and using command line and GUI debuggers.
  • Chapter 6 describes how to analyse code behaviour through MPI summary statistics, and to find th
    e source of performance problems using profilers and analysis tools.
  • Chapter 7 covers some of the systems used for running batch jobs describing commands, job resources, and including sample batch scripts.

2. x86 – Basic Properties

2.1. Basic Properties

x86 is a instruction set architecture first introduced by Intel in 1978 in their 8086 CPU. The term x86 derives from the main CPU in the first succeeding processor generations that had names ending in “86”. The instruction set length of the 8086 CPU was 16 bits. This was extended to 32 bits in the 80386 CPU by Intel in 1985, and later to 64 bits by AMD in their Opteron processor in 2003. The 64-bit extension architecture is also known as x86-64.

The x86 designs are superscalar allowing for instruction-level parallelism (ILP) where several independent instructions can be executed per clock cycle. The efficiency of superscalar CPUs can be additionally improved using simultaneous multithreading (SMT), see description below. x86 processors are also capable of out of order and speculative execution, i.e. instructions are executed in an order governed by the availability of input data rather than the original order in a program, and instructions can be carried out, if resources are available, before it is known if the work is needed. The x86 architecture has byte-addressing enabled and words are stored in memory with little-endian byte order.

Modern x86 processors include SIMD (Single Instruction, Multiple Data) units where instructions can operate in parallel on multiple data objects. These are implemented in the 3DNow! instruction set extension developed by AMD in 1997 and the Streaming SIMD Extensions (SSE) instruction sets first introduced by Intel in 1999. The SIMD registers allow for loading or storing 128 bits of memory data in one single instruction that can be operated on in parallel by vector instructions. Later extensions to SSE (introduced in SSE3) included thread-handling instructions to improve the performance of Intel’s Hyper-Threading Technology (see below). With the introduction of Intel’s SandyBridge processors registers were doubled to 256 bits, and extended functionality was added through the AVX (Advanced Vector Extensions) instruction set.[1]

2.2. Simultaneous Multithreading

Simultaneous Multithreading (SMT) is a technique to increase the throughput of a conventional superscalar processor by multiplexing the execution of concurrent hardware threads. The key idea is to fill up the empty slots in the processor pipeline that arise when an application thread has low instruction-level parallelism (ILP) or suffers from long-latency stalls (e.g. cache misses, branch mispredictions), with instructions from an alternate thread that are ready to execute. Most resources in an SMT processor (e.g., caches, functional units, instruction queues) are either shared or partitioned between threads, while only those that are necessary to keep the architectural state for each thread (e.g. architectural registers) are replicated.

The hardware threads within an SMT processor, often reffered to as logical processors, are exposed to the operating system as conventional processors. Hyper-Threading (HT) Technology is Intel’s 2-way SMT implementation, which makes a single physical processor appear to the software as two logical processors. This is considered to be a cost-effective approach to provide hardware concurrency, since the additional cost in terms of hardware overhead and energy consumption is rather small (e.g., less than 5% in early HT implementations, as compared to their non-HT counterparts). However, due to the fact that hyper-threads operate mostly under an “all-shared”, non-dedicated environment, there are performance implications that one needs to take into account when mapping a parallel application to the underlying architecture. In the following, we list the most important of them:

  • The large degree of sharing in functional units often leads to contention between threads. This results to performance degradation as compared to the case when threads do not share any resources. In general, this slowdown depends on the profile of the application, but as a rule-of-thumb we can argue that the speedup boost that one can expect from HT ranges between 30%-50%.
  • Resource contention, and hence efficiency of HT, highly depend on the heterogeneity of the application threads that are being co-scheduled on the hyper-threaded processor. Generally, it is decreased when threads have complementary needs for functional units, e.g. when we co-execute a memory-intensive thread with a computation-intensive thread, an integer-bound thread with a fp-bound thread, a high-ILP thread with a low-ILP one. Contention increases when threads compete for the same unit at the same time.
  • To maximize the gain of a parallel application from HT, one can employ non-conventional parallelization techniques that target at increasing threads heterogeneity. Such techniques can be:
    • assigning communication operations in an MPI application to a separate thread, and pairing the computation and communication threads on the same HT processor.
    • assigning threads with imbalanced workloads (e.g. busy threads and threads with large idle periods) on the same HT processor
    • using prefetching helper threads that improve memory performance of the main computation thread on the same HT processor

To detect Hyper-Threading Technology and find which logical processors reside in the same hyper-threaded core (sibling threads), one can use the sys pseudo file system (sysfs) provided by the Linux kernel to publish machine topology. The directories and files that are important for extracting HT-related information are summarized in the following table.

Table 1. CPU Topololy Info

Directory / File Information provide
/sys/devices/system/cpu/cpuX/ Contains information regarding CPU with id “X” (the unique processor id as seen by the OS).
/sys/devices/system/cpu/cpuX/topology/ Details on X’s topology.
/sys/devices/system/cpu/cpuX/topology/thread_siblings_list In an architecture with multi-threaded cores (e.g. Intel Hyper-Threading), shows which other CPUs are in the same core with X, as different hardware threads.


3. Programming Environment

3.1. Modules

Software installations on computer systems span many applications, and many different version numbers of the same application. It is not possible (nor desirable) to use them all at the same time, since different versions of the same application may conflict with each other.

In order to simplify the control of which application versions are available in a specific session the Modules package provides for the dynamic modification of the user’s environment via modulefiles (or modules). Each modulefile contains the information needed to configure the environment for an application. The module command is used for interpreting modulefiles and will alter or set shell environment variables such as PATH, LIBRARY_PATH, etc.

The module command uses various sub-commands that perform different actions. A subset of available sub-commands is listed in the table below.

Table 2. Selected module Sub-commands

Sub-command Description
help Print the list of available sub-commands.
avail List available modules.
list List loaded modules.
show <module> Display the full path of <module> and the environment changes made if loaded.
load <module> Load <module> into the shell environment.
unload <module> Remove <module> from the shell environment.
switch <module1> <module2> Unload <module1> and load <module2>.
purge Unload all loaded modules.


For example, check available software packages installed (as modules) on the system:

$ module avail
adf/2012.01(default)          adf/2012.01a
ansys/13.0                    boost/1.46.1
fftw/2.1.5                    fftw/3.3(default)

When more than one version of a software package is installed one of the modules will be suffixed with “(default)“. If no version number is specified for a given software this is the modulefile acted on by module. E.g.

$ module load adf

will load the adf/2012.01 version of the ADF package from the list in the example above.

To display information about a software package use e.g.

$ module show fftw/2.1.5

module-whatis    library for computing discrete the Fourier transform
prepend-path     CPATH /path/to/fftw/fftw-2.1.5/include
prepend-path     FPATH /path/to/fftw/fftw-2.1.5/include
prepend-path     LIBRARY_PATH /path/to/fftw/fftw-2.1.5/lib
prepend-path     LD_LIBRARY_PATH /path/to/fftw/fftw-2.1.5/lib
append-path      MANPATH /path/to/fftw/fftw-2.1.5/share/man

Loads FFTW version 2.1.5

3.2. Compiling

3.2.1. Compilers

Typically, several compilers will be available on most computer systems. The compilers will be accessed by loading the appropriate module file. This guide will cover the compiler suites available from GCC, Intel, Portland and PathScale for the Fortran and C/C++ languages. The table below summarizes invocation commands for the different compilers.

Table 3. Compilers

Compiler Language Invocation command
GCC C gcc
C++ g++
Fortran 77 / Fortran 95 gfortran
Intel C icc
C++ icpc
Fortran 77 / Fortran 95 ifort
Portland C pgcc
C++ pgCC
Fortran 77 pgf77
Fortran 95 pgf95
PathScale C pathcc
C++ pathCC
Fortran 77 / Fortran 95 pathf95


3.2.2. General Compiler Flags

The following table shows options that are common to all compilers listed in Table 3, “Compilers”

Table 4. Common Compiler Options

Option Description
-c Compile or assemble the source files, but do not link.
-o filename Name the outputfile filename.
-g Produces symbolic debug information.
-pg Generate extra code to write profile information suitable for the analysis program gprof (see Section 6.2.1, “GNU gprof”).
-Dname Predefine name as a macro for the preprocessor, with definition 1.
-Idir Specifies an additional directory dir to search for include files.
-llibrary Search the library named library when linking.
-Ldir Search for libraries in a specified directory dir.
-fpic Generate position-independent code.
-mcmodel=medium Generate code for models with global and static data larger than 2GB. GCC

The GNU Compiler Collection (GCC) includes compilers for C, C++ and Fortran, and libraries for these languages on a variety of platforms including x86. GCC offers:

  • Features from Fortran 2003 and Fortran 2008
  • Partial support for ANSI/ISO C99
  • Support for the ISO/ANSI C++ standard and partial C++11 compatibility
  • OpenMP 3.1 support

In addition to the compiler options listed in Table 4, “Common Compiler Options” the table below lists a set of selected options available for the GCC compilers.

Table 5. General GCC Compiler Options

Option Description
--version Display the version number.
--help=<category> List a description of options belonging to a specific <category>. E.g. --help=warnings displays options controlling warning messages.
-std=<val> Conform to a specific language standard <val>. (C/C++ only)
-### Print commands to run during stages of compilation without executing them.
-E Preprocessed source files including #line directives are output to stdout.
-E -P Inhibit generation of #line directives in the output from the preprocessing.
-J dir Save/search for module files in directory dir. (Fortran only)
-static Link libraries statically. Dynamic linking is the default.
-fdefault-integer-8 Set the default integer and logical types to 8 byte. (Fortran only)
-fdefault-real-8 Set the default real type to 8 byte. This will also promote the default width of “DOUBLE PRECISION” to 16 bytes. (Fortran only)
-fdefault-double-8 Set the “DOUBLE PRECISION” type to 8 byte. (Fortran only)


For optimising codes using the GCC compilers see GCC Optimisation Flags, and for debugging see Section 5.1, “Compiler Debug Flags and Environment Variables”.

Further Information Intel

The Intel Compiler suite includes compilers for C/C++ and Fortran, and offers support for:

  • Features from Fortran 2003 and Fortran 2008
  • ANSI/ISO C99 and C++ standards
  • OpenMP 3.1

In addition to the compiler options listed in Table 4, “Common Compiler Options” the table below lists a set of selected options available for the Intel compilers.

Table 6. General Intel Compiler Options

Option Description
-v Display the version number.
-help <category> List a description of options belonging to a specific <category>. Specify -help help to see a list of available categories.
-std=<val> Conform to a specific language standard <val>. (C/C++ only)
-dryrun Driver tool commands will be shown but not executed.
-E Preprocessed source files including #line directives are output to stdout.
-EP Inhibit generation of #line directives in the output from the preprocessing.
-static Link libraries statically. Dynamic linking is the default.
-integer-size <size> Specifies the default KIND for integer and logical variables. The default <size> is 32 for 4 bytes long types. Specify 16 for 2 bytes or 64 for 8 bytes long types.
(Fortran only)
-real-size <size> Specifies the default KIND for real and complex declarations. The default <size> is 32 for 4 bytes long types. Specify 64 for 8 bytes or 128 for 16 bytes long types.
This option will not effect the default size of DOUBLE PRECISION variables. (Fortran only)
-double-size <size> Specifies the default KIND for DOUBLE PRECISION and DOUBLE COMPLEX variables. The default <size> is 64 for 8 bytes long types. Specify 128 for 16 bytes
long types. (Fortran only)
-fp-model <keyword> Controls the semantics of floating-point calculations. See the compiler man page for possible values of <keyword>.


For optimising codes using the Intel compilers see Intel Optimisation Flags, and for debugging see Section 5.1, “Compiler Debug Flags and Environment Variables”. Portland

The Portland Group (PGI) compiler suite is a commercial software product containing compilers for Fortran and C/C++. The PGI compilers offer full support for:

  • Fortran 2003
  • ANSI C99 with K&R extensions
  • ISO/ANSI and GNU standards for C++
  • OpenMP 3.0

In addition to the compiler options listed in Table 4, “Common Compiler Options” the table below lists a set of selected options available fo
r the PGI compilers.

Table 7. General Portland Compiler Options

Option Description
-V Display compiler version information.
-help=<category> Display options belonging to <category>. Specify -help=groups to see a list of available categories.
-c89, -c99 Conform to the C89 or C99 language. (C/C++ only)
-dryrun Print commands to run during stages of compilation without executing them.
-E Preprocessed source files including #line directives are output to stdout.
-module dir Save/search for module files in directory dir. (Fortran only)
-Bdynamic Link libraries dynamically. This is default.
-Bstatic Link libraries statically.
-i8 Specifies the default size for integer variables to be 8 bytes. (Fortran only)
-r8 Specifies the default size for real variables to be 8 bytes. This option will not effect the default size of DOUBLE PRECISION variables. (Fortran only)


For optimising codes using the PGI compilers see Portland Optimisation Flags, and for debugging see Section 5.1, “Compiler Debug Flags and Environment Variables”.

Further Information
  • To access the full documentation and users guides of the PGI compilers see the PGI Documentation page. PathScale

The PathScale EKOPath compiler suite includes C, C++, and Fortran compilers for 64-bit Linux based systems offering support for:

  • Features from Fortran 2003 and Fortran 2008
  • ISO C99/C++ 2003, C++11 & GNU compatibility
  • OpenMP 2.5

In addition to the compiler options listed in Table 4, “Common Compiler Options” the table below lists a set of selected options available for the PathScale compilers.

Table 8. General PathScale Compiler Options

Option Description
-version Write compiler release version information to stdout.
-std=<val> Conform to a specific language standard <val>. (C/C++ only)
-### Print commands to run during stages of compilation without executing them.
-E Preprocessed source files including #line directives are output to stdout.
-E -P Inhibit generation of #line directives in the output from the preprocessing.
-module dir Save/search for module files in directory dir. (Fortran only)
-static Use static linking of libraries. Dynamic linking is the default.
-i8 Specifies the default size for integer variables to be 8 bytes. (Fortran only)
-r8 Specifies the default size for real variables to be 8 bytes. (Fortran only)


For optimising codes using the PathScale compilers see PathScale Optimisation Flags, and for debugging see Section 5.1, “Compiler Debug Flags and Environment Variables”.

Further Information

3.3. MPI

MPI (Message-Passing Interface) is a widely used standard for the message-passing parallel programming model used on distributed memory systems. MPI is a library specification providing a set of routines for point-to-point and collective commun
ication. In addition MPI includes functionalty for other types of operations, e.g. remote-memory access and parallel I/O.

MPI is a specification, not an implementation. There are multiple implementations of MPI available, open source and vendor-specific. The next sections provide an overview of some of the more widely used implementations.

3.3.1. MVAPICH2

MVAPICH2 is an open-source MPI-3 implementation and is developed by the Network-Based Computing Laboratory (NBCL) at the Ohio State University.

Compiling and Running Applications

The compiler wrappers listed in the table below are used to compile and link programs providing the necessary libraries.

Table 9. MVAPICH2 Compiler Wrappers

Language Wrapper script Environment variable Command line
C mpicc MPICH_CC -cc=<compiler>
C++ mpicxx, mpic++ MPICH_CXX -cxx=<compiler>
Fortran 77 mpif77 MPICH_F77 -f77=<compiler>
Fortran 90 mpif90 MPICH_F90 -f90=<compiler>


Specify option -show to see underlying compilers used, i.e. the compilers used to build the MVAPICH2 installation, and necessary paths and libraries without actually running the commands. It is possible to select a different compiler using one of the environment variables or command line options listed above. This should only be done if the compiler is compatible with the MVAPICH2 library.

To launch MVAPICH2 jobs use the mpirun_rsh command:

$ mpirun_rsh -np <N> -hostfile hfile ./mpi_prog

specifying the total number of processes <N> and the name of the hostfile hfile containing hosts, one per line.

Alternatively, MVAPICH2 also provides the Hydra process manager from MPICH2. Hydra can be used issuing the mpirun or mpiexec (they are both symbolic links to mpiexec.hydra) job launcher commands:

$ mpirun -np <N> -f hfile ./mpi_prog

When running under the PBS environment the Hydra mpirun/mpiexec commands will automatically use the PBS_NODEFILE environment variable to find the hosts to run on, i.e. specifying the -f hfile option is not necessary.

Add the -help flag to show available options to the job launcher commands.

MVAPICH2 supports architecture specific CPU mapping through the Portable Hardware Locality (hwloc) software package. CPU affinity is enabled by default. To see a description on how process placement and other features of MVAPICH2 can be tuned at runtime, see Section 4.2.1, “MPI Runtime Environment Variables”.

Further Information

3.3.2. Open MPI

Open MPI is an open source implementation of MPI and started as a merge of other implementations, FT-MPI from the University of Tennessee, LA-MPI from Los Alamos National Laboratory, LAM/MPI from Indiana University, and PACX-MPI from the University of Stuttgart. Open MPI includes all MPI 1.2-compliant and MPI 2-compliant routines.

Compiling and Running Applications

Load the proper module file to add the paths for Open MPI executables and libraries to your environment. The Open MPI’s wrapper compilers listed in the table below add in all the relevant compiler/linker flags and then invoke the underlying compiler, i.e. the compiler the Open MPI installation was built with. Use the option -showme to see the underlying compiler, the compile and link flags, and the libraries that are linked.

Table 10. Open MPI Compiler Wrappers

Language Wrapper script Environment variable
C mpicc OMPI_CC
C++ mpiCC, mpicxx, mpic++ OMPI_CXX
Fortran 77 mpif77 OMPI_F77
Fortran 90 mpif90 OMPI_FC


If necessary, it is possible to change the underlying compiler invoked when calling the compiler wrappers using the environment variables listed in the table. E.g. if you want link with an Intel compiler built Open MPI installation, combining Fortran and C code using ifort and gcc you can specify

$ OMPI_CC=gcc mpicc -c cfile.c
$ mpif90 cfile.o ffile.f90

Open MPI provides the commands mpirun and mpiexec to launch MPI jobs. The two commands are identical, they are symbolic links the back-end launcher command orterun. To see all options available to mpirun issue command

$ mpirun -h

The following lists some of the most important options:

-np <N>
Specify the number <N> of processes to run.
-hostfile <hfile>
Provide a hostfile to use. Notice, for Open MPI installations that include support for some batch systems, e.g. PBS Pro and LoadLeveler, there is no need to specify the -hostfile option. Open MPI automatically detect the list of hosts to run on.
Assign processes round-robin by slot (the default).
Assign processes round-robin by node.
Assign processes round-robin by socket.
Uniform distribution of ranks across all nodes.
Bind processes to cores.
Bind processes to processor sockets.
Report any bindings for launched processes.
-stdin <rank>
The MPI rank that is to receive stdin. The default is to forward stdin to rank 0.
-x <VAR>
Export environment variable <VAR> to remote nodes.
Specify values to MCA parameters.

The default scheduling policy of Open MPI is by slot, i.e. processes are scheduled on a node until all of its default slots are exhausted before proceeding to the next node. This can be changed specifying one of the mapping options listed above.

For example, to run 8 mpi processes on two 2-socket 8-core nodes (node0 and node1) mapping to sockets in a round-robin fashion and binding to cores, use the following command:

$ mpirun -np 8 -bysocket -bind-to-core -report-bindings ./mpi_prog

[node0:..] MCW rank 0 bound to socket 0[core 0]: [B . . . . . . .][. . . . . . . .]
[node0:..] MCW rank 1 bound to socket 1[core 0]: [. . . . . . . .][B . . . . . . .]
[node0:..] MCW rank 2 bound to socket 0[core 1]: [. B . . . . . .][. . . . . . . .]
[node0:..] MCW rank 3 bound to socket 1[core 1]: [. . . . . . . .][. B . . . . . .]
[node1:..] MCW rank 4 bound to socket 0[core 0]: [B . . . . . . .][. . . . . . . .]
[node1:..] MCW rank 5 bound to socket 1[core 0]: [. . . . . . . .][B . . . . . . .]
[node1:..] MCW rank 6 bound to socket 0[core 1]: [. B . . . . . .][. . . . . . . .]
[node1:..] MCW rank 7 bound to socket 1[core 1]: [. . . . . . . .][. B . . . . . .]

As can be seen from the output above using the -report-bindings option, the job will run 2 MPI processes on each socket binding to core id’s 0 and 1.

Runtime tuning can also be done setting global MCA parameters. They can be specified as options to the mpirun command or as environment variables with prefix ’OMPI_MCA_’. For example, to show all MCA parameter values during MPI_INIT specify use the -mca option:

$ mpirun -np 8 -mca mpi_show_mca_params all ./mpi_prog

or using the environment variable:

$ export OMPI_MCA_mpi_show_mca_params=all
$ mpirun -np 8 ./mpi_prog

Command ompi_info displays information about the Open MPI installation. To see all MCA paramters specify

$ ompi_info -param all all

See Section 4.2.1, “MPI Runtime Environment Variables” for a description of a subset of variables supported by Open MPI.

Further Information
  • For more information using Open MPI see the online FAQ page.

3.3.3. Intel MPI

The Intel MPI library is based on MPICH from Argonne National Laboratory and MVAPICH2 from Ohio State University. Intel MPI implements the MPI 2.2 specification on multiple fabrics.

Compiling and Running Applications

The following table shows available Intel MPI compiler commands, the underlying Intel and GNU compilers, and ways to override underlying compilers with environment variables or command line options:

Table 11. Intel MPI Compiler Wrappers

Language Wrapper script Default compiler Environment variable Command line
C mpiicc icc I_MPI_CC -cc=<compiler>
mpigcc gcc
C++ mpiicpc icpc I_MPI_CXX -cxx=<compiler>
mpigxx g++
Fortran 77 / Fortran 95 mpiifort ifort I_MPI_FC -fc=<compiler>
mpifc gfortran


Specify option -show with one of the compiler wrapper scripts to see the underlying compiler together with compiler options, link flags and libraries.

Use the mpirun command to launch programs linked with Intel MPI. The default underlying process manager using mpirun is Hydra. (Alternatively MPD can be selected as process manager, users are referred to the Intel documentation for more information on process managers.)

A list of selected options to mpirun:

-version, -V
Display the version of the Intel MPI Library.
-np <N>
Specify the number <N> of processes to run
-hostfile <hfile>
Specify a hostfile with hosts to run the job on. If a job is started within a batch system, e.g. PBS Pro, LSF, or LoadLeveler, mpirun will automatically extract the hostlist to run on, i.e. this option is not needed.
Prefix lines written to standard output with the MPI process rank number.
-s <rank>
Specify the MPI rank to receive stdin. Default rank is 0.
-genv <VAR> <value>
Set the <VAR> environment variable to the given <value> for all MPI processes.
Propagate all environment variables to all MPI processes.

See Section 4.2.1, “MPI Runtime Environment Variables” for a list of environment variables used to influence program behavior and performance at run time, and Section 6.1, “MPI Statistics” for varibles used for collecting performance statistics of MPI communication operations.

Further Information

3.3.4. Platform MPI

IBM Platform MPI is based on MPICH from Argonne National Laboratory and LAM from the University of Notre Dame and Ohio Supercomputer Center. The implementation fully complies with the MPI 2.2 standard and includes ROMIO, a portable implementation of MPI I/O. It also ships with a prebuilt MPE (MPI Parallel Environment) profiling tool.

Compiling and Running Applications

Load the Platform MPI module file, make sure that the MPI_ROOT environment variable points to the location where Platform MPI is installed. Compiler wrappers are provided
to set up the path for include and library directories, and libraries to link. The underlying compiler used is specified using environment variables or as command line options to the compile wrappers, see the table below.

Table 12. Platform MPI Compiler Wrappers

Language Wrapper script Environment variable Command line
C mpicc MPI_CC -mpicc <compiler>
C++ mpiCC MPI_CXX -mpiCC <compiler>
Fortran 77 mpif77 MPI_F77 -mpif77 <compiler>
Fortran 90 mpif90 MPI_F90 -mpif90 <compiler>


For example, to compile a C program using the PGI compiler:

$ mpicc -mpicc pgcc prog.c

or alternatively specify the compiler environment variable first:

$ export MPI_CC=pgcc
$ mpicc prog.c

To compile code and link against Platform MPI without utilizing the wrapper tools specify the MPI library using the -l option when linking. E.g. using the PGI compiler:

$ pgcc prog.c -lpcmpi64

The above command assumes that the proper paths for Platform MPI include and library directories have been added to the environment.

Use the supplied Platform MPI launch utility mpirun when running applications. Listed below is a description of some of the options to mpirun:

Prints usage information for mpirun.
-np <N>
Specifies the number <N> of processes to run.
-hostfile <hfile>
Launches an executable across multiple hosts listed in file <hfile>.
Prepend the global rank of the originating process to stdout and stderr output.
Broadcasts standard input to all MPI processes.
Use shared memory for small messages, the default size is 256 KB. For larger messages, the interconnect is used for better bandwidth. The default is to use shared memory (-intra=shm) for all intrahost data transfers.
Prints user and system times for each MPI rank.
-e var[=val]
Sets the environment variable var for the program and gives it the value val if provided.
Debug mode. Prints additional information about application launch.

Platform MPI includes a set of environment variables that affect the behavior at run time. See Section 4.2.1, “MPI Runtime Environment Variables” for a description of a subset of these variables. For a complete list of environment variables see the man mpienv page.

3.4. OpenMP

The OpenMP API supports shared-memory parallel programming in C/C++ and Fortran. OpenMP comprises a set of compiler directives, library routines, and environment variables used for writing parallel programs that are portable across different shared-memory architectures. A number of compilers, including the ones covered in this guide, implement the OpenMP API.

3.4.1. Compiler Flags

To enable the compiler to generate multithreaded code based on the OpenMP directives in the code add the option listed in the table below

Table 13. OpenMP Compiler Flags

Compiler Option
GCC -fopenmp
Intel -openmp
Portland -mp
PathScale -mp, -openmp


The OpenMP specification defines several environment variables that control the execution of OpenMP programs, see Section 4.2.2, “OpenMP Runtime Environment Variables” for a description.

3.5. Hybrid Programming

Combining MPI and OpenMP in a parallel application can help to utilize shared memory within a multicore node more efficiently, thus avoiding the need of costly intra-node communication. The typical scenario in this model is for MPI to handle communication operations among nodes, and OpenMP to manage concurrency within each node. This can help application performance in the following ways:

  • The overhead due to intra-node communication (e.g., message packing/unpacking) is mitigated, since OpenMP threads use directly shared memory for data exchange.
  • Global synchronization can be more efficient because it is performed in a group-wise and hierarchical fashion (i.e., first, OpenMP threads within each thread group, and then, MPI processes on behalf of each thread group).
  • The total memory footprint of the original MPI application is reduced, due to the elimination of message buffers and other internal data structures. This enables fitting larger working sets in the same HPC system.
  • Inter-node communication can be improved because there is less pressure put on the network. Since the number of MPI processes per node is reduced, the total number of communication channels and the number of messages are also reduced, eliminating large portion of the startup communication cost. In most cases, it is generally preferable to combine communication data into fewer but larger messages, than the opposite.
  • Applications that suffer from load imbalance can benefit from OpenMP’s ability to schedule work in a dynamic fashion. MPI does not provide similar constructs that would enable dynamic work assignment in an easy way.
  • When scheduled on cores that share some level of cache hierachy (e.g. L2 or L3), threads that work on common or adjacent data can usually execute faster. This is because they exploit shared cache space in a constructive way, either by having effectively larger cache capacity at their disposal, or due to the fact that the data fetched by some threads might be reused by others in the near future.
  • Having a moderate mixture of P MPI processes and T OpenMP threads, as opposed to having a single MPI process and C OpenMP threads
    (where C=PxT is the total number of cores in a node), can help to take advantage of NUMA capabilities more efficiently. The reason is that the P separate address spaces can be naturally mapped to different NUMA domains, which will provide uncontended access to main memory between the MPI processes.

The typical execution scenario of a hybrid MPI-OpenMP application is the following:

  • P MPI processes are spawned on each node in the system
  • each process spawns T OpenMP threads
  • at regular periods, MPI processes communicate with each other to exchange data
  • threads in the team belonging to each process start processing the data until the next global synchronization point

This scenario is implemented in the following sample code:

#include <omp.h>
#include <mpi.h>

int main(int argc, char **argv)
    int nproc, my_rank, curr_step, max_steps;
    float* buf;

    MPI_Init(&argc, &argv);


    while ( curr_step < max_steps ) {
        MPI_Send(&buf, ...); //send data from previous step
        MPI_Recv(&buf, ...); //receive data for current step

        #pragma omp parallel for schedule(runtime)
        for ( i = 0; i < N; i++ ) {
            //process data on curr_buf


    return 0;

To run the application, we may use a bash script like the one that follows. It acts as a wrapper to the MPI program, and additionally initializes certain OpenMP runtime variables before invoking the executable. Therefore, the script itself can be given as argument to the MPI launcher (e.g. mpirun).


export OMP_SCHEDULE="dynamic"

./mpi__omp_prog arg1 arg2 ...

To further tune MPI-OpenMP interaction, the programmer can use appropriate arguments in the MPI_Init_thread function in order to specify the desired level of thread support. These arguments are supported as of MPI-2.1, and are the following:

the process may be multithreaded, but only the main thread will make MPI calls
the process may be multithreaded, and multiple threads may make MPI calls, but only one at a time: MPI calls are not made concurrently from two distinct threads
multiple threads may call MPI, with no restrictions

3.6. Math Libraries

3.6.1. GSL (GNU Scientific Library)

The GNU Scientific Library (GSL) is a free numerical software package for C and C++ covering a range of subject areas including:

  • BLAS (level 1, 2, and 3) and linear algebra routines
  • Fast Fourier transform (FFT) functions
  • Numerical integration
  • Random number generation functions

To use the library make sure that the necessary paths to the GSL installation are added to the environment. The library header files are installed in a subdirectory gsl that must prefix preprocessing include statements, see e.g. program test_blas.c below. The header files define functions to have extern "C" linkage when included in C++ programs allowing functions to be called directly from C++. The GSL library is available in both shared and static versions.

To link against the library specify both the main GSL library and the supporting CBLAS library. For example, compile and link the sample test_blas.c program:


/*  test_blas.c : Example using BLAS matrix multiply code sgemm */

#include <stdio.h>

#ifdef GSL
#include <gsl/gsl_cblas.h>

#elif defined MKL
#include <mkl_cblas.h>

#elif defined ACML
#include <acml.h>

int main(void)
  const int lda=3, ldb=3, ldc=3;
  int m, n, k;
  float alpha, beta;

  float a[] = { 0.11, 0.21, 0.12,
                0.15, 0.13, 0.17,
                0.22, 0.13, 0.23 };

  float b[] = { 1011, 1021, 1018,
                1031, 1012, 1021,
                1022, 1032, 1015 };

  float c[] = { 0.00, 0.00, 0,00,
                0.00, 0.00, 0,00,
                0.00, 0.00, 0,00 };

  m = 3; n = 3; k = 3;

  alpha = 1.0; beta = 0.0;

#ifdef ACML

  sgemm ('N', 'N', m, n, k, alpha, a, lda, b, ldb, beta, c, ldc);

#elif defined MKL || defined GSL

  cblas_sgemm (CblasColMajor, CblasNoTrans, CblasNoTrans,
                m, n, k, alpha, a, lda, b, ldb, beta, c, ldc);


  printf ("[ %g, %g, %gn", c[0], c[1], c[2]);
  printf ("  %g, %g, %gn", c[3], c[4], c[5]);
  printf ("  %g, %g, %g ]n", c[6], c[7], c[8]);

  return 0;

using the GNU C compiler, adding -DGSL to follow the GSL path in the code:

$ gcc -DGSL test_blas.c -lgsl -lgslcblas

It is possible to link with an alternative BLAS library that is conforming to the CBLAS standard (functions prefixed with cblas_ ).

Further Information

3.6.2. MKL (Intel Math Kernel Library)

The Intel Math Kernel Library (Intel MKL) contains highly optimised, extensively multithreaded math routines for different areas of computation. The library includes:

  • BLAS (level 1, 2, and 3) and LAPACK linear algebra routines
  • ScaLAPACK distributed processing routines and BLACS routines for communication
  • Fast Fourier transform (FFT) functions
  • Vectorized math functions
  • Random number generation functions

See the Intel Math Kernel Library Reference Manual for a complete list of all functions included in the library.

Intel MKL is optimised for the latest Intel processors, including processors with multiple cores. The cluster components of the library, ScaLAPACK and Cluster FFT, are built on top of MPI. The library comes with two interface layers, LP64 for 32-bit integers and ILP64 for 64-bit integers.

Dynamic Linking

Since the Intel Compiler version 11.1 the -mkl option is available providing users a quick way to set the environment settings and required libraries to link with MKL:

-mkl or -mkl=parallel   # link with standard multithreaded MKL
-mkl=sequential         # link with sequential version of MKL
-mkl=cluster            # link with MKL cluster (sequential)
                        # components that use Intel MPI

For example, compile and link test_blas.c, adding -DMKL to follow the MKL path in the code:

$ icc -DMKL -mkl test_blas.c

Using the -mkl option the compiler links the application using the LP64 (32-bit integer) interface.

Static Linking

In case of static linking load the proper MKL modulefile. Make sure that the environment variables


are defined. When linking enclose the libraries in grouping symbols. For non-cluster components of MKL specify the following linker options

-Wl,--start-group ${MKLPATH}/libmkl_intel_lp64.a 
${MKLPATH}/<thread_lib> ${MKLPATH}/libmkl_core.a -Wl,--end-group 
<thread_option> -lpthread

and for cluster components of MKL (i.e. ScaLAPACK and Cluster FFT) linking against Intel MPI specify:

-Wl,--start-group ${MKLPATH}/<cluster_lib> 
${MKLPATH}/libmkl_intel_lp64.a ${MKLPATH}/<thread_lib> 
${MKLPATH}/libmkl_blacs_intelmpi_lp64.a -Wl,--end-group 
<thread_option> -lpthread

where you replace above:

sequential library multithreaded library
<thread_lib> libmkl_sequential.a libmkl_intel_thread.a
<thread_option> n/a -openmp
ScaLAPACK cluster FFT
<cluster_lib> libmkl_scalapack_lp64.a libmkl_cdft_core.a

The above specifications link with the LP64 (32-bit integer) interface to the library. In order to link with the ILP64 (64-bit integer) interface replace lp64 with ilp64 in the library names.

For example, to link a Fortran program prog.f with multithreaded ScaLAPACK using the Intel mpiifort compiler wrapper to setup the MPI environment specify:

$ mpiifort -I$(MKLROOT)/include prog.f  
-Wl,--end-group -openmp -lpthread
Further Information

3.6.3. ACML (AMD Core Math Library)

The AMD Core Math Library (ACML) is a set of numerical routines tuned specifically for AMD64 platform processors. ACML supports a variety of compilers, including GCC, Intel and PGI.

ACML consists of the following main components

  • A full implementation of Level 1, 2 and 3 Basic Linear Algebra Subroutines (BLAS)
  • A full suite of Linear Algebra (LAPACK) routines
  • A comprehensive suite of Fast Fourier Transforms (FFTs) in both single-, double-, single-complex and double-complex data types
  • Random Number Generators in both single- and double-precision
  • Optimised versions of most libm functions, including scalar, vector, and array versions

The routines are available via both FORTRAN 77 and C interfaces and multithreaded versions are available for the Level 3 BLAS, many LAPACK routines, and the 2D and 3D FFTs. The library is supplied in both static and shared versions, libacml.a and, respectively.

The ACML installation includes versions of the library for different compilers. To link with the library specify the path to the proper library on the link command line. Set the path to the ACML installation directory, e.g.

$ export ACMLDIR=/opt/acml5.1.0

This may already have been done when loading the ACML modulefile. Check your environment to see the actual name of variable pointing to the ACML installation directory. Add the library subdirectory with the -L option based upon the compiler used when linking the code


For example, to compile and link test_blas.c using the PGI C compiler, adding -DACML to follow the ACML path in the code:

$ pgcc -DACML test_blas.c -L$ACMLDIR/pgi64/lib -lacml

For example, using the GNU Fortran compiler, linking against the multithreaded version of ACML:

$ gfortran -fopenmp prog.f -L$ACMLDIR/gfortran64_mp/lib -lacml_mp
Further Information

3.6.4. FFTW

FFTW is a library developed at Massachusetts Institute of Technology (MIT) for computing the Discrete Fourier Transform (DFT) in one or more dimensions, of both real and complex data. The library supports serial transforms, multi-threaded transforms, and distributed-memory parallel transforms using MPI. FFTW include interfaces for both C and Fortran, and support single, double and long-double precision.

All programs using FFTW must include the header file:

#include <fftw3.h>

This is the same header file regardless of the precision of FFTW used. If using a different version than the (default) double precision version, instances of lowercase 'fftw_' in the source code should be replaced with 'fftwf_' for single precision or 'fftwl_' for long-double precision.

Depending on the installation, FFTW will include serial and parallel libraries for different precisions listed in the table below:

Precision Serial Threads (POSIX) OpenMP MPI
single libfftw3f.a libfftw3f_threads.a libfftw3f_omp.a libfftw3f_mpi.a
double libfftw3.a libfftw3_threads.a libfftw3_omp.a libfftw3_mpi.a
long double libfftw3l.a libfftw3l_threads.a libfftw3l_omp.a libfftw3l_mpi.a

(Installations built with gcc may also include a quadruple-precision version of FFTW.)

The basic usage of FFTW is to link with the serial FFTW library, e.g. in double precision, -lfftw3. When using one of the parallel versions of FFTW in addition add the appropriate library from the table.

For example, to link a C program using OpenMP transforms with the double precision FFTW version using the Intel compiler specify:

$ icc -openmp test_fft.c -lfftw3_omp -lfftw3
Further Information

4. Tuning

This chapter describes how to improve application performance using compiler optimisation flags for single processor tuning, and environment variables used to control MPI and OpenMP behavior at runtime.

4.1. Optimisation

4.1.1. IEEE 754 Compliant Optimisation

The IEEE 754 standard specifies formats and methods for floating-point computations. The standard specifies number formats, basic operations, conversions, and floating-point exceptions and their handling.

Adhering to the IEEE 754 standard means that an application will generate exactly the same data regardless of the target processor. Because compiler optimisations are performed at the possible expense of precision most compilers will not generate full IEEE 754 compliant code unless this is specified explicitly by compiler flags.

See the table below for compiler flags that will produce IEEE 754 compliance for floating-point arithmetic.

Table 14. IEEE 754 Compliant Optimisation Flags

Compiler Option
GCC -frounding-math
Intel -fp-model source
Portland -Kieee
PathScale -OPT:IEEE_arithmetic=1


4.1.2. Compiler Optimisation Flags

The following sections list a number of optimisation flags available for the different compilers, and discuss some general optimisation recommendations.


Table 15. GCC Optimisation Flags

Option Description
--help=optimizers Display all optimisation options.
-Q --help=optimizers Adding -Q with the option above lists optimisations that are enabled at each level.
-mtune=<cpu-type> Produce code optimised for the selcted processor type. See the compiler man page for choices for cpu-type.
-march=<cpu-type> Generate instructions for the selected processor type. Specifying -march=native will tune for the architecture of the compiling machine, and implies -mtune=native.
-m<target> Enable the use of extended intruction sets, where <target> is e.g. avx, sse4.2, 3dnow, etc.
-O0,-O1,-O2,-O3 Specify the level of increasingly aggressive optimisations. Default: -O0
-Ofast Combines all -O3 optimisations with options that disregard strict standards compliance but can result in better optimised code. Turns on -ffast-math, see below.
-Os Optimise for size. Includes -O2 optimisations that do not increase code size.
-ffast-math Turns on optimisations that may violate IEEE standards but can give faster code. Enables flushing of denormal results to zero.
-finline-functions Consider all functions for inlining.
-funroll-loops Perform loop unrolling when iteration count is known.


Notice that the default optimisation level is -O0, i.e. most optimisations are disabled. It is recommended to start optimising with the -O2 flag, since this will provide maximum optimisation without increasing the executable size, and in addition to add-march to produce architecture tuned code, i.e.

$ gcc -march=native -O2 prog.c

Table 16. Intel Optimisation Flags

Option Description
-help opt Display optimisation options.
-help advanced Display advanced optimisation options that allow fine tuning of compilation.
-m<code> Generate specialized code for processors specified by <code>. The executable generated can run on any compatible (Intel and non-Intel) processor
with support for the corresponding instruction set.
-x<code> Generate specialized code for processors specified by <code>. The executable generated can only run on Intel processors with support for the corresponding
instruction set. Specifying -xHost will generate instructions for the highest instruction set available on the compiling machine.
-ax<code> Generate specialized code with multiple code paths for Intel processors based on <code>. The most suitable code path for the processor the application
is running on is chosen at execution time. The default optimised code path can be modified by the -m or -x switches described above.
-O0,-O1,-O2,-O3 Specify the level of increasingly aggressive optimisations. Default: -O2
-fast Maximize speed across the entire program.
-Os Enable optimisations that do not increase code size and produces smaller code size than O2.
-no-prec-div Optimise floating-point division that may not be full IEEE compliant. This option is enabled with -fast. The default is -prec-div, i.e. fully precise IEEE division.
-unroll-aggressive Use more aggressive unrolling for certain loops.
-ipo Enable interprocedural optimisation between files.
-ftz Flush denormal results to zero.
-parallel Generate multithreaded code for loops that can be safely executed in parallel.
-mkl Link to certain parts of MKL, see also Section 3.6.2, “MKL (Intel Math Kernel Library)”.
-fno-exceptions Disable exception handling support. (C++ only)


O2 is the generally recommended optimisation level. The O3 option is recommended for applications that have loops that heavily use floating-point calculations and process large data sets unless loop and memory access transformations take place, O3 optimisations may slow down code in some cases compared to O2 optimisations.

The recommended starting point for optimisation generating optimal instructions for the compiling machine is, e.g. Fortran code

$ ifort -xHost -O2 prog.f90

Table 17. Portland Optimisation Flags

Option Description
-help=opt Print help for optimisation flags.
-Minfo List information about optimisation made during compilation. See the compiler man page for suboptions.
-tp <target> Sets the target architecture. The default is to produce code targeted for the processor of the compiling system.
-O0,-O1,-O2,-O3,-O4 Specify the level of increasingly aggressive optimisations. Default: -O1
-fast Choose generally optimal flags for the target platform, including SSE/SSE2 instructions. (See description below)
-Mflushz Flush denormal results from floating-point calculations to zero.
-Munroll Invoke the loop unroller. See the compiler man page for suboptions.
-Mvect=simd[128|256] Use SSE (128 bit) or AVX (256 bit) SIMD instructions. See the compiler man page for other suboptions to -Mvect
-Mipa Choose generally optimal flags for InterProcedural Analysis (IPA) for the target platform.
-Minline Pass options to the function inliner.
-Mconcur Enable auto-parallelizing of loops. Use with -Minfo to see information about code beeing parallelized.
--no_exceptions Disable exception handling support. (C++ only)


As suggested in the PGI Compiler User’s Guide (PDF) when optimising it is recommended to start with the option -fast. This is an aggregate option that incorporates a generally optimal set of flags for targets that support SSE capability. The option combines optimisation options to enable use of vector streaming SIMD instructions for 64-bit targets. It enables vectorization with SSE instructions, cache alignment, and flushes to zero. Use the -help switch to see the flags included with the -fast option:

$ pgf90 -fast -help

The contents of the –fast option are host-dependent and typically include the following:

Specifies a code optimisation level of 2.
Unrolls loops, executing multiple instances of the loop during each iteration.
Indicates to not generate code to set up a stack frame.
Indicates loop-carried redundancy elimination.
Generates SSE instructions.
Generates scalar SSE code with xmm registers.
Aligns long objects on cache-line boundaries.
Flush denormal results to zero.
Indicates partial redundancy elimination.

The recommended optimisation flags for Fortran and C programs are:

$ pgf95 -fast -Mipa=fast prog.f90
$ pgcc -fast -Mipa=fast prog.c

and for C++ programs:

$ pgCC -fast -Mipa=fast -Minline=levels:10 --no_exceptions

Table 18. PathScale Optimisation Flags

Option Description
-LIST:options=ON List options enabled at a given optimisation level written to a listing (.lst) file.
-march=<cpu-type> Compiler will optimise code for the selected CPU type. The default is auto which means to optimise for the host platform that the compiler is running on.
-m<target> Enable the use of extended intruction sets, where <target> is e.g. avx, sse4_2, 3dnow, etc.
-OPT: ... This option group controls miscellaneous optimisations. See the man eko page for a complete list of available suboptions.
-LNO: ... This group specifies options and transformations performed on loop nests. See the man eko page for a complete list of available suboptions. The -LNO options are enabled only if the optimisation level of -O3 or higher is in effect.
-IPA ... The InterProcedural Analysis option group. See the man eko page for a complete list of available suboptions.
-ipa InterProcedural Analysis with default IPA suboptions used.
-O0,-O1,-O2,-O3 Specify the level of increasingly aggressive optimisations. Default: -O2
-Ofast Use optimisations selected to maximize performance.
-inline Request inline processing.
-OPT:IEEE_arithmetic=<N> Specify the level of floating pointing roundoff/overflow behavior. Setting <N>=1 will adhere to IEEE accuracy (see Section 4.1.1, “IEEE 754 Compliant Optimisation”). Setting <N> equal to either 2 or 3 will enable flushing of denormal numbers to zero.
-apo Enable auto-parallelizing of code.
-fno-exceptions Disables exception handling. The default is to enable exception handling. (C++ only)


When tuning codes it is recommended in the EKOPath User Guide to try the following set of options:

  • -O2
  • -O3
  • -O3 -OPT:Ofast
  • -Ofast

listed in the order of increasing optimisation level. The -fast option is equivalent to

Turn on aggressive optimisation.
Invoke inter-procedural analysis (IPA).
Tells the compiler to assume that the program does not test errno after calls to math library functions.
Improves execution speed by relaxing ANSI and IEEE rules.
Departure from source language floating-point, round-off, and overflow semantics. Allows extensive transformations, such as the reordering of reduction loops.
Optimise code regardless of program size.
Enable changing x/y into x*(recip(y))
Assume that the code adheres to the ANSI/ISO C standard which states that two pointers of different types cannot point to the same location in memory.

Although the optimisations listed are generally safe, they may affect floating point accuracy due to rearrangement of computations.

4.2. Runtime Environment Settings

4.2.1. MPI Runtime Environment Variables

This section describes environment variables that can be used to modify the behavior of MPI at run time. Default settings are based on the best performance for most applications but some codes may benefit from adjusting the values of variables.

Below is a list of environment variables for different MPI implementations.

    Enable (1) or disable (0) CPU affinity. The default is CPU affinity enabled. For multi-threaded programs, e.g. MPI/OpenMP, having CPU affinity enabled all threads may be scheduled to the same CPU. To avoid this MV2_ENABLE_AFFINITY should be disabled, i.e. set to 0.

    Shows the current process to core bindings of all processes
    on the node which contains the mpi rank 0 process written to stderr.

    Set the mapping of process to core policy. Specifying bunch processes are assigned in a block fashion, e.g. on a 2-socket node processes will map to the first socket until all cores are utilized before moving to the second socket. If scatter is set processes are mapped in a round-robin fashion.

    Specify a user defined mapping to bind processes to cores.

    For example, to run 4 processes on a 2-socket 8-core node binding to core id’s 0,1 (on the first socket) and 8,9 (on the second socket):

    $ export MV2_CPU_MAPPING=0:1:8:9
    $ export MV2_SHOW_CPU_BINDING=1
    $ mpirun -np 4 ./mpi_prog
    -------------CPU AFFINITY----------------
    RANK:0  CPU_SET:   0
    RANK:1  CPU_SET:   1
    RANK:2  CPU_SET:   8
    RANK:3  CPU_SET:   9
    Specify the process to core binding level. The default (core) is to bind processes to spesific cores.

    For example, running 4 processes on a 2-socket 8-core node binding to sockets in a round-robin fashion:

    $ export MV2_CPU_BINDING_LEVEL=socket
    $ export MV2_CPU_BINDING_POLICY=scatter
    $ export MV2_SHOW_CPU_BINDING=1
    $ mpirun -np 4 ./mpi_prog
    -------------CPU AFFINITY----------------
    RANK:0  CPU_SET:   0  1  2  3  4  5  6  7
    RANK:1  CPU_SET:   8  9 10 11 12 13 14 15
    RANK:2  CPU_SET:   0  1  2  3  4  5  6  7
    RANK:3  CPU_SET:   8  9 10 11 12 13 14 15

    Processes can migrate between the 8 cores on the socket.

    Specify the message size for switching from the eager to the rendezvous protocol for point-to-point communication. For better performance, the value of MV2_IBA_EAGER_THRESHOLD should be set to the same as MV2_VBUF_TOTAL_SIZE.

    Specify the message size for switching from the eager to the rendezvous protocol for intra-node communication.
    The size of the basic basic communication buffer of MVAPICH2. For better performance, the value of MV2_IBA_EAGER_THRESHOLD should be set the same as MV2_VBUF_TOTAL_SIZE.

  • Open MPI

    The specifications listed in this section can alternatively also be specified as options to the Open MPI mpirun job launcher. For additional runtime tuning, e.g. process binding, see the Open MPI section.

    Specify the size of messages that is sent using the eager protocol for intra-node communications.
    Specify the size of messages that is sent using the eager protocol for inter-node communications.
    Setup a fully connected topology during MPI_INIT. This might be beneficial for communication intensive applications.

  • Intel MPI
    Set the intended optimisation platform. See the Intel MPI documentation for the list of possible <platform> specifications.

    Specify the default placement of processes on nodes. Possible selections of <value> :

    N Place N processes per node.
    all All logical CPUs on a node.
    allcores All cores (physical CPUs) on a node.
    Turn on/off process pinning. The default is to enable process pinning.
    Define the processor placement. See the Intel MPI documentation for a detailed description of available spesifications. To print pinning information set the I_MPI_DEBUG environment variable equal to 4.

    For example, to run 8 mpi processes on two 2-socket 8-core nodes (node0 and node1) pinning to core id’s 0,1 (on the first socket) and 8,9 (on the second socket) on each node, specify:

    $ export I_MPI_PIN_PROCESSOR_LIST='grain=2,shift=4'
    $ export I_MPI_DEBUG=4
    $ mpirun -n 8 ./mpi_prog
    [0] MPI startup(): Rank    Pid      Node name    Pin cpu
    [0] MPI startup(): 0       21075    node0           0
    [0] MPI startup(): 1       21076    node0           1
    [0] MPI startup(): 2       21077    node0           8
    [0] MPI startup(): 3       21078    node0           9
    [0] MPI startup(): 4       24762    node1           0
    [0] MPI startup(): 5       24763    node1           1
    [0] MPI startup(): 6       24764    node1           8
    [0] MPI startup(): 7       24765    node1           9
    Define a number of non-overlapping subsets (domains) of logical processors on a node and start 1 MPI process per domain. Specifying this variable is useful when running hybrid MPI/Open MP applications. Threads started within a domain can freely migrate between logical processors in the domain.

    For example, to run 4 MPI processes on a 16 core node with 4 Open MP threads per MPI process, use:

    $ export OMP_NUM_THREADS=4
    $ export I_MPI_PIN_DOMAIN=omp
    $ export I_MPI_DEBUG=4
    $ mpirun -n 4 ./mpi_omp_prog
    [0] MPI startup(): Rank    Pid      Node name    Pin cpu
    [0] MPI startup(): 0       63703    node0        {0,1,2,3}
    [0] MPI startup(): 1       63704    node0        {4,5,6,7}
    [0] MPI startup(): 2       63705    node0        {8,9,10,11}
    [0] MPI startup(): 3       63706    node0        {12,13,14,15}

    See the Intel MPI documentation for a description of available spesifications for I_MPI_PIN_DOMAIN.

    Print out debugging information when an MPI program starts running.
    Set the size of stdout/stderr output buffer. The default size is 1 KB.

    Specify the eager/rendezvous protocol message size for point-to-point communication. Default size is 256 KB.

    Specify the eager/rendezvous protocol message size for intra-node communication. Default size is 256 KB.

    Turn on/off the intranode direct copy communication mode. Messages shorter than the specified I_MPI_INTRANODE_EAGER_THRESHOLD size are transferred using shared memory, and messages larger are transferred through the direct process memory access.

    Turn on/off the dynamic connection establishment. If disabled a fully connected topology is established upfront. This is the default for less than 64 processes.
  • Platform MPI
    This variable can be used for specifying additional command-line arguments to the mpirun command.

    This variable modifies the general behavior of Platform MPI. The syntax is a comma-separated list of flags.

    For example, turn on function parameter error checking, specify:

    $ export MPI_FLAGS=Eon

    For example, to enable zero-buffering mode, specify:

    $ export MPI_FLAGS=z

    This can be used to verify that the application is MPI-safe, i.e. not dependent on a particular implementation of MPI with spesific default settings for buffer sizes.

    Bind ranks to logical processors according to a given binding strategy. E.g. schedule ranks on CPUs according to packed rank ID:
    $ export MPI_CPU_AFFINITY=rank
    Allows specification of the integer CPU numbers when binding ranks to CPUs. E.g. map 8 MPI processes according to the given logical numbering of CPUs :
    $ export MPI_BIND_MAP=0,1,4,5,2,3,6,7
    $ export MPI_CPU_AFFINITY=map_cpu
    Specifies message protocol length.

4.2.2. OpenMP Runtime Environment Variables

The OpenMP specification defines a set of environment variables that affect several parameters of application execution during runtime. These variables are:

Sets the maximum number of threads to use in parallel regions if no other value is specified in the application. As of version 3.1 of the OpenMP specification, the environment variable takes a comma-separated number list that specifies the number of threads at each nesting level.
OMP_SCHEDULE type[,chunk_size]
Sets the iteration scheduling policy for parallel loops to one of static, dynamic, guided or auto types, and optionally defines their chunk size.

Enables or disables the dynamic adjustment of threads the runtime system uses for parallel regions. Valid values for dynamic are true or false. When enabled, the runtime system adjusts the number of threads so that it makes the most efficient use of system resources. This is meaningful when the user has not specified the desired number of threads globally or on a per-region basis.

Enables or disables nested parallelism, i.e. whether OpenMP threads are allowed to create new thread teams. Valid values for nested are true or false.

Sets the maximum number of nested active parallel regions.
Sets the maximum number of threads participating in the OpenMP program. When the number of threads in the program is not specified (e.g. with the OMP_NUM_THREADS variable), the number of threads used is the minimum between this variable and the total number of CPUs.

Defines the behavior of threads waiting on synchronization events. When policy is set to ACTIVE, threads consume processor cycles while waiting (“busy-waiting”). When it is set to PASSIVE, threads do not consume CPU power but may require more time to resume.

Sets the default thread stack size in kilobytes, unless the number is suffixed by B, K, M or G (bytes, KB, MB, GB, respectively).

Specifies whether threads may be moved between processors. If set to true, OpenMP threads should not be moved. If set to false, the operating system may move them.

The following table shows the default values for the aforementioned environment variables for different compilers.

Table 19. OpenMP Variables, Default Values

Environment Variable GCC Intel PathScale Portland
OMP_NUM_THREADS Number of processors visible to the operating system 1
OMP_SCHEDULE dynamic, chunk size = 1 static, no chunk size specified
OMP_DYNAMIC false (not supported)
OMP_NESTED false (not supported)
OMP_MAX_ACTIVE_LEVELS unlimited (undocumented) (not supported)
OMP_THREAD_LIMIT unlimited (undocumented) 64
OMP_WAIT_POLICY (undocumented) PASSIVE (undocumented) ACTIVE
OMP_STACKSIZE system dependent 4M (undocumented) 2M
OMP_PROC_BIND (undocumented) false (undocumented)


Besides the environment variables defined in the OpenMP standard, many implementations provide their own environment variables. Below we present briefly the most important ones. The user is referred to the corresponding manuals for a more detailed description of the variables and their possible values.

  • GCC
    Specifies a list of CPUs where OpenMP threads should be bound.
  • Intel
    Sets the time (in milliseconds) that a thread should wait, after completing the execution of a parallel region, before sleeping.
    If enabled secure a specific ordering of the reduction operations for implementing the reduction clause for an OpenMP parallel region. This means, for a given number of threads, in a given parallel region, for a given data set and reduction operation, a floating point reduction done for an OpenMP reduction clause will have a consistent floating point result from run to run, since round-off errors will be identical.
    Selects the method used to determine the number of threads for a parallel region when OMP_DYNAMIC is set to true.

    Selects the OpenMP runtime library execution mode.
    Enables the runtime system to bind OpenMP threads to CPUs.

    For example, to run 16 threads on a node with two 8-core sockets and hyper-threading enabled (i.e. a total of 32 logical cores), scattering threads on physical cores, use:

    $ export OMP_NUM_THREADS=16
    $ export KMP_AFFINITY=scatter,verbose
    $ ./a.out
    KMP_AFFINITY: Initial OS proc set respected:
    KMP_AFFINITY: 2 packages x 8 cores/pkg x 2 threads/core (16 total cores)
    KMP_AFFINITY: Threads may migrate across 1 innermost levels of machine
    KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,16}
    KMP_AFFINITY: Internal thread 1 bound to OS proc set {8,24}
    KMP_AFFINITY: Internal thread 2 bound to OS proc set {1,17}
    KMP_AFFINITY: Internal thread 15 bound to OS proc set {15,31}
  • Portland
    Set to yes or y to bind threads to physical processors. The default is to not bind processes to processors.

    Specifying a list of CPUs for mapping threads to CPUs.
    Specify the thread stack size.
  • PathScale
    Enables the operating system’s affinity mechanism to assign threads to CPUs. The default value is true.

    Specifying a list of CPUs for mapping threads to CPUs.
    Specify the thread stack size.
    Sets the number of times that the spin loops will spin at user-level before falling back to OS schedule/reschedule mechanisms.

5. Debugging

This chapter shows how to help debugging applications using compiler options and runtime environment variables. Also in this chapter is a description on how to use debugging tools like GNU GDB and the parallel GUI debuggers Totalview and DDT.

5.1. Compiler Debug Flags and Environment Variables

The table below shows a list of debugging actions and the corresponding compiler options for different compilers.

Table 20. Compiler Debug Flags

Action Compiler Option
Generate symbolic debugging information GCC -g
Add runtime array bounds checking GCC -fcheck=bounds (Fortran only)
Intel -check bounds (Fortran only)
Portland -Mbounds
PathScale -ffortran-bounds-check (Fortran only)
Check for uninitialized variables GCC -Wuninitialized
Intel -check uninit (Fortran only)
-check=uninit (C/C++ only)
Portland n/a
PathScale -trapuv
Trap floating point exceptions:
  • divide by zero
  • invalid operands
  • floating point overflow
GCC -ffpe-trap=zero,invalid,overflow (Fortran only)
Intel -fpe-all=0 (Fortran only)
-fp-trap-all=common (C/C++ only)
Portland -Ktrap=fp
PathScale -TENV:simd_zmask=OFF (zero-divide)
-TENV:simd_imask=OFF (invalid)
-TENV:simd_omask=OFF (overflow)
Add debug information for runtime traceback GCC -fbacktrace (Fortran only)
Intel -traceback
Portland -Meh_frame[a]
PathScale n/a

[a] To be used with the PGI_TERM environment variable, see below. (The PGI compilers also has a -traceback option, however, this does not currently seem to work.)


Environment Variables

The following list describes a set of compiler environment variables used to control runtime debug information.

  • Intel
    Set this variable to y or Y to force a core dump for severe errors that do not usually cause a core file to be created. (Fortran only)

  • Portland
    Controls the stack traceback functionality. The runtime libraries use the value of PGI_TERM to d
    etermine what action to take when a program abnormally terminates.

    E.g. compile to trap floating point exceptions and link with the -Meh_frame option to create a stacktrace

    $ pgf95 -Ktrap=fp -Meh_frame prog.f90

    and set the PGI_TERM variable as follows:

    $ export PGI_TERM=trace

    before running the executable.

  • PathScale
    Set to YES to abort program execution on first bounds check violation when compiled with the -ffortran-bounds-check option. (Fortran only)

    Dump memory mapping information when a segmentation fault occurs. (Fortran only)

5.2. Debuggers

5.2.1. GNU GDB

GDB, the GNU Project debugger, is a free software debugger that supports several programming languages including C, C++ and Fortran. GDB has a command-line interface and do not contain its own graphical user interface (GUI).

GDB Commands

To begin a debug session compile the code with the option -g to add debugging information and start GDB by running the command gdb adding the executable program as argument:

$ gdb prog

Once inside the GDB environment, indicated by the (gdb) prompt, you can issue commands. A list of selected GDB commands:

– display a list of named classes of commands
– start the program
– attach to a running process outside GDB
– go to the next source line, will step into a function/subroutine
– go to the next source line, function/subroutine calls are executed without stepping into them
– continue executing
– sets breakpoint
– set a watchpoint to stop execution when the value of a variable or an expression changes
– display (default 10) lines of source surrounding the current line
– print value of a variable
– displays a stack frame for each active subroutine
– detach from a process
– exit GDB

Commands can be abbreviated to one or the first few letters of the command name if that abbreviation is unambiguous or in some cases where a single letter is specifically defined for a command. E.g. to start a program:

(gdb) r
Starting program: /path/to/executable/prog

To execute shell commands during the debugging session issue shell in front of the command, e.g.

(gdb) shell ls -l
Attaching to running processes

GDB can attach to already running processes using the attach process-id command. After attaching to a process GDB will stop it from running. This allows you to prepare the debug session using GDB commands, e.g. setting breakpoints or watchpoints. Then use the continue command to let the process continue running.

Although GDB is a serial debugger you can examine parallel programs by attaching to individual processes of the program.

The listing below displays a sample debug session attaching to one of the processes of a running MPI job for examining data. Lines starting with # are comments:

$ gdb

(gdb) # List the processes of the MPI program
(gdb) shell ps -eo pid,comm | grep mpi_prog
14957   mpi_prog
14961   mpi_prog
14962   mpi_prog

(gdb) # Attach to one of the MPI processes
(gdb) attach 14961
Attaching to process 14961
Reading symbols from /path/to/executable/mpi_prog...done.

(gdb) # Set a watchpoint to stop execution when the variable Uc is updated
(gdb) watch Uc
Hardware watchpoint 1: Uc

(gdb) # Continue the execution of the program
(gdb) continue

Hardware watchpoint 1: Uc
Old value = -3.33545399
New value = -2.11184907
POTTEMP::ptemp (ldiad=...etc) at ptemp1.f90:298
298              Vc= dsdx(2,1,ie2)*u0 + dsdx(2,2,ie2)*v0 + dsdx(2,3,ie2)*w0

(gdb) # Set the list command to display 16 lines...
(gdb) set listsize 16
(gdb) # ...and display the source backwards starting 2 lines below the current one
(gdb) list +2
284              do k= 1, 8
285                kp= lnode2(k,ie2)
286                u0= u0 + u12(kp)
287                v0= v0 + u22(kp)
288                w0= w0 + u32(kp)
289                vt= vt + vtef2(kp)
290              enddo
292              u0= 0.125*u0;  v0= 0.125*v0;  w0= 0.125*w0;  vt= 0.125*vt
294     !
295     !----    Contravariant velocity
296     !
297              Uc= dsdx(1,1,ie2)*u0 + dsdx(1,2,ie2)*v0 + dsdx(1,3,ie2)*w0
298              Vc= dsdx(2,1,ie2)*u0 + dsdx(2,2,ie2)*v0 + dsdx(2,3,ie2)*w0
299              Wc= dsdx(3,1,ie2)*u0 + dsdx(3,2,ie2)*v0 + dsdx(3,3,ie2)*w0

(gdb) # Print a 5 element slice of the variable u12
(gdb) print u12(3006:3010)
$1 = (0.0186802763, 0.0188683271, 0.0145201795, 0.00553302653, -0.00918145757)

(gdb) # Release the process from GDB control
(gdb) detach
Detaching from program: /path/to/executable/mpi_prog, process 14961

(gdb) quit
Examining Core Files

Core files can be examined specifying both an executable program and the core file:

 $ gdb prog core

One can also produce a core file from within the GDB session to preserve a snapshot of a program’s state using the command:

 (gdb) generate-core-file
Further Information

5.2.2. Totalview

Totalview is a GUI-based source code debugger from Rogue Wave Software, Inc. It allows for debugging of serial and parallel codes. Program execution is controlled by stepping line by line through the code, setting breakpoints, or by setting watchpoints on variables. It is also efficient for debugging of memory errors and leaks and diagnosis problems like deadlocks.

Totalview comes with the ReplayEngine tool embedded. When debugging a program crash ReplayEngine lets you start from the point of failure and work backward in time to find the error that caused it. Notice, ReplayEngine increases the amount of memory your program uses as it keeps history and state information in memory.

TotalView works with C, C++ and Fortran applications, and supports OpenMP and several MPI implementations including MVAPICH2, Open MPI, Intel MPI and Platform MPI.

Starting Totalview

After compiling your MPI code with the -g flag, start Totalview with your executable, e.g. mpi_prog, by issuing the command:

$ totalview mpi_prog

Three windows, the TotalView Root window, Startup Parameters Dialog Box and the Process Windows, will appear.

The Startup Parameters Dialog Box allows you to specify arguments (command line and environment variables), standard I/O files, MPI implementation and number of MPI tasks, and enabling of ReplayEngine and memory debugging.

Figure 1. Totalview Startup Parameters Dialog Box

Totalview Startup Parameters Dialog Box


Fill in the information that allows TotalView to load your program into the Process Window.

Figure 2. Totalview Process Window

Process Window


You are now ready to start a debugging session doing different actions, e.g.:

  • Click the Step or Next buttons to go through the code statement by statement. For function calls Step goes into the function while Next executes the function.
  • Create a breakpoint by clicking the line number displayed to the left in the Process Window. Click the Go button to run to this line.
  • Monitor a variable’s value by creating a watchpoint, select Action Points > Create Watchpoint. A watchpoint stops execution when a variable’s data changes.
  • Examine variables. Dive into a variable by clicking View > Lookup Variable or double-click the variable using your left mouse button. The Variable Window appears.
  • Visualize variables across processes by diving into a variable and click View > Show Across > Processes in the Variable Window.
  • Examine array data. Dive into an array variable. Display array subsections by editing the Slice field in an array’s Variable Window. Show statistics information about the array (or slice of array) by clicking Tools > Statistics in the Variable Window.

Figure 3. Examining Data

Examining Data


Examining Core Files

If a program encounters a serious error and a process dumps a core file you can look at this file starting Totalview with:

$ totalview mpi_prog core

The Process Window displays the core file, with the Stack Trace, Stack Frame, and Source Panes showing the state of the process when it dumped core. The title bar of the Process Window names the signal that caused the core dump. The state of all variables at the time the error occurred can be examined.

Further Information

5.2.3. DDT

Allinea DDT is a scalable, graphical debugging tool for scalar, multi-threaded and large-scale MPI applications. DDT has included a memory debugging feature that is able to detect some errors before they have caused a program crash and it can show the current state of communication between processes in a program.

DDT supports C, C++, Fortran, OpenMP, CUDA, UPC, Fortran 2008 Coarrays and many MPI implementations, including those mentioned in this guide.

Starting DDT

After compiling your program with -g, start DDT by issuing a command like this:

 $ ddt mpi_prog <input arguments>

Two windows will appear: DDT’s Main Window and t
he Run Dialog Box. The Run Dialog Box allows you to specify program arguments, input file, working directory, MPI implementation, number of processes, OpenMP support, CUDA support, support for memory debugging and environment variables.

Figure 4. DDT Startup Window

DDT Startup Window


Press the Run button at the bottom of the dialog box, to start debugging your program.

The Main Window

DDT will now show your code in the Main Window:

Figure 5. DDT Main Window

DDT Main Window


The most used buttons on the toolbar with shortcut keys, from left to right, are:

  • Play/Continue (F9): Start/continue your program
  • Pause (F10): Pause your program
  • Add Breakpoint: Press this button and add the line number. Notice, breakpoints can also be added by double-clicking a line.
  • Step Into (F5), Step Over (F8), Step Out (F6): Step into, over, or out of function calls.
  • Run To Line: Press the button and enter the line number you want to run to.
  • Down Stack Frame (Ctrl-D), Up Stack Frame (Ctrl-U), Bottom Stack Frame (Ctrl-B): After your program has stopped inside of some function, you can use buttons to step up and down in the program’s call stack.

The bottom window has a number of pages that can be selected through the tabs on the top of them. There, you can:

  • Watch the output that the program prints to standard output
  • Watch the breakpoints that you have set
  • Monitor variables’ value by creating watchpoints. A watchpoint stops execution, when a variable’s data changes.
  • Examine the current call stack. Clicking on a line in the call stack will make DDT show the corresponding program code. The number to the left of each line is the number of processes that are in that place in the code.
  • Examine and set tracepoints. Tracepoints allow you to see what lines of code that your program is executing without stopping it. DDT prints the file and line number, when a tracepoint is reached. You can also capture the value of any variable at that point.

The window to the right allows the examination of:

  • All the local variables in the current function/subroutine.
  • The variables and their current values that are used in the current line(s). A group of lines can be selected.
  • The current call stack with the value of each argument for each function call
Examining Data

If you right-click on a variable in the code a menu appears showing the type of the variable and for scalars also the value. For arrays select the View Array (MDA) menu option. A window will appear showing the content of the array, or e.g. showing a slice of an array as in the figure below.

Figure 6. Examining Data

DDT Main Window


Examining Core Files

If a program encounters a serious error and a process dumps a core file this can be examined in DDT. Start DDT without arguments and select Open Core Files on the Welcome Screen. This allows for selecting an executable and a core file to debug.

Further Information

6. Performance Analysis

Understanding how an application behaves during execution is important to identify and possibly eliminate bottlenecks in the code to improve program performance. This chapter describes some of the utilities, profilers and performance analysis tools available to assist in code analysing and tuning.

6.1. MPI Statistics

Some MPI implementations include statistics gathering facilities that can be used to determine potential performance problems. Below is a set of environment variables used to control MPI statistics recording.

  • Intel MPI
    Collect performance statistics with various level of output information:
    n, m Output
    1 Amount of data sent by each process
    2 Number of calls and amount of transferred data
    3 Statistics combined according to the actual arguments
    4 Statistics defined by a buckets list
    10 Collective operation statistics for all communica
    tion contexts
    Intel MPI also supports integrated performance monitoring (IPM) summary as part of the statistics gathering mechanism by specifying ipm as the output format. This allows for gathering statistics in regions of source codes using the MPI_Pcontrol function.

    Sample Fortran program:

     program main
     include 'mpif.h'
     INTEGER :: i, numProc, myrank, rank, tot, ierr
     CALL MPI_INIT(ierr)
     CALL MPI_PCONTROL( 1,"region_a"//char(0))
     do i = 1, numProc-1
     if(myrank == 0) then
       tot = tot + rank
     elseif(myrank == i) then
       CALL MPI_SEND(myrank,1,MPI_INTEGER,0,myrank,MPI_COMM_WORLD,ierr)
     CALL MPI_PCONTROL(-1,"regioan_a"//char(0))
     end program

    Compile the code and run:

    $ export I_MPI_STATS=ipm
    $ mpirun -n 8 ./mpi_prog

    This will produce an output file stats.ipm that contains region summary:

    # region  : region_a   [ntasks] = 8
    #                    [total]       <avg>         min           max
    # entries            8             1             1             1
    # wallclock          0.000180483   2.25604e-05   1.78814e-05   3.69549e-05
    # user               0             0             0             0
    # system             0             0             0             0
    # mpi                9.2268e-05    1.15335e-05   4.76837e-06   2.31266e-05
    # %comm                            51.1229       26.6667       69.403
    # gflop/sec          NA            NA            NA            NA
    # gbytes             0             0             0             0
    #                    [time]        [calls]       <%mpi>       <%wall>
    # MPI_Send           5.38826e-05   7             58.40         29.85
    # MPI_Finalize       3.24249e-05   8             35.14         17.97
    # MPI_Recv           5.96046e-06   7             6.46          3.30
    # MPI_TOTAL          9.2268e-05    22            100.00        51.12
    Select operations to collect statistics for. The subsystem defines the type of operation (point-to-point, collective) and ops defines the spesific MPI routine (e.g. bcast for MPI_Bcast)

    For example, collect information on operations MPI_Bcast and MPI_Reduce:

    $ export I_MPI_STATS=4
    $ export I_MPI_STATS_SCOPE="coll:bcast,reduce"
    $ mpirun -np 4 ./mpi_prog

    the output may look like (for process 1):

    Process 1 of 4 on node node0 lifetime = 427.01
    Data Transfers
    Src     Dst     Amount(MB)      Transfers
    001 --> 000     3.814697e-06    1
    001 --> 001     0.000000e+00    0
    001 --> 002     0.000000e+00    0
    001 --> 003     0.000000e+00    0
    Totals          3.814697e-06    1
    Communication Activity
    Operation       Volume(MB)      Calls
    Bcast           3.814697e-06    1
    Reduce          3.814697e-06    1
    Communication Activity by actual args
    Operation  Context  Algo  Comm size  Message size  Calls  Cost(%)
    1                0     1          4             4      1    9.38
    1                0     2          4             4      1    3.07
    Define the statistics output file. Default name for IPM statistics is stats.ipm, otherwise stats.txt

  • Platform MPI
    Enables counter instrumentation for profiling Platform MPI applications. See the documentation for available options.

    For example, specifying the api option:

    $ export MPI_INSTR=<filename>:api
    $ mpirun -np 4 ./mpi_prog

    will collect and write to the instrumentation output file <filename>.instr detailed information about MPI routine calls for each rank. Sample output:

     ------------------------   Instrumentation Data   --------------------------
     Application Summary by Rank (second):
     Rank       Proc Wall Time          User                       MPI
       0           15.399824         12.705724( 82.51%)        2.694100( 17.49%)
       1           15.314737         12.660289( 82.67%)        2.654449( 17.33%)
       2           15.346766         12.641753( 82.37%)        2.705013( 17.63%)
       3           15.322787         12.698256( 82.87%)        2.624531( 17.13%)
     ###########################################################             api
     ##                                                       ##             api
     ##    Detailed   MPI_Allreduce   routine information     ##             api
     ##                                                       ##             api
     ###########################################################             api
     Rank   MPI_Op  MPI_Datatype  Num Calls ...etc.  Message Sizes   Total Bytes
     R: 0   sum     fortran real  66486              (8 - 11016)     61712104
       Num Calls    Message Sizes     Total Bytes     Time(ms)       Mb / sec
      ----------- ------------------  ------------ --------------  -----------
           58815   [0..64]            716696               273           2.627
              29   [65..256]          5720                   2           2.406
              18   [257..1024]        9744                  27           0.359
               3   [1025..4096]       8928                   0         135.187
            7621   [4097..16384]      60971016             164         371.720

6.2. Tools

6.2.1. GNU gprof

GNU gprof is a widely used profiling tool for Unix systems which produces an execution profile of C and Fortran programs. It can show the application call graph, which represents the calling relationships between functions in the program, and the percentage of total execution time spent in each function. The tool first instruments the application executable with information that will help to build the call graph. It then runs the application and samples periodically the execution to track what code is currently being executed. This process is known as statistical sampling. Instrumentation can accurately determine the number of times a function is called, while sampling approximates the time spent within each function.

Profiling an application with gprof includes the following steps:

  1. Compile and link the program with profiling enabled.

    To compile a source file for profiling, specify the -pg option when you run the compiler. This option injects special code in the executable that will collect information about the call graph during runtime, and additionally links in versions of the library routines that are compiled for profiling. It is advisable to turn on symbol information with the -g3 option when using the gcc compiler, which is necessary if the user wishes the source code to be annotated with profiling information.

  2. Execute the application to generate profile data.

    In this step, the user runs the instrumented application as usual, and a special output file is generated (by default, it is named gmon.out). Because of the bookkeeping overhead of instrumentation and the use of a special system call needed for sampling, it is likely that gprof will slow down the application execution.

  3. Invoke gprof to analyse and display profiling results.

    The general syntax of the command is as follows:

    $ gprof options [executable-file] [profile-data-file]

    If the executable file name is omitted, the file a.out is used by default. If the profile data file is omitted, gmon.out is used. The most important options are the following:

    • --flat-profile : prints the total amount of time spent and the number of calls to each function
    • --graph : prints the call-graph analysis from the application execution
    • --annotated-source : prints profiling information next to the original source code

The main advantage of gprof is that it offers a “quick-and-dirty” way to identify hot-spots in an application, so that the programmer can focus on the most time-consuming parts of it as candidates for optimisation or parallelization. Its drawback is that it can only profile the “user-level” part of an application, i.e. it cannot profile external shared libraries (e.g. libc) or system calls. A serious side-effect of this is that multithreaded applications (e.g. OpenMP) cannot be profiled. In such cases, profiling information is gathered only for the main application thread, which might not be particularly helpful.

6.2.2. Oprofile


OProfile is a low overhead, system-wide profiler for Linux that uses performance monitoring hardware incorporated in most modern processors to spot locations and possible causes of performance bottlenecks. In contrast to gprof, OProfile collects system-wide profiling information rather than application specific, for all threads on all processors in the system, and regardless of whether they execute user-level, shared library or kernel code. Apart from counting CPU cycles to measure where execution time is spent, the tool can also measure low-level, architecture-specific microarchitectural events such as cache misses, branch mispredictions, memory references or different types of instructions.

OProfile utilizes the underlying processor’s performance counters to measure such events. Like gprof, it uses statistical sampling to associate their occurrence with specific locations in the program. This means that it does not record for every event where it happened, but for every S events. The lower the value of S, the more accurate the results are but the higher the overhead inserted is. Generally, by setting S to some moderate value we can have low overhead and at the same time quite accurate results. A list of common x86-based processors supported by OProfile is given in the following table.

Table 21. Typical x86-based processor models supported by Oprofile

Processor Model
Intel Atom, Core, Core2, Core i7, Nehalem, Westmere, Sandybridge, Pentium 4 (HT/non-HT), Pentium M, Pentium II/III/Pro, Itanium, Itanium 2
AMD Athlon, family 10, family 11h, family12h, family14h, family15h, AMD64 (Hammer)


Configuration and Use

The basic steps to use OProfile are the following:

  1. Setup

    The first thing we need is to load the kernel module by running the following command as root:

    # opcontrol --init

    To verify that the module was successfully loaded, run the lsmod command and the module should appear as oprofile.
    Furthermore, a /dev/oprofile entry should be available, with the file /dev/oprofile/cpu_type containing the specific model of the CPU being monitored. To get a list of events supported by the current CPU, we can simply execute:

    $ opcontrol --list-events

    Each event usually provides a set of unit masks. These masks can be used to further qualify the event according to certain aspects of its occurrence. For example, in many Intel processors we can use unit masks for data cache events to select only those that concern a specific MESI state (i.e., modified, exclusive, shared, invalid) or a combination of states (by logically or-ing the unit masks). In general, it is advised to consult the processor manuals to get a detailed description of events and possible fallacies regarding their measurement and interpretation.

    The next step is to configure OProfile to collect data for a specific event. This is done with the following command line:

    $ opcontrol --setup
                [--no-vmlinux | --vmlinux=kernel]

    The event argument specifies one or more events that will be sampled. Note that, depending on the event type and the processor model, it is possible that the specified events cannot be counted simultaneously. In this case, separate runs of the application are needed, one for each event. The field name refers to the event name, as reported with the list-events option. The field count specifies that the processor will be sampled every count occurrences of the event to track the current value of the program counter. For rare events a small value may be needed to get a sufficient number of samples. If this field is left empty, a default value will be used. unitmask specifies a unit mask for the event, otherwise a default one will be used (usually the logical or of all supported unit masks). The kernel and user fields specify whether OProfile should sample when the processor is running in kernel or user mode, respectively.

    If no event is specified at all, then a default one will be used. This is usually a time-based event (e.g. processor cycles) configured at a sample rate of 100,000 occurrences. It can be reported using the following command:

    $ ophelp --get-default-event

    The following table lists the time-based events for some common processor models.

    Table 22. Time-based events for common processor models

    Processor model Event name
    Intel Core, Core2, Core i7, Nehalem, Westmere, Sandybridge CPU_CLK_UNHALTED
    Intel Pentium 4 GLOBAL_POWER_EVENTS


    The argument no-vmlinux specifies that we are interested in getting profile information for user-level code only. If we want to attribute profiling samples to specific kernel functions, then the vmlinux option can be used, along with an uncompressed and unstripped image of the current kernel given as argument. The argument separate=cpu specifies that samples should be separated for each CPU in the system. This is particularly useful in order to profile parallel applications whose threads are mapped to different processors.

  2. Profile application

    To profile the application, we need to perform the following steps, one after the other:

    $ opcontrol --start
    $ ./program
    $ opcontrol --dump

    With the start option, the profiling daemon is launched. To verify that it is actually running, we check that there is a process named oprofiled. Once launched, the daemon starts collecting statistics for all programs running on all processors of the system. Therefore we can immediately start the application that we want to profile. After it finishes, we should collect the profiling data. By giving the dump option, we ensure that all the current profiling data is flushed to some special sample files before they can be analysed.

    Because OProfile automatically collects data after it has started on any program that runs, we must take care to minimize “noise” inserted by other applications that run concurrently with the application under test. For this reason, we should ensure that the intervals between opcontrol --start and application’s launch, and between application’s termination and opcontrol --dump are as short as possible. In any case, we should strive to use a dedicated and “empty” execution environment for the profiling data to be as accurate as possible.

    The daemon can be stopped by executing the command:

    $ opcontrol --shutdown

    To save the profiling data for future reference we can use the command:

    $ opcontrol --save=session_name

    This allows us to keep data from multiple experiments separate. The session_name can be given later as argument to a reporting tool to display the profile for a specific session.

  3. Results analysis and report

    After dumping samples to disk we can use the reporting tools of OProfile to analyse and display profile results. We execute the following command:

    $ opreport --threshold=1 session:session_name

    We get a report for the specified session, containing overall system profile regarding the events that were monitored, for all executables that ran in the profile period and had 1% (threshold) or more of the samples . The report lists in descending order the number of events that occurred and the name of the executable. For example, for a simple multithreaded program (mt_test) executed on a dual-core system, the output produced is like the following:

    CPU: Core 2, speed 800 MHz (estimated)
    Counted INST_RETIRED_ANY_P events (number of instructions retired) with
    a unit mask of 0x00 (No unit mask) count 10000
    Samples on CPU 0
    Samples on CPU 1
      cpu:0|            cpu:1|
    samples|      %|  samples|      %|
     554786 77.9251    418087 69.4634 mt_test
      94429 13.2635     88184 14.6514 no-vmlinux
      15267  2.1444     12080  2.0070
       7151  1.0044     16694  2.7736 chrome

    For this session we counted the number of instructions retired on each processor, with a sample rate of 10000 instructions. We can see that for our multithreaded application there were counted 554K and 418K samples on CPU 0 and CPU 1, respectively, which corresponded to 77.9% and 69.4% of all instructions retired during the profile period. The rest of the instructions were executed in the Linux kernel (no-vmlinux), in glibc and the chrome application. Other applications that gathered less than 1% of total instructions are not shown. Note that the output of opreport contains sample count and not event count. To find the actual number of events, we have to multiply the sample count by the sample rate (10000 in the example).

    It is also possible to restrict opreport to output results for a certain executable using the imag: argument. Furthermore, using the --symbols option, we can see what functions called in the executable have the most samples attributed to them. The prerequisite to have access to such information is to turn on symbol information while compiling the application using the -g option of the compiler. In the previous example, to produce output specific for mt_test application we could simply execute:

    $ opreport --threshold=1 session:session_name image:mt_test --symbols

    and get the following results:

    CPU: Core 2, speed 800 MHz (estimated)
    Counted INST_RETIRED_ANY_P events (number of instructions retired) with
    a unit mask of 0x00 (No unit mask) count 10000
    Samples on CPU 0            Samples on CPU 1
    samples  %                  samples  %        symbol name
    554786   100.000  418087    100.000           thread_fn

    This shows that 100% of the samples for the mt_test application in bothprocessors were collected while executing the thread_fn function, with thesamples being actually distributed unevenly to processors.

    It is possible to further refine the profiling results by attributing samples to specific source code lines or assembly instructions. This can be done using the opannotate command of OProfile as follows:

    $ opannotate --source session:session_name image:mt_test

    The output we get is the original source code, annotated with the profiling results as follows:

     * Total samples for file : "/home/user/mt_test.c"
     * 554786 100.0  418087 100.0
                                 :void* thread_fn(void *args)
                                 :{ /* thread_fn total:                  */
                                    /*         554786 100.0 418087 100.0 */
    ...                          :  ...
                                 : if ( ta->id == 0 ) { //thread0 code
    222654 40.1333     0       0 :     for ( i = 0; i < iters0; i++ )
    332132 59.8667     0       0 :         s.val1++;
                                 : } else if ( ta->id == 1 ) {//thread1 code
         0        153575 36.7328 :     for ( i = 0; i < iters1; i++ )
         0        264512 63.2672 :         s.val2++;
                                 : }

    With threads being mapped on different processors, we see how samples are attributed to the lines corresponding to each thread’s source code.

    OProfile is a very useful tool but there are circumstances under which it can provide inaccurate results. First, the processor does not always trigger a sample interrupt on the exact instruction that caused the event. This is due to the out-of-order instruction execution performed by most modern processors, that could cause a sample to be attributed to an instruction near the one that actually triggered the event. Second, another possible cause for weird results are the aggressive optimisations done for performance reasons by most compilers. These optimisations transform the program to an intermediate form that often has little similarity with the original source code, in terms of line-by-line correspondence. Therefore, a general advice for the user is not to interpret the results of OProfile literally, but to treat them mostly as a hint on which regions of source code to look for hot-spots.

Further Information

6.2.3. IPM

IPM is a portable profiling tool for parallel codes. It provides a low-overhead profile of the performance and resource utilization in a parallel program, focusing on communication, computation, and IO.

IPM is easy to use, it requires no modifications of the code (unless you want more information), it adds little overhead, and results are presented in text and web reports.

Using IPM
  1. If IPM is installed as a module package load the proper modulefile. This will set the environment for using the IPM library and tools.

    There are two ways to use IPM, either dynamic usage (requires no relinking of the code) by preloading the library:

    $ export LD_PRELOAD=/path/to/ipm/lib/

    or static usage by relinking the code

    $ mpicc prog.c -lipm

    (given the path to the library is set when loading the IPM modulefile).

  2. Run the program. IPM will generate an XML file looking like
  3. Generate a graphical webpage using the ipm_parse perl script. First specify
    $ export IPM_KEYFILE=/path/to/ipm/include/ipm_key

    if this is not already set in the IPM modulefile. Generate the HTML file:

    $ ipm_parse -html username.1235369687.131403.0

    This will generate a subdirectory with files. Open the index.html in a browser. See sample output in the figure below.

Figure 7. IPM Sample Output

IPM Sample Output


Using Hardware Performance Monitor Counters (HPM) with IPM

IPM has the ability to hook up to a system’s hardware performance monitors to study e.g. cache misses. Using HPM, also means that IPM gets less overhead, and results are more accurate. The counters are accessed by setting the IPM_HPM environment variable, for example:

Defining regions

More detailed information can be gained by partitioning the code into regions using the MPI_Pcontrol function. For example:

int main(int argc, char **argv)
    MPI_Pcontrol( 1, "region-a");
    MPI_Pcontrol(-1, "region-a");
    MPI_Pcontrol( 1, "region-b");
    MPI_Pcontrol(-1, "region-b");

IPM will then generate separate output for each defined region.

6.2.4. Scalasca

Scalasca is a performance analysis toolset developed by the Jülich Supercomputing Centre and the German Research School for Simulation Sciences (Aachen). Scalasca supports measurement and analysis of MPI, OpenMP and hybrid MPI/OpenMP programs written in C, C++ and Fortran.

The analysis identifies potential performance bottlenecks, e.g. concerning communication and synchronization. Among the features of Scalasca is the ability to identify wait states that occur, for example, as a result of unevenly distributed workloads.

To use Scalasca add the appropriate action flag to the scalasca command. For an explanation of the usage, type:

$ scalasca -help

This will load the “Quick Reference” document (PDF) that contains detailed instructions on how to use Scalasca.

Instrumenting and analysing

In order to use Scalasca, the source code of the program to be profiled must be instrumented. This is done by prepending the scalasca command with the instrumentation flag when compiling the code. a Makefile, replace

FC = mpif90
LD = mpif90


FC = scalasca -instrument mpif90
LD = scalasca -instrument mpif90

This will insert calls to the Scalasca measurement system into the code. Note that the instrumenter must also be used at the link stage.

Run the instrumented code by inserting scalasca -analyze in front of the job launching command, e.g. in a batch job script:

scalasca -analyze mpirun -np 160 ./mpi_prog

A unique directory is made, based on the name of the executable and the number of processes, containing log files and analysis reports with aggregate performance metrics for individual function call paths in the code. E.g. from the above example

Report examination

The analysis reports can be processed and examined using the scalasca -examine command specifying the experiment reports directory:

$ scalasca -examine epik_mpi_prog_160_sum

This will launch the report viewer, see Figure 8, “Scalasca GUI”. The window is split into three panels showing:

  • left panel: performance properties
  • middle panel: call-tree or a flat profile of the source code
  • right panel: topological view of the application’s processes

The bottom horizontal bar shows color coding according to severity, increasing from left to right.

After selecting a performance property, the middle panel shows its distribution across the call tree. Choosing one of the functions, the panel below shows the absolute value for that function and the fraction of the total (shown below the metric panel). The right panel shows the distribution across processes for that function.

Figure 8. Scalasca GUI

Scalasca GUI


Clicking the right mouse button in different parts of the panels will show additional information. For example, right-clicking on:

  • a metric in the left panel choosing Online description will show information about that performance property
  • a function name choosing Called region > Source code will show the source code of the corresponding function
  • one of the boxes representing processes in the right panel will show the MPI process rank, the coordinate in the process grid, and values (for the metric chosen) for that process.

In addition to the automatic compiler-based instrumentation described above, Scalasca provides other ways to instrument codes. For example, instrumentation can be done manually inserting EPIK API calls (a measurement library used by Scalasca) directly into the code. Refer to the Scalasca documentation for details.

Further Information
  • For more information see the Scalasca User guide (PDF).

6.2.5. TAU

TAU is a performance evaluation tool that supports parallel profiling and tracing. TAU provides a suite of static and dynamic tools that provide graphical user interaction and inter-operation to form an integrated analysis environment for parallel Fortran, C++, C, Java, and Python applications. Profiling shows how much (total) time was spent in each routine, and tracing shows when the events take place in each process along a timeline. Profiling and tracing can measure time as well as hardware performance counters from the CPU.

Instrument and run your program
  1. Set the proper environment by loading the TAU modulefile, e.g.
    $ module load tau
  2. TAU provides three scripts,,, and to instrument and compile Fortran, C, and C++ programs respectively. E.g. edit your Makefile as follows
    CC  =
    CXX =
    F90 =

    and compile your program. This will result in an instrumented executable that automatically calls the Tau library to record the values of performance registers during program execution.

    Run the program.

  3. At job completion you should have a set of profile.* files. These files can be packed into a single file with the command:
    $ paraprof --pack app.ppk
  1. Launch the paraprof visualization tool together with the profile data file:
    $ paraprof app.ppk

    Two windows will appear, the ParaProf Manager Window and the Main Data Window, see the figure below.

    Figure 9. ParaProf Manager and Main Data Windows

    ParaProf Manager and Main Data Windows


  2. The Main Data Window shows where the program used most time. The longer the bar, the more time was spent in this part of the code.
  3. Right-click on the longest bar and select Show Source Code. This will show a file where one of the functions is highlighted in blue. This function is where your program spends most of its time.
  4. Get more detailed information of the function found in the previous step.

    Exit paraprof and create a file named select.tau in the source code directory with the following content:

    loops routine="#"
  5. Type at the command line:
    $ export TAU_OPTIONS="-optTauSelectFile=select.tau -optVerbose"

    Delete the object file (*.o) of the source code file containing the function found earlier. Recompile the code and rerun the executable.

  6. Viewing the new profile data file with paraprof as before, in the source code only the single loop where most time is spent is highlighted, instead of the entire function.
Using Hardware Counters for Measurement
  1. For example, to analyse the code for floating-point operations versus cache misses ratio, set the environment variable TAU_METRICS:

    Delete the profile.* files and rerun the program. This will result in 3 MULTI__* directories.

  2. Pack and view the results as before.
  3. In the ParaProf Manager Window choose Options -> Show derived Metric Panel
  4. In the Expression panel click to make the derived metric “PAPI_FP_INS”/”TIME”. Click apply.
  5. Double-click this new metric listed. In the window that appears find the text node 0 and single-click it.

    Another window appears listing values for the floating-point operations versus cache misses ratio.

You can now consider how to change your program, e.g. changing the order of the loops, in order to make the ratio between floating point operations and cache misses go up. Once you have made a change, recompile your program and repeat the process listed above.

Further Information

6.2.6. VTune

Intel’s VTune is a profiling tool for scalar, multi-threaded and MPI applications running on Intel CPUs. Although VTune can be used with MPI programs, its functionality is aimed at analysing multi-threaded programs. VTune can find lines in the source code where a program spends the most time executing and show how much of the execution time each thread was actually working.

Running Analysis

In order to analyse a program using VTune compile the code with the -g debug flag. Collecting analysis data can be done in the VTune GUI using the amplxe-gui command or on the command line using the amplxe-cl command.

The VTune GUI provides a complete environment for collecting and examining data about programs. Start the program

$ amplxe-gui

and create a new project by clicking the button File > New > Project. In the next window specify the target type:

  • Launch Application
  • Attach to Process
  • Profile System

The first option can be used for running non-MPI programs while the second option is useful for attaching to one of the processes of a already running MPI program. The third option is used for performing a system wide lightweight analysis of the CPI (Cycles Per Instruction) value of all running programs and is more likely only interesting for system administrators. Once the target is selected, start the anlysis File > New > Analysis and chose the analysis type, e.g. Hotspots. This will identify the most time-consuming part of the source code. Click the Start button to collect analysis data.

Use the VTune command-line mode for remote analysis, e.g. running in a batch system. Specify:

$ amplxe-cl -help

to see general help and add one of the available actions to see detailed help. For example:

$ amplxe-cl -help collect

will list info about data collection options.

To perform hotspots collection on the given application prog, specify:

$ amplxe-cl -collect hotspots ./prog

Collected data will be stored in directory r000hs where r=”result”, 000=the result number and hs=”hotspots”.

Examining results

When the analysis run is complete the results can be viewed specifying

$ amplxe-gui r000hs/r000hs.amplxe

This will load the window below showing a summary of the collected performance data including a list of the top hotspots functions in the application.

Figure 10. Summary Window

Summary Window


Clicking the Bottom-up button will load the main analysis window showing the call stack function list and the sum of the time threads spent in each function. The bottom pane of the window shows a time-line for each thread with brown color showing thread activity and green idle time. Double-clicking on a function name loads the source code browser:

Figure 11. Source Window

Source Window


The horizontal blue bar besides the shown source code lines show the sum of the time that the threads have spent on each of the lines. In the simple example above most of the time is spent updating the sum reduction variable in the given function.

Further Information

< /div>

7. Batch Systems

On large scale computers users must share available resources. To manage this process most systems use a workload management system or batch system to avoid system overload and enforce a fair share of the available resources. Users will need to specify resources they need for their job, the batch system will then queue up jobs until they are ready to run. The process of selecting which jobs to run is done according to a predetermined site policy.

7.1. PBS Pro

Portable Batch System Professional (PBS Pro) is the commercial version of the PBS software offered by Altair Engineering and originally developed for NASA.

PBS Commands

PBS Pro supplies a set of user commands that are used to submit, monitor, track, delete, and manipulate jobs. The table below lists a subset of these commands.

Table 23. PBS User Commands

Command Description
qsub Submit a job.
qdel Delete a job.
qalter Alter a job’s attributes.
qmove Move a job to a different queue.
qhold Place a hold on a queued job to prevent it from running.
qrls Release hold on job.
qmsg Send a message to be printed in the job output.
qstat Request the status of jobs and queues.
tracejob Report job history for a given jobid.
xpbs Graphical User Interface to PBS commands.


The qsub command is used to submit a batch job to PBS. Submitting a job specifies a task, requests resources and sets job attributes. The qsub command can read from a job script, from standard input, or from the command line. The following table shows the most important options of the qsub.

Table 24. qsub Submission Options

Command Description
-N job_name Specify the job name.
-A account_string Specify the account number to charge for the job.
-o path Specify path for the stdout file.
-e path Specify path for the stderr file.
-l resource_list Specify the set of resources requested for the job.
-q destination Specify the job queue.
-m a,b,e Send mail when the job begins (b), ends (e), or if it aborts (a).
-M user_list List of users to whom mail about the job is sent.
-V All environment variables in the qsub command’s environment are exported to the batch job.
-J range Specify that the job is an array job.


PBS Pro also provides a set of environment variables that can be used in batch jobs. Some of these are:

The job name supplied by the user using the -N option.

The job identifier assigned to the job.
The absolute path of directory where command qsub was executed.

The filename containing a list of nodes assigned to the job.
Identifier for job arrays.

PBS Resources

Resources can be available on the server level and on the node (host) level and are specified using the -l option to command qsub. Resources made available by defining them at the server level are only used as job-wide resources. These resources (e.g. walltime) are requested as follows


A chunk is the smallest set of resources that will be allocated to a job. Requesting resources (e.g. number of processors) at node level is done using the select specification statement followed by the number of chunks # and a set of resources requested for each chunk separated by colons


See the table below for a description of some of the resources provided by PBS Pro

Table 25. PBS Resources

Command Description
walltime The wall clock time limit of the job.
cput Amount of CPU time used by all processe for the job.
mem Amount of physical memory allocated to the job.
ncpus Number of processors requested for each chunk.
mpiprocs Number of MPI processes to run in each chunk. Defaults to 1 if ncpus>0, 0 otherwise.
ompthreads Number of threads per process in a chunk.


Sample PBS Pro Job Script

Although qsub can read input e.g. from the command line it is adviced that users create batch job scripts that request resources for a job and contain the commands to be executed. Since such a job file is a shell script, the first line of the file specifies which shell to use to execute the script. Next follows the PBS directives requesting resources beginning with the default string “#PBS”.

For example, the following script will run a 1024 procesess MPI job on a 8-processor node system. Selecting ncpus=8 defines the chunk size to be equal to the size of one node. The job is run in a working subdirectory in the users home directory.

#PBS -N test
#PBS -q normal
#PBS -l walltime=24:00:00
#PBS -l select=128:ncpus=8:mpiprocs=8


# Create the working directory
if [ ! -d $wdir ]; then mkdir -p $wdir; fi

# Copy inputfile and move to working directory
cp case.inp $wdir
cd $wdir

# Run my MPI job
mpirun -np 1024 /path/to/executable/mpi_prog

Save the script, e.g. as myjob.pbs and submit it to the queueing system using the command qsub:

$ qsub myjob.pbs

When the job has submitted, PBS returns the job identifier for that job specifying a sequence_number and the servername, 384472.svcs1 in the example. The PBS_JOBID environment varibale used in the script above is translated to this number.

Display the status of batch jobs:

$ qstat
Job id            Name             User              Time Use S Queue
---------------   ------------     ------------      -------- - -----
38432.svcs1       myjob            user1             287:39:0 R normal
38434.svcs1       case1            user1             262:32:5 R bigmem
38447.svcs1       test             user2             219:59:4 R normal
38448.svcs1           user3                    0 Q normal

Add the -f option to display full job status information for a selected job:

$ qstat -f 38447.svcs1

The job identifier is used when deleting a job:

$ qdel 38447.svcs1

Further Information

7.2. Platform LSF

The IBM Platform LSF batch system uses built-in and configured resources to track resource availability and usage. Jobs are scheduled according to the resources available on individual hosts.

Jobs submitted through the LSF system will have the resources they consume monitored while they are running. This information is used to enforce resource limits and load thresholds as well as fairshare scheduling.

LSF Commands

The following table shows some of the most frequently used LSF batch commands.

Table 26. LSF User Commands

Command Description
bsub Submit job to batch subsystem.
bmod Modifies job submission options of a job.
bpeek Displays standard output and standard error output of unfinished jobs.
bstop Suspend jobs.
bresume Resume jobs.
bkill Kill a job.
bhist Displays historical information about jobs.
bacct Display accounting statistics about finished jobs.
bqueues Displays information about queues.
bjobs List jobs in the batch subsystem.
bhosts Display information about available hosts.


The bsub command submits a job to a host that satisfies all requirements of the job. If LSF cannot run all jobs immediately, scheduling policies determine the order of dispatch.

The most used options of the bsub command are listed in the table below.

Table 27. bsub Submission Options

Command Description
-J job_name Specify the job name.
-J job_name[index_list] Specify the name of a job array and the index_list.
-P project_name Specifies the project name to charge for the job.
-o path Specify a file path for the standard output of the job.
-e path Specify a file path for the standard error output of the job.
-W [hour:]minute Specify the wall clock time limit of the job.
-c [hour:]minute Limit the total CPU time the job.
-n proc Specify the number of processors required to run the job.
-q queue_name Specify the job queue.
-N Send the job report by mail when the job finishes.
-u mail_user Send mail to the specified email destination.


LSF transfers most environment variables between the submission and the execution hosts. In addition to environment variables inherited from the user environment, LSF also sets several other environment variables for batch jobs, some of these are:

Name of the job.
Job ID assigned by LSF.
The directory from where the job was submitted.
The list of hosts selected to run the job.

Sample LSF Job Script

Build job scripts starting with the shell to use to execute the script and followed by the bsub directives, each line starting with “#BSUB”. The last part of the script contains the commands to be executed.

For example, the following script will run a 1024 procesess MPI job. The job is run in a working subdirectory in the user’s home directory.

#BSUB -J test
#BSUB -q normal
#BSUB -W 24:00
#BSUB -n 1024


# Create the working directory
if [ ! -d $wdir ]; then mkdir -p $wdir; fi

# Copy inputfile and move to working directory
cp case.inp $wdir
cd $wdir

# Run my MPI job
mpirun -np 1024 /path/to/executable/mpi_prog

# Report the job statistics
bjobs -l $LSB_JOBID

Save the script. e.g. as myjob.lsf and submit it to the queueing system using the command bsub:

$ bsub < myjob.lsf
Job <38447> submitted to queue <normal>

Notice the less-than sign “<”, i.e. the script is redirected as input to the bsub command. This means the script is spooled and can be modified right after submission. Leaving out the “<” means the script is not spooled and later changes to myjob.lsf before the job completes may affect this job.

Display the status of batch jobs:

$ bjobs
38432  user1  RUN   normal  svcs1      node16     myjob     May 13 07:34
38434  user1  RUN   bigmem  svcs1      node18     case1     May 14 08:48
38447  user2  RUN   normal  svcs1      node21     test      May 16 03:38
38448  user3  PEND  normal  svcs1           May 18 14:31

7.3. LoadLeveler

The IBM Tivoli Workload Scheduler LoadLeveler is a parallel job scheduling system that allows users to run more jobs in an efficient way by maximizing resource utilization.

LoadLeveler Commands

A set of useful LoadLeveler commands is listed in the table below.

Table 28. LoadLeveler Commands

Command Description
llsubmit Submit a job.
llhold Place a hold on a job. Release the job by adding the -r switch.
llcancel Cancel job from the queue.
llclass Returns information about classes.
llq Query information about jobs in the queues.
llstatus Returns status information about nodes in the cluster.


Job submission to LoadLeveler is done using a job command file. A job command file is a shell script containing LoadLevler keywords. These keywords inform LoadLeveler of the job’s environment and the resources required for the job to run. A list of the most used keywords is shown in the table below.

Table 29. LoadLeveler Keywords

Keyword Description
job_name Specify the name of the job.
account_no Specify the account number to charge for the job.
class Specify the name of the class were the job will be run. Use command llclass to find available job classes.
job_type Specify serial for serial and SMP (OpenMP) programs, parallel for IBM POE jobs, MPICH for programs running other MPI implementations.
node Specify the number of nodes requested by a job step.
tasks_per_node Specify the number of tasks per node.
task_geometry Group tasks of a parallel job step to run together on the same node.
node_usage Specify whether the job shares nodes with other jobs.
resources Specify the resources consumed by each task of a job step.
dependency Set dependencies between job steps.
output Specify the name of the file to use as standard output.
error Specify the name of the file to use as standard error.
wall_clock_limit Set the wall clock time limit for the job.
environment Specify login initial environment variables set by LoadLeveler. Set COPY_ALL to copy all environment variables from your shell.
env_copy Specify all to copy environment variables to all nodes.
notification Specify when to notify user by mail. Default is mail sent when job ends.
queue Place the job step in the queue.


A group of environment variables are set by LoadLeveler for all jobs, including the following

Name of the job.
The job step ID.
The initial working directory.
The file that contains the host names assigned to all the tasks of the step.
Specifies the total number of tasks of the job step. This variable is available only when the job_type is set to MPICH.

Sample LoadLeveler Job Script

When building a LoadLeveler job script the keywords are listed at the top of the script embedded in comments beginning with “# @” followed by the command statement section.

For example, the following script will run a 1024 procesess MPI job on a 16-processor node system. The job is run in a working subdirectory in the users home directory.

# @ job_name         = test
# @ job_type         = MPICH
# @ class            = normal
# @ wall_clock_limit = 24:00:00
# @ node             = 64
# @ tasks_per_node   = 16
# @ queue


# Create the working directory
if [ ! -d $wdir ]; then mkdir -p $wdir; fi

# Copy inputfile and move to working directory
cp case.inp $wdir
cd $wdir

# Run my MPI job
mpirun -np 1024 /path/to/executable/mpi_prog

# Report the job statistics

Save the script, e.g. as myjob.ll and submit it to the queueing system using the command llsubmit:

$ llsubmit myjob.ll
llsubmit: The job "svcs1.38447" has been submitted

Display the status of batch jobs:

$ llq
Id               Owner    Submitted   ST PRI Class    Running On
--------------   -------- ----------- -- --- -------  ----------
svcs1.38432.0    user1    5/13 07:34  R  50  normal   node16
svcs1.38434.0    user1    5/14 08:48  R  50  bigmem   node18
svcs1.38447.0    user2    5/16 03:38  R  50  normal   node21
svcs1.38448.0    user3    5/18 14:31  I  50  normal

Further Information