Nanos6 Runtime

Nanos6 is a runtime that implements the OmpSs-2 parallel programming model, developed by the Programming Models group at the Barcelona Supercomputing Center.

Installation

Build requirements

To install Nanos6 the following tools and libraries must be installed:

1. automake, autoconf, libtool, pkg-config, make and a C and C++ compiler
2. boost >= 1.59
3. hwloc
4. numactl

It's highly recommended to have an installation of the OmpSs-2 LLVM/Clang which supports the OmpSs-2 model. When installing OmpSs-2 for the first time, you can break the chicken and egg dependency between Nanos6 and LLVM/Clang in the following way: you can build Nanos6 without specifying any LLVM/Clang, then, build LLVM/Clang specifying that Nanos6 installation, and finally, re-configure and build Nanos6 passing the --with-nanos6-clang to specify the LLVM/Clang path.

Important: The Mercurium source-to-source compiler is the OmpSs-2 legacy compiler and it is unsupported now. You do not have to install it to use OmpSs-2. We recommend using the LLVM/Clang compiler instead.

Optional libraries and tools

In addition to the build requirements, the following libraries and tools enable additional features:

1. Extrae to generate execution traces for offline performance analysis with Paraver
2. CUDA to enable CUDA tasks
3. PGI or NVIDIA HPC-SDK to enable OpenACC tasks
4. PQOS to generate real-time statistics of hardware counters
5. DLB to enable dynamic management and sharing of computing resources
6. jemalloc to use jemalloc as the default memory allocator, providing better performance than the default glibc implementation. Jemalloc must be compiled with --enable-stats and --with-jemalloc-prefix=nanos6_je_ to link with the runtime
7. PAPI >= 5.6.0
8. Babeltrace2 to enable the fast CTF converter (ctf2prv --fast) and the multi-process trace merger (nanos6-mergeprv)
9. ovni >= 1.5.0 to generate execution traces for performance analysis with Paraver

Build procedure

Nanos6 uses the standard GNU automake and libtool toolchain. When cloning from a repository, the building environment must be prepared through the following command:

$ ./autogen.sh

Use this script instead of the autoreconf command. When the code is distributed through a tarball, it usually does not need that command.

Then execute the following commands:

$ ./configure --prefix=INSTALLATION_PREFIX ...other options...
$ make all check
$ make install

where INSTALLATION_PREFIX is the directory into which to install Nanos6.

The configure script accepts the following options:

1. --with-nanos6-clang=prefix to specify the prefix of the LLVM/Clang installation which supports OmpSs-2
2. --with-nanos6-mercurium=prefix to specify the prefix of the Mercurium installation
3. --with-boost=prefix to specify the prefix of the Boost installation
4. --with-libnuma=prefix to specify the prefix of the numactl installation
5. --with-extrae=prefix to specify the prefix of the extrae installation
6. --with-dlb=prefix to specify the prefix of the DLB installation
7. --with-papi=prefix to specify the prefix of the PAPI installation
8. --with-pqos=prefix to specify the prefix of the PQoS installation
9. --with-cuda[=prefix] to enable support for CUDA tasks; optionally specify the prefix of the CUDA installation, if needed
10. --enable-openacc to enable support for OpenACC tasks; requires PGI compilers
11. --with-pgi=prefix to specify the prefix of the PGI or NVIDIA HPC-SDK compilers installation, in case they are not in $PATH
12. --enable-chrono-arch to enable an architecture-based timer for the monitoring infrastructure
13. --with-babeltrace2=prefix to specify the prefix of the Babeltrace2 installation and enable the fast CTF converter (ctf2prv --fast) and the multi-process trace merger (nanos6-mergeprv)
14. --with-ovni=prefix to specify the prefix of the ovni installation and enable the ovni instrumentation

The hwloc dependency is mandatory, and, by default, an internal hwloc is embedded to the Nanos6 library. This behavior can be modified through the --with-hwloc option, which can take the following values:

• --with-hwloc=embedded: The hwloc is built and embedded into the Nanos6 library as an internal module. This is useful when user programs may have third-party software (e.g., MPI libraries) that depend on a different hwloc version and may conflict with the one used by Nanos6. In this way, the hwloc library is internal and is only used by Nanos6. This is the default behavior if the option is not present, or no value is provided. See Embeddeding software dependencies for more information
• --with-hwloc=pkgconfig: The hwloc is an external installation and Nanos6 should discover it through the pkg-config tool. Make sure to set the PKG_CONFIG_PATH if the hwloc is not installed in non-standard directories
• --with-hwloc=<prefix>: A prefix of an external hwloc installation

The jemalloc dependency is optional but highly recommended. This allocator significantly improves the performance of the Nanos6 runtime by optimizing the memory allocations. By default, an internal jemalloc is embedded to the Nanos6 library. This behavior can be modified through the --with-jemalloc option, which can take the following values:

• --with-jemalloc=embedded: The jemalloc is built and embedded into Nanos6 as an internal library. The building process installs the jemalloc headers and libraries in $INSTALLATION_PREFIX/deps/nanos6/jemalloc and dynamically links our runtime against the jemalloc library. This is the default behavior if the option is not provided. See Embeddeding software dependencies
• --with-jemalloc=<prefix>: A prefix of an external jemalloc installation configured with the --enable-stats and --with-jemalloc-prefix=nanos6_je_ options
• --with-jemalloc=no or --without-jemalloc: Disable the jemalloc allocator (not recommended)

The location of an external hwloc can be retrieved through pkg-config when specifying --with-hwloc=pkgconfig. If it is installed in a non-standard location, pkg-config can be told where to find it through the PKG_CONFIG_PATH environment variable. For instance:

$ export PKG_CONFIG_PATH=/apps/HWLOC/2.0.0/INTEL/lib/pkgconfig:$PKG_CONFIG_PATH

To enable CUDA the --with-cuda flag is needed. The location of CUDA can be retrieved automatically, if it is in standard system locations (/usr/lib, /usr/include, etc), or through pkg-config. Alternatively, for non-standard installation paths, it can be specified using the optional =prefix of the parameter.

The --enable-openacc flag is needed to enable OpenACC tasks. The location of PGI compilers can be retrieved from the $PATH variable, if it is not specified through the --with-pgi parameter.

Embedding software dependencies

As mentioned above, there are some software dependencies that may be embedded into Nanos6. This is the case for hwloc and jemalloc, which will be embedded by default. The sources of these embedded dependencies are taken from the deps sub-directory in this repository. Inside the sub-directory, there is a default hwloc and jemalloc source tarballs. These tarballs are automatically extracted into deps/hwloc and deps/jemalloc by our autogen.sh script.

These are the source packages that are then built when choosing --with-hwloc=embedded or --with-jemalloc=embedded. You may change the embedded software version by placing the desired tarball inside the deps folder and re-running autogen.sh with the option --embed-<SOFTWARE> <VERSION>. Currently, <SOFTWARE> can be hwloc or jemalloc and <VERSION> should be the desired version number of that software. For instance, a valid option could be ./autogen.sh --embed-jemalloc 5.3.0.

For the moment, the tarballs must follow the format deps/<SOFTWARE>-<VERSION>.tar.gz.

Compiling and executing programs

You can compile OmpSs-2 programs that are written in C and C++ with the LLVM/Clang compiler. Remember that you must specify the -fompss-2 flag in both compilation and linking commands. The LLVM/Clang will generate a parallel binary that will be linked with the Nanos6 runtime library. For instance:

$ clang -c -fompss-2 a_part_in_c.c
$ clang++ -c -fompss-2 a_part_in_c_plus_plus.cxx
$ clang++ -fompss-2 a_part_in_c.o a_part_in_c_plus_plus.o -o app

We still recommend using the Mercurium source-to-source compiler for Fortran applications.

Then, the application can be executed as is. The runtime system will leverage the system cores specified by the CPU mask of the process. The number of cores that are used is controlled by running the application through the taskset command. For instance:

$ taskset -c 0-2,4 ./app

would run app on cores 0, 1, 2 and 4.

Runtime settings

The behaviour of the Nanos6 runtime can be tuned after compilation by means of a configuration file. The default configuration file nanos6.toml can be found in architecture-independent data directory of the Nanos6 installation, usually in the $INSTALLATION_PREFIX/share folder. You can check where is the default configuration file by running the nanos6-info --default-config command. Currently, the supported format is TOML v1.0.0-rc1 (https://toml.io/en/v1.0.0-rc.1).

To override the default configuration, we recommended to copy the default file and change the relevant options. The Nanos6 runtime will only interpret the first configuration file found according to the following order:

1. The file pointed by the NANOS6_CONFIG environment variable.
2. The file nanos6.toml found in the current working directory.
3. The file nanos6.toml found in the installation path (default file).

Alternatively, if configuration has to be changed programatically and creating new files is not practical, configuration variables can be overriden using the NANOS6_CONFIG_OVERRIDE environment variable. The contents of this variable have to be in the format key1=value1,key2=value2,key3=value3,.... For example, to change the dependency implementation and CTF instrumentation: NANOS6_CONFIG_OVERRIDE="version.dependencies=discrete,version.instrument=ctf" ./ompss-program.

Scheduling options

The scheduling infrastructure provides the following configuration variables to modify the behavior of the task scheduler.

• scheduler.policy: Specifies whether ready tasks are added to the ready queue using a FIFO (fifo) or a LIFO (lifo) policy. The fifo is the default.
• scheduler.immediate_successor: Probability of enabling the immediate successor feature to improve cache data reutilization between successor tasks. If enabled, when a CPU finishes a task it starts executing the successor task (computed through their data dependencies). Default is 0.75.
• scheduler.priority: Boolean indicating whether the scheduler should consider the task priorities defined by the user in the task's priority clause. Enabled by default.

Benchmarking, tracing, debugging and other options

There are several Nanos6 variants, each one focusing on different aspects of parallel executions: performance, debugging, instrumentation, etc. Nanos