Java
Overview
Please read the following information carefully if running java applications on the CSF, whether they be your own code or 3rd-party applications. This is to ensure you use the correct number of threads when running java.
Java Runtime Environment Versions 1.6.0, 1.7.0 and 1.8.0 are installed on the CSF. Note that loading the JRE modulefile will also give you access to the OpenJDK development packages (including the javac compiler).
Restrictions on use
There are no licensing restrictions to running this software. Please note that Java (and the Java compiler) will not run on the login node due to memory restrictions on that node. Please see our Java compilation notes for more information on compiling Java code.
Set up procedure
The default is 1.8.0. It is not strictly necessary to load a modulefile for this version, but you can if you wish:
module load tools/java/1.8.0
1.6.0 and 1.7.0 and 11 (which is newer than 1.8.0) are also available:
module load tools/java/11 module load tools/java/1.7.0 module load tools/java/1.6.0
Running the application
Java will not run on the login node and you will receive the following error if you try to run java there:
Error occurred during initialization of VM Cannot create VM thread. Out of system resources.
Jobs should be submitted to the compute nodes via batch. Note that Java may use several cores for garbage collection unless explicitly instructed not to. See below for how to run serial java code correctly.
Serial batch job submission
You must restrict the number of threads used by the java garbage collector otherwise it will attempt to use all of the cores in a node. If you are submitting a serial job then you will be using cores which have not been assigned to your job and will be trampling on other users’ jobs. Please add the following flags to your java command line:
-XX:ParallelGCThreads=1 -XX:ParallelCMSThreads=1
Java may also try to use more than one core’s share of the memory in a node. Add the following flag to your java command line to restrict memory usage to 4GB for example (if using a single core on an Intel 64GB compute node). For other nodes, please check the available memory per core and specify that number.
-Xmx4G # 4G tells Java it can use 4 GB of RAM
The above flags can be added to the java command-line (see jobscript examples below). For example, the following jobscript will run a serial java program safely:
#!/bin/bash --login #$ -cwd # Job runs in current directory - where submitted from # Load the modulefile tools/java/1.8.0 in the jobscript module load tools/java/1.8.0 java -XX:ParallelGCThreads=1 -XX:ParallelCMSThreads=1 -Xmx4G my_app_class arg1 arg2... # If you have a .jar file: java -XX:ParallelGCThreads=1 -XX:ParallelCMSThreads=1 -Xmx4G -jar myapp.jar arg1 arg2...
Submit the jobscript using:
qsub scriptname
where scriptname is the name of your jobscript.
If you are running another application which uses Java, for example MATLAB or a python program that uses Java libraries (SK-Learn is an example), then you will need to check the documentation for that application to see how pass the Java flags to the application. You could also try setting the environment variable _JAVA_OPTIONS with a list of the flags. For example:
#!/bin/bash --login #$ -cwd # Job runs in current directory - where submitted from # Load the modulefile tools/java/1.8.0 in the jobscript module load tools/java/1.8.0 # Add the following to a jobscript to set java options. Your application may # be able to use this setting and you should check if it has worked or not! export _JAVA_OPTIONS="-XX:ParallelGCThreads=1 -XX:ParallelCMSThreads=1 -Xmx4G" # Now run your app that uses java myapp arg1 arg2...
Note that by setting the above environment variable, Java will usually create a log file in your home directory showing which settings have been applied. You should delete this file to keep your home directory tidy. You will have a different log file every time you run a job that uses this environment variable.
Parallel batch job submission
If your application uses Java Threads for parallelism then you should submit the job to the smp.pe parallel environment and request an appropriate number of cores.
Note that some Java code will use as many threads as possible and there is no way to restrict the number of threads (there is no equivalent of OMP_NUM_THREADS in Java). Hence you should submit a job of 12, 16, 24, 28 or 32 cores as per the examples below. This ensures you will have the entire node to yourself and so do not need to restrict the garbage collection threads or heap size. For example:
Example Job Script
#!/bin/bash --login #$ -cwd # Job runs in current directory - where submitted from #$ -pe smp.pe n # See table below for valid numbers to replace n #$ -l arch # See table below for SGE flags to use in place of arch # Load the modulefile module load tools/java/1.8.0 java my_parallel_app_class arg1 arg2...
| SGE arch Flag | Cores for smp.pe line | Additional usage guidance |
|---|---|---|
-l haswell |
24 cores (do not set a higher or lower number than this) | Standard Haswell node. |
-l broadwell |
28 cores (do not set a higher or lower number than this) | Standard Broadwell node. |
-l skylake |
32 cores (do not set a higher or lower number than this) | Standard Skylake node. |
-l mem512 |
16 cores (do not set a higher or lower number than this) | Job will run on an Ivybridge or haswell high memory node (32GB per core of RAM).Jobs must genuinely need the higher memory |
(11.08.2021 – Sandybridge nodes removed from service.)
Which compute node type should I run on?
All of the above examples are for all of the possible Intel node types and which of these you use makes very little difference for most java applications. The batch system spreads works as evenly as possible across all the different types of node so no one set of nodes is likely to be busier or quieter than another.
Further info
Updates
None.