Please refer to the Intel oneAPI<\/a> toolkit documentation to access the latest and recommended Intel compilers.<\/DIV> C ( Programming in Fortran or C\/C++ is currently beyond the scope of this webpage.<\/p>\n The Math Kernel Library (MKL) which is a part of the Intel Compilers contains optimised routines (eg BLAS) that you are encouraged to use for performance gains if not using OpenBLAS (which provides equivalent level 3 BLAS performance on the CSF hardware). Information about MKL<\/a>.<\/p>\n The CSF uses the University license. The Intel compilers are available to any student or member of staff of the University of Manchester for the purpose of the normal business of the University. Such use includes research, personal development and administration and management of the university. The software may not be used for research, consultancy or services leading to commercial exploitation of the software.<\/p>\n Code may be compiled on the login node, but aside from very short test runs (e.g., one minute on fewer than 4 cores), executables must always be run by submitting to the batch system, SGE.<\/p>\n You will need to load one<\/strong> of the following modulefiles to compile<\/em> your code and<\/strong> whenever you want to run<\/em> the executables you compiled. We recommend using the latest version if compiling new code. <\/p>\n Note, as of 17.0.7 you only need to load the combined modulefile instead of an individual fortran or c modulefile.<\/p>\n We give example firstly of how to compile and then how to run the compiled executable.<\/p>\n Make sure you have loaded the modulefile first (see above).<\/p>\n Make sure you have loaded the modulefile first (see above).<\/p>\n Make sure you have loaded the modulefile first (see above).<\/p>\n Note that is it perfectly acceptable to run the compiler as a batch job. This is recommended if you have a large compilation to perform – one that has a lot of source files and libraries to compile. <\/p>\n <\/a><\/p>\n Note that in general, you will not<\/strong> need to recompile or reinstall any applications, python envs, R packages, conda envs for the AMD Genoa nodes. Things will run perfectly well on the new nodes.<\/p>\n The AMD Genoa hardware provides the You may wish to compile code, to be optimized a little more for the AMD nodes. We will be providing more information about this in the next few months, but for now, we have some advice below.<\/p>\n We recommend using the GCC 13.3.0 compiler (or newer) as this supports the AMD AMD provide some recommended compiler flags (PDF)<\/a> to use with various compilers (GNU compiler collection, Intel OneAPI C\/C++ and the AMD AOCC compiler.) You will need to use at least anarchitecture flag<\/em> to enable the AVX-512 extensions available in the Genoa CPUs:<\/p>\n The above PDF provides further optimization flags you may wish to use in addition to the above architecture flags.<\/p>\n An example of using the common The Intel compilers will inform you when it compiles a function it thinks can benefit from optimizations specific to a particular architecture. You’ll see a message of the form:<\/p>\n See below<\/a> for more information on compiling for multiple architectures.<\/p>\n Once you have compiled your application you will run that as a batch job as normal on the CSF. For example, your compiled application will do some data processing. <\/p>\n You must<\/em> ensure the same compiler modulefile used to compile your application is loaded when you run your application as a batch job. This is because there are system libraries (and other libraries such as the MKL) that are specific to the compiler version. If you compile your code with Intel compiler 17.0.7, say, then you must run your application with that compiler’s modulefile loaded.<\/p>\n We now recommend loading modulefiles within your jobscript so that you have a full record of how the job was run. See the example jobscript below for how to do this. <\/p>\n Ensure that you have the same fortran or C\/C++ module loaded that was used to compile the application. The create a batch script similar to:<\/p>\n Submit the job to the batch system using:<\/p>\n where jobscript<\/em> is replaced with the name of your submission script.<\/li>\n<\/ul>\n Your code, and thus the resulting executable, must use either OpenMP (for single-node multicore jobs) and\/or MPI (for multi-node multicore jobs) in order to run in parallel. Please follow these links to find out how to submit batch jobs of these types to SGE:<\/p>\n Please see the relevant We suggest a few commonly used flags below but this is by no means an exhaustive list. Please see the man pages<\/em>.<\/p>\n Compile your code for use with a debugger.<\/p>\n Profiling of loops and function calls can be enabled using the following flags (for For example, to generate a profiling report of You should see something like:<\/p>\n <\/a><\/p>\n Compiling FORTRAN code with This is because linux is moving away from static linking and so the static version of the maths library <\/a><\/p>\n The CSF compute nodes have been added to the system at different times. Hence the Intel CPU architecture changes as new hardware is made available for us to add to the system. The following architecures are available with different features:<\/p>\n While a complete discussion of this is beyond the scope of this webpage (and users are encouraged to read the Intel AVX docs<\/a>), the following compiler flags will allow you to compile a single copy of your application<\/em> which is optimized for the all<\/em> of the various Intel architectures:<\/p>\n Add these flags to your The The Note that if you know you will only be running on CSF Intel hardware and not AMD hardware, you may use The compiler will report which functions have been compiled for both the various The combinations of Please note: the information in this section is repeated on the CSF FFTW Library page<\/a>.<\/p>\n The Intel Compiler’s Math Kernel Library (MKL) comes with an FFTW3 interface built-in<\/em>, and FFTW2 interfaces (and FFTW3) can be compiled by you if needed to give yourself a local copy of the Intel library (see below). Hence, if you have source code that uses FFTW functions, you can compile and link your application, without source-code changes, to the MKL libraries and it will use Intel’s built-in version of FFTW3 instead of the FFTW libraries discussed on this page.<\/p>\n Hence there are three methods of using the FFTW if your application makes FFTW function calls:<\/p>\n You should be able to use either method without changing your source code. Which to use is really a matter of performance and you could compile your app with both versions and do some timing tests to see which is best. The Intel MKL version may<\/em> run faster on some of the Intel hardware in the CSF but we make no recommendation either way – you must test your own application.<\/p>\n See further below for how to use the original FFTW libraries by loading All of the examples below assume your C\/C++ code contains:<\/p>\n <\/a><\/p>\n To compile your application using the Intel built-in FFTW3 library, without referring to a specific You should now be able to run your application with just the Intel compiler modulefile loaded – you should not<\/em> use the <\/a><\/p>\n If you wish to use the Intel MKL FFTW2 functions, or you wish to have a local copy of Intel’s interface to FFTW3 (e.g. some apps insist on linking against a You should now be able to run your application with just the Intel compiler modulefile loaded – you should not<\/em> use the Overview The instructions below refer to older Intel compilers and are no longer recommended for the CSF3. These details are kept here for reference and legacy support only. Please refer to the Intel oneAPI toolkit documentation to access the latest and recommended Intel compilers. The Intel Compilers are highly optimising compilers suitable for the CSF’s Intel processors (all compute node architectures). The compilers include features such as OpenMP and automatic parallelization which allows programmers to.. Read more »<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":130,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-168","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/comments?post=168"}],"version-history":[{"count":22,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/168\/revisions"}],"predecessor-version":[{"id":11132,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/168\/revisions\/11132"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/130"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/media?parent=168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nThe Intel Compilers are highly optimising compilers suitable for the CSF’s Intel processors (all compute node architectures). The compilers include features such as OpenMP and automatic parallelization which allows programmers to easily take advantage of multi-core processors.<\/p>\nIntel Composer XE and Intel Parallel Studio<\/h3>\n
icc<\/code>), C++ (icpc<\/code>) and Fortran (ifort<\/code>)<\/p>\n\n
MKL<\/h3>\n
Restrictions on use<\/h2>\n
Set up procedure<\/h2>\n
Intel Composer XE and Intel Parallel Studio<\/h3>\n
\r\n# Provides access to the C, C++ and FORTRAN compilers\r\nmodule load compilers\/intel\/19.1.2 # Provides icc\/ifort 2020.2.254\r\nmodule load compilers\/intel\/18.0.3 # Provides icc\/ifort 2018.3.222\r\nmodule load compilers\/intel\/17.0.7 # Provides icc\/ifort 2017.7.259\r\n<\/pre>\n
Running the application – Compiling<\/h2>\n
Example fortran compilation<\/h3>\n
\r\nifort hello.f90 -o hello\r\n #\r\n # Generates a binary executable \"hello\" from source code file \"hello.f90\"\r\n<\/pre>\n
Example C compilation<\/h3>\n
\r\nicc hello.c -o hello\r\n #\r\n # Generates a binary executable \"hello\" from source code file \"hello.c\"...\r\n<\/pre>\n
Example C++ compilation<\/h3>\n
\r\nicpc hello.cpp -o hello\r\n #\r\n # Generates a binary executable \"hello\" from source code file \"hello.cpp\"...\r\n<\/pre>\n
Optimizing Flags for CSF Hardware<\/h2>\n
avx<\/code>, avx2<\/code> and avx512<\/code> vector instructions found in the CSF’s Intel CPUs. So applications are expected to perform at least as well on the new nodes. A full discussion of this hardware is outside of the scope of this page, so please see the AMD documentation<\/a> if you want more in-depth information.<\/p>\nznver4<\/code> microarchitecture, which enables the AVX-512 extensions.<\/p>\n\r\n# GNU compilers\r\n-march=znver4 # Code will only<\/em> run on AMD Genoa and Intel Skylake (or newer)\r\n-march=haswell -mtune=znver4 # Code will run on all CSF3 node types, with some further\r\n # tuning for the AVX-512 extensions found in the AMD and\r\n # Intel Skylake nodes where possible. \r\n\r\n# Intel OneAPI compilers - see the Intel OneAPI page elsewhere in our docs<\/strong>\r\n-mavx2 -axCORE-AVX512,CORE-AVX2,AVX # Code will run on all CSF3 node types, with AVX-512\r\n # instruction enabled if supported\r\n\r\n# AMD AOCC compilers (not yet<\/em> installed on the CSF - coming soon)\r\n-march=znver4 # Code will only<\/em> run on AMD Genoa and Intel Skylake (or newer)\r\n\r\n# Note that the above flags can be applied when compiling code on the login nodes.\r\n# An alternative is to login to the AMD nodes, using qrsh, and then compile for\r\n# the \"current\" node's architecture, using:\r\n-march=native\r\n<\/pre>\n
configure<\/code> command when compiling on CSF3 that we’ve used when installing applications, is:<\/p>\n\r\n.\/configure 'CFLAGS=-march=haswell -mtune=znver4' CPPFLAGS='-march=haswell -mtune=znver4' --prefix=path\/to\/install\/area<\/em>\r\n<\/pre>\n
\r\nfilename.c<\/em>(linenum<\/em>): remark: function_name<\/em> has been targeted for automatic cpu dispatch\r\n<\/pre>\n
Running your Compiled Code<\/h2>\n
Serial Job submission<\/h3>\n
\r\n#!\/bin\/bash --login\r\n#SBATCH -p serial\r\n#SBATCH -t 1-0 # Wallclock time limit (e.g., 1-0 is 1 day)\r\n\r\n# Load the required version\r\nmodule load compilers\/intel\/17.0.7\r\n\r\n# Run your application, found in the current directory\r\n.\/hello\r\n<\/pre>\n
\r\nsbatch jobscript<\/em>\r\n<\/pre>\n
Parallel Job submission<\/h3>\n
\n
Useful Compiler Flags<\/h2>\n
man<\/code> pages. Note that ifort<\/code>, icc<\/code> and icpc<\/code> generally have the same set of flags but will differ for language-specific features.<\/p>\nOptimising and Parallelizing your Code<\/h3>\n
\n
-O0<\/code>, -O1<\/code>, -O2<\/code>, -O3<\/code>, -fast<\/code> (see fortran note<\/a>): this give you zero (ie no) optimisation (for testing numerics of the code) through to maximum optimisation (which may be processor specific).<\/li>\n-vec<\/code> (which is on by default<\/em>) vectorizes your code where possible.<\/li>\n-openmp<\/code>, -parallel<\/code>: for producing hand-coded OpenMP or automatically threaded executables, respectively, to run on smp.pe. Note that -parallel<\/code> requires -O2<\/code> or -O3<\/code> and that using -O3<\/code> will enable -opt-matmul<\/code>. Please consult the man page<\/em> for full details.<\/li>\n-opt-report<\/code>, -vec-report<\/code>, -par-report<\/code>: informs the compiler to produce a report detailing what’s been optimised, vectorized or parallelised. Can increase verbosity and detail by adding optional =N<\/em><\/code> where N<\/em> is 0<\/code>, 1<\/code>, 2<\/code> … (0<\/code> to turn off – see man page<\/em> for range of possible values and what they report). E.g.: -par-report=2<\/code>.<\/li>\n-ax<\/code> and -m<\/code> architecture flags: see below<\/a> for more information on compiling for flags used to optimize code for specific and multiple CPU architectures.<\/li>\n<\/ul>\nDebugging your Code<\/h3>\n
\n
-g<\/code>, -traceback<\/code>, -C<\/code>: flags for including debug symbols, producing a stacktrace upon runtime errors and for checking array bounds<\/li>\n<\/ul>\nProfiling your Code<\/h3>\n
icc<\/code>, icpc<\/code> and ifort<\/code>)<\/p>\n\n
-profile-functions -profile-loops=all -profile-loops-report=2<\/code><\/li>\n<\/ul>\nprog.exe<\/code> and display the results in a graphical tool supplied with the Intel compiler, use the following example<\/p>\n\r\n# Ensure you have loaded the required compiler modulefile. Then:\r\n\r\n# Compile with profiling options. \r\nicc -profile-functions -profile-loops=all -profile-loops-report=2 prog.c -o prog.exe\r\n #\r\n # Or use the 'ifort' fortran compiler\r\n # You may also want to add -fno-inline-functions if the profiler\r\n # appears not to report calls to some of your functions. \r\n # Alternatively add -O0 (which also disables inlining as part of\r\n # the optimizations). However, the loop profiler will not work\r\n # with -O0.\r\n\r\n# Copy executable to scratch and run from there\r\ncp prog.exe ~\/scratch\r\ncd ~\/scratch\r\n\r\n# Run your prog.exe code. In this case it is a serial code hence no PE needed.\r\nsrun -p interactive -t 0-1 .\/prog.exe\r\n\r\n# You'll now have a .xml file (and some .dump files) generated during profiling. \r\n# View the xml file in the Intel Java GUI tool.\r\nloopprofileviewer.sh loop_prof_1350468736.xml # The number will be different\r\n #\r\n # Note: if you get an error from Java, try running the tool again. \r\n #\r\n<\/pre>\n
\n
![\"Intel](\"http:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-content\/uploads\/loopprofileviewer.png\")
<\/a><\/li>\n<\/ul>\nFortran Static Linking (-lm error)<\/h3>\n
-static<\/code> or -fast<\/code> (which enables -static<\/code>) may produce an error:<\/p>\n\r\nld: cannot find -lm\r\n<\/pre>\n
libm.a<\/code> is installed in a path not searched by default. Use the following compile line to add the path explicitly:<\/p>\n\r\nifort -fast mycode.f90 -o mycode.exe -L\/usr\/lib\/x86_64-redhat-linux5E\/lib64\/\r\n<\/pre>\n
Optimizing for Ivybridge, Haswell, Broadwell, Skylake<\/h2>\n
\n
\r\n-msse4.2 -axCORE-AVX512,CORE-AVX2,AVX\r\n<\/pre>\n
icc<\/code> or icpc<\/code> or ifort<\/code> command. For example:<\/p>\n\r\nicc -o my_c_app -msse4.2 -axCORE-AVX512,CORE-AVX2,AVX myapp.c\r\n<\/pre>\n
-ax<\/strong>ARCH1<\/em>,ARCH2<\/em>,...<\/code> flag instructs the compiler to generate multiple, processor-specific auto-dispatch<\/em> code paths if there is a benefit. The code will be compiled in to a single executable. At runtime the executable will select the version of the compiled code suitable for the current architecture where your job is running. For example, if your job is running on a Skylake node then it may be executing the CORE-AVX512<\/code> version of particular sections of code if the compiler was able to compile a version using CORE-AVX512<\/code> optimizations. <\/p>\n-m<\/strong>ARCH<\/em><\/code> flag sets a baseline<\/em> architecture optimization for all code for which the compiler could not produce processor-specific auto-dispatch<\/em> code paths. The sse4.2<\/code> architecture will run on all CSF Intel hardware but not all AMD hardware. The older and slower sse2<\/code> (lower-case is needed) architecture will run on all CSF hardware.<\/p>\n-x<\/strong>ARCH<\/em><\/code> instead of -m<\/strong>ARCH<\/em><\/code>. This -x<\/strong><\/code> flag compiles code specifically for Intel hardware that will not run on AMD hardware.<\/p>\n-ax<\/code> architectures with a message such as<\/p>\n\r\n..\/my_solver.c(139): (col. 1) remark: solver_func() has been targeted for automatic cpu dispatch.\r\n<\/pre>\n<\/li>\n<\/ul>\n
-x<\/code>, -ax<\/code> and -m<\/code> flags are somewhat complicated and so you are encouraged to read the Intel Compiler manual page (man icc<\/code> or man ifort<\/code>).<\/p>\nIntel Compiler Interface to FFTW3 vs Original FFTW3<\/h2>\n
\n
libfftw{2,3}XX_intel.a<\/code> library for use in your compilation (some software wants to use this specific library name in the compilation process so it is sometimes easier to compile a local copy then you know you always have it available). [jump there<\/a>]<\/li>\n<\/ol>\nfftw<\/code> modulefiles.<\/p>\n\r\n#include <fftw3.h>\r\n<\/pre>\n
Using the Intel MKL Interface to FFTW (without compiling your own local copy)<\/h3>\n
libfftw3XX_intel.a<\/code> library, use the standard Intel compiler MKL flags (note in the example we use the Intel C compiler (icc<\/code>) but you can also use the Fortran (ifort<\/code>) or C++ (icpc<\/code>) compiler with the same flags):<\/p>\n\r\n# Load the Intel compiler modulefile\r\nmodule load compilers\/intel\/17.0.7\r\n\r\n# Compile a serial FFTW application using the MKL built-in FFTW3\r\nicc myfftapp.c -o myfftapp.exe -I${MKLROOT}\/include\/fftw -mkl=sequential -lm\r\n\r\n# Compile an OpenMP (multi-core) application using the MKL built-in FFTW3\r\nicc myfftOMPapp.c -o myfftOMPapp.exe -qopenmp -I${MKLROOT}\/include\/fftw -mkl -lm \r\n<\/pre>\nfftw<\/code> modulefiles discussed later on this page.<\/p>\nUsing the Intel MKL Interface to FFTW (by compiling your own local copy)<\/h3>\n
libfftw{2,3}XX_intel.a<\/code> library), you can compile the Intel MKL interface yourself so you have a local copy of the libfftw{2,3}XX<\/em>_intel<\/code> libraries. Note: If you are using fftw3 functions and want to use the Intel MKL version, you should not need to do the following compilation steps. But it won’t harm if you do:<\/p>\n\r\n# Decide on the name and location of a folder where you want to compile a local\r\n# copy of the Intel FFTW3 interface. This might be inside your source tree. For example:\r\nexport VBASE=$HOME\/myfftapp\/fftw3_intel\r\n\r\n# Add a sub-directory name for the particular flavour of FFTW3 interface (single precision etc):\r\n# and create that entire directory tree. For example we want the
fftw3xf_intel.a<\/code> library:\r\nmkdir -p $VBASE\/fftw3xf\/lib\r\n\r\n# Load the Intel compiler modulefile for your required version. EG:\r\nmodule load compilers\/intel\/17.0.7\r\n\r\n# Now, go to the Intel compiler central install area so that you can see\r\n# the FFTW interface files that Intel provides:\r\npushd $INTELROOT\/mkl\/interfaces\/fftw3xf\r\n\r\n# Finally, compiler their<\/em> interface code but make\r\n# sure it writes to your<\/em> installation area created above:\r\nmake libintel64 compiler=intel install_to=$VBASE\/fftw3xf\/lib<\/strong> 2>&1 | tee $VBASE\/fftw3xf\/lib<\/strong>\/make-csf.log\r\n\r\n# Go back to where you were previously:\r\npopd\r\n\r\n# Check we have a library:\r\nls $VBASE\/fftw3xf\/lib\/*.a\r\n #\r\n # You should see a libfftw3xf_intel.a file listed\r\n\r\n# Now, when you compile your source code, you should tell the intel\r\n# compiler where your local copy of the Intel FFTW files are:\r\nicc myfftapp.c -o myfftapp.exe -I${MKLROOT}\/include\/fftw -L$VBASE\/fftw3xf\/lib -lfftw3xf_intel\r\n<\/pre>\nfftw<\/code> modulefiles discussed later on this page.<\/p>\nFurther info and help<\/h2>\n
\n
\r\nman ifort\r\n # for the fortran compiler\r\n\r\nman icc\r\n # for the C\/C++ compiler\r\n<\/pre>\n<\/li>\n
ifort --help<\/code> and icc --help<\/code> give similar information<\/li>\n