Weather Research & Forecasting (WRF)<\/a> Model is a next-generation mesoscale numerical weather prediction system designed to serve both atmospheric research and operational forecasting needs.<\/p>\n WPS + WRF ARW (Advanced Research WRF) v3.6 compiled for Real cases, (distributed memory parallelism – dmpar – version) with basic nesting is installed on the CSF.<\/p>\n Version 3.6 has been compiled for use on AMD Bulldozer nodes only.<\/p>\n Version 3.8 has been compiled for use on Intel nodes only.<\/p>\n All users of WRF on the CSF are strongly encouraged to work through the tutorial at the UoM WRF Community WRF \/ CSF tutorial page<\/a>. This provides a complete example of a WRF simulation including pre-processing with WPS and post-processing with NCL. Thanks to Jonathan Fairman (CAS<\/a>) for developing this tutorial.<\/p>\n WPS + WRF and their dependency libraries were compiled using the PGI 13.6 with ACML (math libraries) optimized for Bulldozer FMA4 instructions (see modulefiles below). The following compiler flags were used:<\/p>\n See RAC Community Wiki WRF Build Pages<\/a> for how this version was compiled.<\/p>\n WPS + WRF and their dependency libraries were compiled using Intel 15.0, using the following compiler flags:<\/p>\n -w -O3 -ip -msse2 -axSSE4.2,AVX,CORE-AVX2<\/p>\n See here<\/a> for build details.<\/p>\n For post processing of WRF results NCL<\/a> is installed on the CSF. This is a binary install so can be run on any CSF nodes (not just AMD Bulldozer).<\/p>\n None — Public Domain<\/a>.<\/p>\n To access the software you must first load the appropriate modulefile from the options below (if you’re not sure, it’s probably the first one):<\/p>\n The Both To see the actual values of these variables (e.g. so you can put the WPS_GEOG directory in a namelist file) run:<\/p>\n The Please do not run WRF on the login node. Jobs should be submitted to the compute nodes via batch.<\/p>\n Important:<\/strong> you must run WRF in your scratch<\/em> directory, not your home<\/em> directory. WRF input and output files can be large and you can generate lots of them. You will very likely fill up the home<\/em> filesystem which is shared with other members of your group. This will cause your jobs and other users’<\/strong> jobs to fail – resulting in very unhappy colleagues! Please run in scratch<\/strong>. Any important results (and small input\/config files such as namelist<\/em> files and batch jobscripts) can be copied back to home<\/em> for safe-keeping (home is backed up, scratch is not).<\/p>\n The WRF and WPS executables were built as dmpar executables during WRF config (i.e., they use MPI). The following are available:<\/p>\n Make sure you have loaded the modulefile, then create a batch submission script using the template Note: If trying to run any of the above executables on the login node you will get an error message:<\/p>\n This is because the executables are compiled for the AMD Bulldozer architecture. The login node uses an Intel CPU.<\/p>\n Make sure you have the modulefile loaded then create a batch submission script to run on the AMD Bulldozer nodes, for example:<\/p>\n Submit the jobscript using:<\/p>\n where scriptname<\/em> is the name of your jobscript.<\/p>\n If you need to compile a model against the WRF installation or wish to quickly run one of the WPS tools then you’ll need to start an interactive session on the Bulldozer short<\/em> node as follows:<\/p>\n You can now run WRF and WPS tools interactively. However, please note:<\/p>\n <\/p>\n None.<\/p>\n","protected":false},"excerpt":{"rendered":" Overview Weather Research & Forecasting (WRF) Model is a next-generation mesoscale numerical weather prediction system designed to serve both atmospheric research and operational forecasting needs. Version Installed WPS + WRF ARW (Advanced Research WRF) v3.6 compiled for Real cases, (distributed memory parallelism – dmpar – version) with basic nesting is installed on the CSF. 3.6 Version 3.6 has been compiled for use on AMD Bulldozer nodes only. 3.8 Version 3.8 has been compiled for use.. Read more »<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":31,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-1565","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/1565","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/comments?post=1565"}],"version-history":[{"count":20,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/1565\/revisions"}],"predecessor-version":[{"id":4134,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/1565\/revisions\/4134"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/31"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/media?parent=1565"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Version Installed<\/h3>\n
3.6<\/h4>\n
3.8<\/h4>\n
WRF Tutorial for CSF Users<\/h3>\n
Compilation Info<\/h3>\n
3.6<\/h4>\n
-tp bulldozer -O3 -fast\r\n<\/pre>\n
3.8<\/h4>\n
Related Tools<\/h3>\n
Restrictions on use<\/h2>\n
Set up procedure<\/h2>\n
module load apps\/intel-15.0\/WRF\/3.8\u00a0\r\nmodule load apps\/pgi-13.6-acml-fma4\/wrf\/3.6-ib-amd-bd\r\n<\/pre>\n
Settings applied by the WRF modulefile<\/h3>\n
WRF<\/code> modulefiles will automatically load the following modulefiles, which indicate the dependencies used when compiling WRF:<\/p>\n3.6<\/h4>\n
# These are automatically loaded for you\r\n\r\ncompilers\/PGI\/13.6-acml-fma4 # PGI 13.6 with optimized maths libraries\r\nmpi\/pgi-13.6-acml-fma4\/openmpi\/1.6-ib-amd-bd # OpenMPI 1.6 with InfiniBand\r\nlibs\/pgi-13.6-acml-fma4\/zlib\/1.2.8-ib-amd-bd # ZLIB compression\r\nlibs\/pgi-13.6-acml-fma4\/hdf\/5\/1.8.13-ib-amd-bd # HDF-5 (serial I\/O)\r\nlibs\/pgi-13.6-acml-fma4\/netcdf\/4.3.2-ib-amd-bd # NetCDF4 inc FORTRAN libraries\r\n<\/pre>\n
3.8<\/h4>\n
# These are automatically loaded for you \r\n\r\ncompilers\/intel\/c\/15.0.3\r\ncompilers\/intel\/fortran\/15.0.3\r\nmpi\/intel-15.0\/openmpi\/1.8.30-ib\r\nlibs\/intel-15.0\/netcdf\/4.4.0\r\nlibs\/intel-15.0\/zlib\/1.2.8\r\nlibs\/intel-15.0\/hdf\/5\/1.8.16<\/pre>\n
Both versions<\/h4>\n
wrf<\/code> modulefiles will set the following environment variables (for convenient access to the installation directories and for some compilation settings):<\/p>\n\n
$WRF_DIR<\/code> – the WRFV3 installation directory<\/li>\n$WPS_DIR<\/code> – the WPS installation directory<\/li>\n$WPS_GEOG<\/code> – the new static WPS_GEOG data installation directory<\/li>\n$JASPERLIB<\/code> – Location of JPEG libraries for GRIB2<\/li>\n$JASPERINC<\/code> – Location of JPEG headers for GRIB2<\/li>\n$WRFIO_NCD_LARGE_FILE_SUPPORT<\/code> – Is set to 1 to enable compilation of Large File support in NetCDF<\/li>\n$NETCDF4<\/code> – Is set to 1 to enable compilation of NetCDF support<\/li>\n<\/ul>\necho $WPS_GEOG\r\n<\/pre>\n
$WRF_DIR\/run\/<\/code> and $WPS_DIR<\/code> directories will also be added to your $PATH<\/code> environment variable.<\/p>\nRunning the application<\/h2>\n
Parallel batch job submission – all versions<\/h3>\n
\n
$WRF_DIR\/run\/<\/code>: ndown.exe, nup.exe, real.exe tc.exe wrf.exe<\/li>\n$WPS_DIR\/<\/code>: geogrid.exe, metgrid.exe, ungrib.exe(this one is not an MPI executable)<\/li>\n<\/ul>\nVersion 3.8 only<\/h3>\n
$WRF_DIR\/run\/qsub-wrf.sh<\/code> as an example:<\/p>\n#!\/bin\/bash\r\n#!\/bin\/bash\r\n#$ -S bash\r\n#$ -cwd # Job will run from the current directory\r\n#$ -V # Job will inherit current environment settings\r\n#$ -N WRF_job # Give the job a name of your choosing.\r\n#$ -m bea # Send yourself an email when job starts, ends, or on error.\r\n#$ -M firstname.surname@manchester.ac.uk # Change to your email address.\r\n\r\n##### Multi-node MPI #####\r\n#$ -pe orte-24-ib.pe 48 -l haswell \u00a0 \u00a0\u00a0 # Use a multiple of 24\r\nmpirun -n $NSLOTS .\/wrf.exe # $NSLOTS is automatically set to number of cores<\/pre>\n
Version 3.6 only<\/h3>\n
Illegal instruction\r\n<\/pre>\n
#!\/bin\/bash\r\n#$ -S \/bin\/bash\r\n#$ -cwd # Job will run from the current directory\r\n#$ -V # Job will inherit current environment settings\r\n\r\n########### Choose ONE of these PEs ##########\r\n\r\n##### Multi-node MPI #####\r\n\r\n#$ -pe orte-64bd-ib.pe 128 # 128 cores or more in this PE, multiples of 64 only\r\n\r\n##### Single-node MPI or OpenMP #####\r\n\r\n#$ -pe smp-64bd.pe 64 # 64 cores or fewer\r\n\r\n\r\nmpirun -n $NSLOTS real.exe\r\n<\/pre>\n
qsub scriptname<\/em><\/pre>\n
Interactive Use and Compilation<\/h3>\n
qrsh -l bulldozer -l short\r\n #\r\n # Wait for new prompt (or try again later if asked)\r\n\r\n# Now set up environment on backend Bulldozer node\r\nmodule load apps\/pgi-13.6-acml-fma4\/wrf\/3.6-ib-amd-bd\r\n<\/pre>\n
\n
qrsh<\/code> command reserves you only one core. If you plan to run small MPI jobs interactively you must reserve the correct number of cores using:\nqrsh -l bulldozer -l short -pe smp-64bd.pe 8 # e.g., 8 cores\r\n<\/pre>\n<\/li>\n<\/ul>\n
Further info<\/h2>\n
\n
Updates<\/h2>\n