{"id":298,"date":"2013-04-26T08:20:57","date_gmt":"2013-04-26T08:20:57","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/?page_id=298"},"modified":"2020-02-27T12:55:56","modified_gmt":"2020-02-27T12:55:56","slug":"starccm","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/software\/applications\/starccm\/","title":{"rendered":"StarCCM+"},"content":{"rendered":"

Overview<\/h2>\n

StarCCM+ is a computation continuum mechanics application which can handle problems relating to fluid flow, heat transfer and stress. See below for the available versions.<\/p>\n

Restrictions on Use<\/h2>\n\n\n\n
Only users who have been added to the StarCCM group can run the application (run groups<\/code> to see your group memberships). Owing to licence restrictions, only users from the School of MACE can be added to this group. Requests to be added to the StarCCM group should be emailed to
\nits-ri-team@manchester.ac.uk<\/a>.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Please note:<\/strong> As of October 2014 all CSF users are encouraged to use a total of 64 cores running StarCCM jobs<\/strong> initially. This will allow you determine if the job will complete using that number of cores and the amount of memory available to those cores. If you need more cores (and memory) you should then increase the number of cores requested in the jobscript (see below). Using more cores than you need will increase queueing time and license usage which may prevent teaching clusters in MACE from running the software during term time.<\/p>\n

When the job runs there may not be enough licenses for the number of cores you have requested. There is currently a high demand for licenses and they may run out. If this happens your CSF jobs will fail with a license error. To check whether there are at least enough licenses for you job you can add the following line to your jobscript:<\/p>\n

. $STARCCM_HOME\/liccheck.sh\r\n<\/pre>\n
(that’s a full-stop followed by a space at the start of the line – please copy it carefully.)<\/p>\n
If there are not<\/em> enough licenses for your job the job will automatically re-queue in the CSF batch system. If there are enough licenses the job will run StarCCM.<\/p>\n
Set Up Procedure<\/h2>\n
Once you have been added to the StarCCM+ group, you will be able to access the executables after issuing one<\/strong> of the following module commands:<\/p>\n
# Choose required precision (double is more accurate but slower)\r\nmodule load apps\/binapps\/starccm\/13.04-double\r\nmodule load apps\/binapps\/starccm\/13.04-mixed\r\nmodule load apps\/binapps\/starccm\/12.04-double\r\nmodule load apps\/binapps\/starccm\/12.04-mixed\r\nmodule load apps\/binapps\/starccm\/12.02-double\r\nmodule load apps\/binapps\/starccm\/12.02-mixed\r\nmodule load apps\/binapps\/starccm\/11.06-double\r\nmodule load apps\/binapps\/starccm\/11.06-mixed\r\nmodule load apps\/binapps\/starccm\/11.02-mixed\r\nmodule load apps\/binapps\/starccm\/10.02-single\r\nmodule load apps\/binapps\/starccm\/10.02-double\r\nmodule load apps\/binapps\/starccm\/9.06-single\r\nmodule load apps\/binapps\/starccm\/9.06-double\r\n\r\n# Double precision only\r\nmodule load apps\/binapps\/starccm\/9.04\r\n\r\n# Single precision only\r\nmodule load apps\/binapps\/starccm\/9.02\r\nmodule load apps\/binapps\/starccm\/8.06\r\nmodule load apps\/binapps\/starccm\/8.04\r\nmodule load apps\/binapps\/starccm\/7.06\r\nmodule load apps\/binapps\/starccm\/7.04\r\nmodule load apps\/binapps\/starccm\/5.04\r\n<\/pre>\n
Both a double precision and single precision version of 9.06 and later have been installed – reflected in the above modulefile names.<\/p>\n
Version 9.04 & is double precision, all other versions are single precision.<\/p>\n
Running the Application<\/h2>\n
Currently, use of StarCCM+ in batch mode only is supported; no attempt should be made to use the StarCCM+ GUI directly on the CSF (however, see below for the client-server<\/a> method). Once you have loaded the modulefile for the version you wish to run please write a jobscript based on one of those below and then submit it to the batch system with this command:<\/p>\n
qsub myjobscript<\/em>\r\n<\/pre>\n
replacing myjobscript<\/em><\/code> with the name of your file.<\/p>\n
Available Parallel Environments for StarCCM+<\/h2>\n
When submitting StarCCM+ jobs to the batch system, the following hp-mpi-*.pe<\/code> SGE parallel environments must be used. This ensures proper allocation of StarCCM+ processes to compute nodes. If not specified correctly you could be slowing down your own or other users’ jobs.<\/p>\n\n\n\n\n\n\n\n\n
PE Name<\/th>\nDescription<\/th>\n<\/tr>\n
hp-mpi-smp.pe<\/td>\nSuitable for small parallel jobs, between 2 and 32 cores, using the AMD Magny-Cours nodes with 2GB per core. See example<\/a>.<\/td>\n<\/tr>\n
hp-mpi-smp-64bd.pe<\/td>\nSuitable for medium, parallel jobs, between 2 and 64 cores, using the AMD Bulldozer nodes with 2GB per core. You should use the hp-mpi-smp.pe<\/code> PE above for small jobs initially. See example<\/a><\/td>\n<\/tr>\n
hp-mpi-32-ib.pe<\/td>\nSuitable for large parallel jobs, 64 or more cores in multiples of 32, using the AMD Magny-Cours nodes. See example<\/a>.<\/td>\n<\/tr>\n
hp-mpi-64bd-ib.pe<\/td>\nSuitable for large parallel jobs, 128 or more cores in multiples of 64, using the AMD Bulldozer nodes. See example<\/a>.<\/td>\n<\/tr>\n
orte-64bd-ib.pe<\/td>\nSuitable for large parallel jobs using an experimental<\/em> method, 128 or more cores in multiple of 64, using the AMD Bulldozer nodes. See example<\/a>.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Please pay attention to the jobscript examples – some of the flags needed on the starccm+<\/code> command-line change depending on the type of job you are running.<\/p>\n
Small Parallel Jobs<\/h2>\n
These are jobs that use a single compute node. If your jobs use fewer than 64 cores please submit your jobs to the 32-core AMD Magny-Cours nodes initially if possible. Small jobs submitted to the AMD Bulldozer nodes (64-core nodes) may be suspended if there is a queue of much larger jobs waiting for the 64-core nodes, and you will be asked to resubmit the job to the AMD Magny-Cours nodes (32-core nodes).<\/p>\n
<\/a><\/p>\n
Magny-Cours – jobs of up to 32 cores<\/h3>\n
This example describes how to submit an SMP job (uses cores in a single compute node) to the AMD Magny-Cours nodes on the CSF.<\/p>\n
#!\/bin\/bash\r\n#$ -pe hp-mpi-smp.pe 32           # 2-32 cores permitted\r\n#$ -cwd                           # Run in current directory\r\n#$ -V                             # Inherit environment settings from modulefile\r\n\r\n# Check for licenses, requeue if not enough available.\r\n. $STARCCM_HOME\/liccheck.sh\r\n\r\n# $NSLOTS is automatically set by the batch system to the\r\n# number of slots you specified above on the -pe line.\r\n\r\nstarccm+ -batch myinput.sim<\/em> -rsh ssh -np $NSLOTS -machinefile machinefile.$JOB_ID\r\n                  #\r\n                  #\r\n                  # Replace myinput.sim<\/em> with your input file.\r\n<\/pre>\n
If submitting a job with fewer<\/em> than 32 cores in the hp-mpi-smp.pe<\/code> PE then you should add the following flag to the starccm+<\/code> command-line (e.g., at the end of the line):<\/p>\n
-cpubind off\r\n<\/pre>\n
For example:<\/p>\n
starccm+ -batch myinput.sim<\/em> -rsh ssh -np $NSLOTS -machinefile machinefile.$JOB_ID -cpubind off<\/strong>\r\n<\/pre>\n
If your job uses exactly<\/em> 32 cores you must not<\/em> use this flag – it will slow down your job.<\/p>\n
<\/a><\/p>\n
Bulldozer – jobs of up to 64 cores<\/h3>\n
This example describes how to submit a job to a single AMD Bulldozer node on the CSF.<\/p>\n
#!\/bin\/bash\r\n#$ -pe hp-mpi-smp-64bd.pe 64        # 2-64 cores permitted\r\n#$ -cwd                             # Run in current directory\r\n#$ -V                               # Inherit environment settings from modulefile\r\n\r\n# Check for licenses, requeue if not enough available.\r\n. $STARCCM_HOME\/liccheck.sh\r\n\r\n# $NSLOTS is automatically set by the batch system to the\r\n# number of slots you specified above on the -pe line.\r\n\r\nstarccm+ -batch myinput.sim<\/em> -rsh ssh -np $NSLOTS -machinefile machinefile.$JOB_ID -mpi intel<\/strong>\r\n                  #                                                                 #\r\n                  #                                                                 # -mpi intel<\/strong> for versions\r\n                  #                                                                 # higher than 8.06 except 9.04 !!<\/strong>\r\n                  # Replace myinput.sim<\/em> with your input file.\r\n<\/pre>\n
If submitting a job with fewer<\/em> than 64 cores in the hp-mpi-smp-64bd.pe<\/code> PE then you should add the following flag to the starccm+ command-line (e.g., at the end of the line):<\/p>\n
-cpubind off\r\n<\/pre>\n
For example<\/p>\n
starccm+ -batch myinput.sim<\/em> -rsh ssh -np $NSLOTS -machinefile machinefile.$JOB_ID -mpi intel -cpubind off<\/strong>\r\n<\/pre>\n
If your job uses exactly<\/em> 64 cores you must not<\/em> use this flag – it will slow it down.<\/p>\n
Please check how your job scales before running large parallel jobs!<\/h2>\n
Do not assume that using twice as many cores will run you job in half the time! Jobs do not always scale linearly. Users are encouraged to test how their problem scales when considering a new problem or mesh size. This can be achieved easily, using an inbuilt script provided by CD-Adapco, which can be invoked using the -benchmark<\/code> flag. For example:<\/p>\n
#!\/bin\/bash\r\n########## Choose ONE<\/strong> of the following lines (to benchmark the AMD Bulldozer or Magny-Cours nodes)\r\n#$ -pe hp-mpi-64bd-ib.pe 128        # The benchmark will use up to 128 cores (two 64-core nodes)\r\n#$ -pe hp-mpi-32-ib.pe 64           # The benchmark will use up to 64 cores (two 32-core nodes)\r\n########## Choose ONE<\/strong> of the above lines\r\n#$ -cwd                             # Run in current directory\r\n#$ -V                               # Inherit environment settings from modulefile\r\n\r\n# Check for licenses, requeue if not enough available.\r\n. $STARCCM_HOME\/liccheck.sh\r\n\r\nstarccm+ -benchmark<\/strong> -rsh ssh -np $NSLOTS -machinefile machinefile.$JOB_ID myinput.sim<\/em>\r\n              #                                                             #\r\n              # Run the automatica benchmarks                               #\r\n              # on your input model                                         # Replace myinput.sim<\/em> with\r\n                                                                            # your input file.\r\n<\/pre>\n
This will run a few iterations on 1, 2, 4, 8, 16, 32, 64 and finally 128 cores, timing the runs. The output is a .html<\/code> file which can be opened using a web browser. This file will show how your problem scales, which can be used to decide how many cores to use. The parallel efficiency can be computed as speedup \/ number of workers from the resulting table. The point at which efficiency drops below ~85% users should consider using fewer cores, which may actually increase the speed of your computations.\u00a0 Example output<\/a> of benchmark.<\/p>\n
Large Parallel Jobs<\/h2>\n
<\/a><\/p>\n
Magny-Cours – multi-node jobs of 64 cores or more<\/h3>\n
This is a multi-node example using two 32-core Magny-Cours nodes (hence 64 cores in total). You can specify more nodes by using multiples of 32 for the number of cores (e.g., 96 for three 32-core nodes).<\/p>\n
#!\/bin\/bash\r\n#$ -pe hp-mpi-32-ib.pe 64           # 64 or more cores in multiples of 32\r\n#$ -cwd                             # Run in current directory\r\n#$ -V                               # Inherit environment settings from modulefile\r\n\r\n# Check for licenses, requeue if not enough available.\r\n. $STARCCM_HOME\/liccheck.sh\r\n\r\nstarccm+ -batch myinput.sim<\/em> -rsh ssh -np $NSLOTS -machinefile machinefile.$JOB_ID\r\n                  #\r\n                  #\r\n                  # Replace myinput.sim<\/em> with your input file.\r\n<\/pre>\n
<\/a><\/p>\n
Bulldozer – multi-node jobs of 128 cores or more<\/h3>\n
This is a multi-node example using two 64-core Bulldozer nodes (hence 128 cores in total). You can specify more nodes by using multiples of 64 for the number of cores (e.g., 192 for three 64-core nodes).<\/p>\n
#!\/bin\/bash\r\n#$ -pe hp-mpi-64bd-ib.pe 128        # 128 cores or more in multiple of 64\r\n#$ -cwd                             # Run in current directory\r\n#$ -V                               # Inherit environment settings from modulefile\r\n\r\n# Check for licenses, requeue if not enough available.\r\n. $STARCCM_HOME\/liccheck.sh\r\n\r\nstarccm+ -batch myinput.sim -rsh ssh -np $NSLOTS -machinefile machinefile.$JOB_ID -mpi intel<\/strong>\r\n                  #                                                                     #\r\n                  #                                                                     # -mpi intel<\/strong> for versions\r\n                  #                                                                     # 8.06 and above except 9.04 !!<\/strong>\r\n                  # Replace myinput.sim<\/em> with your input file.\r\n<\/pre>\n
Please note that starccm+ run on multiple Bulldozer nodes will output errors\/warnings in your output file similar to this:<\/p>\n
node201:87b8:b67519e0: 92 us(92 us):  open_hca: device mlx4_0 not found\r\nnode201:87b8:b67519e0: 724 us(632 us): node201:87c3:e41719e0: 86 us(86 us):  open_hca: device mlx4_0 not found\r\nnode201:87c3:e41719e0: 189 us(103 us):  open_hca: device mlx4_0 not found\r\n open_hca: device mlx4_0 not found\r\nnode204:c94e:6a9d720: 24120 us(24120 us):  open_hca: getaddr_netdev ERROR: No such device. Is ib1 configured?\r\nnode204:c94e:6a9d720: 36487 us(36090 us):  open_hca: device mthca0 not found\r\n<\/pre>\n
but your job should run and complete. It is not able to use the full efficiency of the infiniband. Investigations as to why are ongoing.<\/p>\n
<\/a><\/p>\n
Experimental Bulldozer – jobs of 128 cores or more<\/h3>\n
If you are having problems running large Bulldozer jobs try the following alternative method. Note that the jobscript uses a different PE (the standard orte-64bd-ib.pe<\/code> rather than the special hp-mpi-64bd-ib.pe<\/code> PE). You must<\/strong> also set the extra environment variable as indicated in the jobscript below.<\/p>\n
Before using this jobscript please load the extra<\/em> Bulldozer MPI modulefile (normally StarCCM uses its own MPI files but here we are going to use the centrally-installed MPI on the CSF):<\/p>\n
# Extra modulefile needed: for StarCCM version 8.x to 11.x\r\nmodule load mpi\/open64-4.5.2\/openmpi\/1.6-ib-amd-bd\r\n\r\n# Extra modulefile needed: for StarCCM version 12.x and later\r\nmodule load mpi\/open64-4.5.2.1\/openmpi\/1.8.3-ib-amd-bd\r\n\r\n# Now load the starccm version you require as usual - for example:\r\nmodule load apps\/binapps\/starccm\/11.06-mixed\r\n<\/pre>\n
Then use the following jobscript – this is different to your normal StarCCM jobscripts so please copy it carefully (we use 320 cores as an example – this is a large job!)<\/p>\n
#!\/bin\/bash \r\n#$ -pe orte-64bd-ib.pe 320            # Note: Using the standard orte-64bd-ib.pe, not hp-mpi-64bd-ib.pe\r\n                                      # (still requires 128 cores or more in multiple of 64)\r\n#$ -cwd                               # Run in current directory\r\n#$ -V                                 # Inherit environment settings from modulefile\r\n\r\n# Check for licenses, requeue if not enough available.\r\n. $STARCCM_HOME\/liccheck.sh\r\n\r\n### Extra<\/strong> environment variable needed by StarCCM+\r\nexport OPENMPI_DIR=$MPI_HOME\r\n\r\nstarccm+ -batch -rsh ssh -mpi openmpi -batchsystem sge .\/myinput.sim<\/em>\r\n                                 #                #         #\r\n                                 #                #         # Replace with your input file.\r\n                                 #                #\r\n                                 #                # Extra flag to indicate we are using\r\n                                 #                # the CSF batch system's machinefile.\r\n                                 #\r\n                                 # Extra flag to indicate we are using\r\n                                 # the CSF's central OpenMPI software.\r\n<\/pre>\n
The above should also work for smaller job sizes.<\/p>\n
Runtime Limits and Licensing<\/h2>\n
Please note the following advice:<\/p>\n