PowerFLOW<\/a> (from EXA Corporation) is a fluid dynamics package providing LBM simulation techniques for real-world flow predictions.<\/p>\n Version 5.3a is installed on the CSF<\/p>\n Access to this software is restricted to a specific research group. Please contact its-ri-team@manchester.ac.uk<\/a> for access information.<\/p>\n To access the software you must first load the modulefile:<\/p>\n Please do not run PowerFLOW simulations on the login node. Jobs should be submitted to the compute nodes via batch. The PowerFLOW user interface (PowerCASE) is not<\/strong> normally run on the CSF. Only the discretize \/ decompose \/ simulation batch processes should be used. You should upload your PowerFLOW simulation on the CSF then typically consists of three steps:<\/p>\n By default, each phase will write statistics and other messages to files named The simulation (and decomposition) stage requires an extra process to be run and this will be run automatically by PowerFLOW. This command process<\/em> application is the executable which reads input files and distributes the data to the simulator processes<\/em> (a separate executable and you will usually have many simulator processes running). When specifying the number of cores to use for your simulation you will need to think about how many CSF compute nodes you wish to use and remember that PowerFLOW will add one (by default) to the number of cores requested. This extra core will be used by the command processor<\/em>. Hence you may need to reduce the number of cores you request for simulation to allow PowerFLOW to increase it for the command process!<\/p>\n For example: suppose you wish to run a simulation on two 24-core nodes (48 cores available in total). In this case you should either<\/p>\n See below for more detailed examples.<\/p>\n PowerFLOW uses its own script to submit jobs to the batch system named In particular you must specify the parallel environment on the command-line and the number of cores to use. The general forms of the commands to use on the CSF are given below. <\/p>\n Ensure you replace PEname<\/em> with a valid CSF Parallel Environment<\/a> name (for example Note that the extra command process<\/em> will be given 1 core (the default if the flag is not given). Hence you must remember that the job submitted to the batch system will request The following examples use the Note that the However, if you wish to do this step on the CSF, run the following commands (which assume you have already done the previous commands to obtain the files, load the modulefile etc):<\/p>\n We can now run any of the following commands to submit the The following command will use 4 cores in a single Intel node to run all three job steps (see above). It does this by submitting two batch jobs: the first performs discretization, the second performs decomposition and simulation (these two steps are normally run together as a single batch job). The jobs will automatically run in the correct order ( The important thing to note is that we run this as a single-node job, as indicated by our choice of PE and number of cores. It must be remembered that the discretizer cannot<\/em> be run across multiple compute nodes because it is not<\/em> an MPI program. It is<\/em> is multi-threaded so can use multiple cores in the same compute node. By using the simple command below we ask all three phases to run in the same<\/em> PE (and same number of cores) and so we must choose a PE and number of cores suitable for the discretizer. See later for how to run the simulation phase using a much higher number of cores. <\/p>\n In the above command we also ensure the batch job requests 4 cores (not 5 cores) in the batch system by adding the The discretize step can be performed on its own. As noted earlier, this must be run on a single compute node, but can use multiple cores. Hence your choice of PE must ensure this condition. For example, to run on an Intel node with 16 cores:<\/p>\n Note that the To submit a multi-node job to the 64-core AMD Bulldozer nodes, using 128 cores (must be a multiple of 64 hence two complete nodes) performing the simulate<\/em> step and, if necessary the decompose<\/em> step (assuming you have run the discretize<\/em> step earlier) and also giving the extra command-process<\/em> two cores to work with (it will only use one but can access the memory of two cores):<\/p>\n Notice that we request 126 cores for simulation and 2 cores for the command-process<\/em> giving a total of 128 cores. Running To do the same on the Intel Nodes, we recommend using the InfiniBand connected nodes (which use faster networking). In this case the PE is Finally, to combine the submission of the Discretize step as a single-node job and the Simulate (and possible decompose) step as a multi-node job, specify the PEs for discretization and simulation separately using the flags For a large simulation running on the AMD compute nodes (suitable for MACE users) a similar command-line would be:<\/p>\n To rename the output log files from the different job types and include the batch job id:<\/p>\n To have start and end times reported to a file, add the following:<\/p>\n To have an email sent when the job starts and ends, add the following:<\/p>\n For a complete list of flags run, on the login node:<\/p>\n None.<\/p>\n","protected":false},"excerpt":{"rendered":" Overview PowerFLOW (from EXA Corporation) is a fluid dynamics package providing LBM simulation techniques for real-world flow predictions. Version 5.3a is installed on the CSF Restrictions on use Access to this software is restricted to a specific research group. Please contact its-ri-team@manchester.ac.uk for access information. Set up procedure To access the software you must first load the modulefile: module load apps\/binapps\/powerflow\/5.3a Running the application Job Steps Please do not run PowerFLOW simulations on the login.. Read more »<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":31,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-2250","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/2250","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/comments?post=2250"}],"version-history":[{"count":20,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/2250\/revisions"}],"predecessor-version":[{"id":3228,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/2250\/revisions\/3228"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/31"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/media?parent=2250"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Restrictions on use<\/h2>\n
Set up procedure<\/h2>\n
\r\nmodule load apps\/binapps\/powerflow\/5.3a\r\n<\/pre>\n
Running the application<\/h2>\n
Job Steps<\/h3>\n
.cdi<\/code> file (prepared with the PowerCASE GUI) to the CSF for processing.<\/p>\n\n
.cdi<\/code> file (usually done on your local workstation), load your myinput<\/em>.case<\/code> file in to the PowerCASE gui and run the Process > Prepare CDI…<\/em> menu command to generate a myinput<\/em>.cdi<\/code> file. It is this .cdi<\/code> file which must be uploaded to the CSF.<\/li>\n<\/ul>\n\n
myinput<\/em>.cdi<\/code> file in to discrete voxels and surfels. Creates a myinput<\/em>.lgi<\/code> file to be used by the decomposer or simulator. This file can be very large. Hence you should run jobs in your scratch<\/em> directory to ensure they do not run out of disk space. If you run from your home<\/em> area you could exceed the quota applied to that area. Note:<\/strong> the discretizer is multithreaded<\/em> but does not<\/strong> run across multiple nodes via MPI. Hence this step is usually run in a separate batch job to the simulator (and decomposer) which can be<\/em> run with MPI across multiple nodes.<\/li>\nmyinput<\/em>.lgi<\/code> written by the discretizer, passing the results to the simulator without writing a file. A file will<\/em> be written if this step is run individually or the --decomp_to_file<\/code> is added to the simulation step (see below).<\/li>\ndiscretizer.o<\/code>, decomposer.o<\/code> and simulator.o<\/code> respectively. You should check these files for error messages if something is wrong with your simulation results.<\/p>\nThe Extra Command-process<\/h3>\n
\n
\n
Submitting Jobs to the Batch System<\/h3>\n
exaqsub<\/code>. Hence you do not<\/strong> need to write a jobscript. Everything is done with command-line flags to the exaqsub<\/code> command.<\/p>\nsmp.pe<\/code>). Replace N<\/em> with the number of cores to use.<\/p>\n\n
\r\nexaqsub --mpirsh ssh --submit_all_now --pe PEname<\/em> --nprocs N<\/em> --cpcores 0 myinputfile<\/em> \r\n<\/pre>\n
\n
exaqsub<\/code> command, the discretizer and simulator will both use the same PE and number of cores. But the discretizer must<\/em> run on a single node (it is multithreaded but not an MPI program) so the PE and number of cores must request only a single node. If you wish to run your simulation across multiple nodes, possibly using hundreds of cores, you will need to run the discretizer separately from the simulator. See below for how to do this.<\/li>\n--cpcores 0<\/code> option the default action would be that PowerFLOW requests N+1<\/em> cores in the batch system for decomposition and simulation processes. This may result in a bad submission request for the CSF.<\/li>\n<\/ul>\n<\/li>\n\r\nexaqsub --mpirsh ssh --pe PEname<\/em> --nprocs N<\/em> --disc myinput<\/em> \r\n<\/pre>\n<\/li>\n
\r\nexaqsub --mpirsh ssh --pe PEname<\/em> --nprocs N<\/em> --sim --cpcores 1 myinput<\/em> \r\n<\/pre>\n
N<\/em>+1<\/code> cores.<\/li>\n--sim<\/code> step will automatically run the --decomp<\/code> step if it needs to. No intermediate decomposition output file will be written unless you add --decomp_to_file<\/code> to the --sim<\/code> step (or run the --decomp<\/code> step separately), in which case the myinput<\/em>.lgi<\/code> file will be updated with decomposition information.\n<\/li>\n<\/ul>\nExamples \/ Tutorial<\/h2>\n
Obtain the Example files<\/h3>\n
nano.cdi<\/code> file supplied with PowerFLOW. Replace this name with your own file as required. To copy the nano example file run the following on the CSF login node:<\/p>\n\r\n# Create a directory in scratch for your work\r\ncd ~\/scratch\r\nmkdir pfexample\r\ncd pfexample\r\n\r\n# Set up to use PowerFLOW\r\nmodule load apps\/binapps\/powerflow\/5.3a\r\n\r\n# Now copy the nano.case and nano.cdi files\r\ncp $EXA_DIST\/testfiles\/nano.c* . # The .<\/strong> at the end is important\r\n<\/pre>\n
Prepare the Simulation<\/h3>\n
.cdi<\/code> file has already been prepared<\/em> for simulation from the .case<\/code> file and so we do not need to run the PowerCASE GUI on the CSF. You should do this step on your own workstation with your own .case<\/code> files and upload the .cdi<\/code> to the CSF.<\/p>\n\n
\r\nqrsh -l inter -l short -V -cwd powercase nano.case\r\n<\/pre>\n<\/li>\n
nano.cdi<\/code> name, pressing the Prepare<\/em> button. Then exit the PowerCASE GUI (you don’t need to save the .case<\/code> file).\n<\/li>\n<\/ul>\nnano.cdi<\/code> file to the batch system for simulation. For your real simulations you may need to use much larger core counts and choose an appropriate PE<\/a> for that number of cores.<\/p>\nSingle-node Discretize + Decompose + Simulate Job<\/h3>\n
exaqsub<\/code> uses a job dependency to make the second job wait until the first job has finished).<\/p>\n\r\nexaqsub --mpirsh ssh --submit_all_now --pe smp.pe --nprocs 4 --cpcores 0 nano\r\n<\/pre>\n
--cpcores 0<\/code> flag. This causes the extra command-process<\/em> to share a core with one of the simluator processes. This is NOT recommended for large simulations. An alternative would be to request 3 cores for simulation and allow 1 core to be used for the command process<\/em> using:<\/p>\n\r\nexaqsub --mpirsh ssh --submit_all_now --pe smp.pe --nprocs 3 --cpcores 1 nano\r\n<\/pre>\n
\n
forces.csnc<\/code>, simvol.fnc<\/code>, simvol.snc<\/code>.<\/li>\nsimulation.o<\/code>.<\/li>\nnano.lgi<\/code> and discretize.o<\/code>.<\/li>\ndecompose.o<\/code> (no intermediate result file from the decomposer is written).<\/li>\n<\/ul>\nSingle-node (multicore) Discretize Job<\/h3>\n
\r\nexaqsub --pe smp.pe --nprocs 16 --disc nano\r\n<\/pre>\n
--mpirsh ssh<\/code> flags have been removed because the discretizer doesn’t use MPI. It won’t do any harm if you leave these flags on the command-line though. The discretizer does not use an extra command-process<\/em> and so we do not specify the --cpcores<\/code> flag.<\/p>\nMulti-node Decompose + Simulate Job<\/h3>\n
\r\nexaqsub --mpirsh ssh --sim --pe orte-64bd-ib.pe --nprocs 126 --cpcores 2 nano\r\n<\/pre>\n
qstat<\/code> will show that a request to the batch system has been made for 128 cores, thereby satisfying the requirements of the orte-64bd-ib.pe<\/code> parallel environment which requires the number of cores to be a multiple of 64.<\/p>\norte-24-ib.pe<\/code> and the number of cores must be 48 or more and always a multiple of 24. For example:<\/p>\n\r\nexaqsub --mpirsh ssh --sim --pe orte-24-ib.pe --nprocs 46 --cpcores 2 nano\r\n<\/pre>\n
Single-node Discretize with Multi-node Decompose + Simulate Job<\/h3>\n
--pe disc,PEname<\/em><\/code> and --pe sim,PEname<\/em><\/code> as follows:<\/p>\n\r\nexaqsub --mpirsh ssh --submit_all_now \\\r\n --disc --pe disc,smp.pe --discnprocs 4 \\\r\n --sim --pe sim,orte-24-ib.pe --simnprocs 47 --cpcores 1 nano\r\n<\/pre>\n
\r\nexaqsub --mpirsh ssh --submit_all_now \\\r\n --disc --pe disc,smp-64bd.pe --discnprocs 64 \\\r\n --sim --pe sim,orte-64bd-ib.pe --simnprocs 126 --cpcores 2 nano\r\n<\/pre>\n
\n
--submit_all_now<\/code> flag ensures that two jobs are submitted to the batch system immediately from the login node (a job dependency makes them run in the correct order). If you omit this flag the discretizer job will run and then it will try to submit the simulator job to the batch system. This is not allowed on the CSF – jobs running on compute nodes are not able to submit further jobs to the batch system. Hence the discretizer job will fail at the point it tries to submit the simulator job. By using the --submit_all_now<\/code> flag, both jobs are submitted from the login node, which is<\/em> the correct way to submit jobs.<\/li>\n<\/ul>\nAdditional Flags<\/h3>\n
\r\n--joblog @N_@J.o # @N will be replaced with the job name, @J by the job id\r\n<\/pre>\n
\r\n--logfile filename<\/em>\r\n<\/pre>\n
\r\n--notify\r\n<\/pre>\n
\r\nexaqsub --help | less\r\n<\/pre>\n
Further info<\/h2>\n
\n
\r\nevince $POWERFLOW_HOME\/doc\/PowerFLOW-UsersGuide.pdf # User Guide\r\nevince $POWERFLOW_HOME\/doc\/PowerFLOW-CLR.pdf # Command-line reference\r\n<\/pre>\n<\/li>\n
Updates<\/h2>\n