<\/p>\n
Nextflow<\/a> is a scientific workflow system for creating scalable, portable, and reproducible workflows. It is based on the dataflow programming model, which greatly simplifies the writing of parallel and distributed pipelines, allowing you to focus on the flow of data and computation.<\/p>\n It consists of a Domain Specific Language (DSL), currently called Nextflow DSL2, based on Apache Groovy<\/a>. It runs on the Java Virtual Machine (JVM).<\/p>\n You can either code your own workflow pipeline using the nextflow DSL2 or you can use one of the pre-existing pipelines developed and published by the community. You can find these hosted in community repositories such as the nf-core project<\/a> or sequencing vendor specific such as Oxford Nanopore EPI2ME<\/a>.<\/p>\n Nextflow is released under the open source Apache 2.0 license<\/a> and can be accessed by all CSF users. Be aware that your usage must still adhere to the license terms and that publicly available pipelines you use may have different licenses e.g. nf-core uses the standard MIT License, but Oxford Nanopore (EPI2ME) apply their own Public License.<\/p>\n We have created a module which will set the required environment variables, load the Java virtual machine, and provide a Nextflow profile, which is compatible with the CSF3 Slurm partitions.<\/p>\n To load the module:<\/p>\n Nextflow is a bit different to other software on the CSF, please read and understand the information below before raising support requests for issues related to Nextflow.<\/p>\n When you run Nextflow, it launches a lightweight main task, which in turn submits a series of sub-tasks to Slurm on the CSF to run the steps in the pipeline.<\/p>\n Because the main task must be available for the whole of the pipeline run, we do not recommend submitting it as a Slurm job itself as it may time-out if sub-tasks have to wait for resources. Instead, you may run the Nextflow main task on one of the login nodes.<\/p>\n Pipelines (e.g. from nf-core) and their dependencies will be automatically downloaded into the Please always run Nextflow from within a directory you have created in If you are an experienced Nextflow user, you can find the path to the Manchester profile configuration file in the environment variable If you want to inspect the NXF_xxx variables we defined in our module file, as some of theme change the default storage locations, you can try We recommend launching Nextflow from one of the three login nodes<\/strong>. Make a note of which login node you use, as if you logout you would need to (re)connect to the same node<\/strong> in order to monitor<\/a> or cancel<\/a> the pipeline run. Specific login nodes can be specified when logging-in like: The generic command sequence to run Nextflow on the CSF3 is:<\/p>\n Explanation of the options shown:<\/p>\n Run Nextflow as a background process. With this option the Nextflow process will keep running in the background (in the login node), even if you log out. You can find the Nextflow process ID (PID) stored in the For a local pipeline this is your pipeline script, e.g. A custom configuration file<\/a> is needed to define the parameters and limits specific to the CSF. By loading our nextflow module you have access to our custom University of Manchester configuration file, using the variable use one or more of the defined profiles (in a comma separated list). A profile<\/a> is a set of configuration settings to be used during pipeline execution. Profiles may be defined in a pipeline’s configuration file (in the pipeline’s project directory) and in the custom configuration file loaded with the In our One or more specific pipeline<\/strong> arguments. They always start with a double dash( This section of the command redirects any standard output or error messages from the terminal into a file called The basic command to run the test case for the rnaseq pipeline<\/a> published in the nf-core project is:<\/p>\n To run the pipeline for real, remove If you want to run a specific revision of the pipeline<\/strong> (recommended), include the Detailed instructions to run the workflow are provided in its Github project page<\/a>.<\/p>\n Create a working directory in Then run the pipeline:<\/p>\n If you need a specific Nextflow version<\/strong> set the It doesn’t matter if this version is older or newer than the one we have installed on the CSF. The requested version will be downloaded into the Defining the version as shown above applies only to the run in which it is specified, it does not persist for future runs.<\/p>\n Run To check the progress of a currently running pipeline, in the launch directory read the If you want more detail, then check the more verbose To monitor the Slurm jobs launched by Nextflow, bear in mind that usually their names start with the Another helpful tip, is to add the Each hash value in the 2nd column, corresponds to a subdirectory in Finally, you can use the To cleanly cancel a pipeline run (along with all related Slurm jobs):<\/p>\n Make sure you are logged in the same login node you launched the pipeline from<\/a>.<\/p>\n<\/li>\n Type An elegant option to do this is to change to the launch directory, that contains the .pid file, and run: Nextflow also provides the ability to resume a cancelled or interrupted pipeline run. Please read the official documentation<\/a> on how to achieve this.<\/p>\n Generally, as long as you have not cleaned the launch directory, all you have to do is:<\/p>\n https:\/\/docs.seqera.io\/nextflow\/<\/a><\/p>\n https:\/\/training.nextflow.io\/latest\/<\/a><\/p>\n https:\/\/www.nextflow.io\/docs\/edge\/tutorials\/rnaseq-nf.html<\/a> https:\/\/www.nextflow.io\/docs\/latest\/vscode.html<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Overview Nextflow is a scientific workflow system for creating scalable, portable, and reproducible workflows. It is based on the dataflow programming model, which greatly simplifies the writing of parallel and distributed pipelines, allowing you to focus on the flow of data and computation. It consists of a Domain Specific Language (DSL), currently called Nextflow DSL2, based on Apache Groovy. It runs on the Java Virtual Machine (JVM). You can either code your own workflow pipeline.. Read more »<\/a><\/p>\n","protected":false},"author":24,"featured_media":0,"parent":86,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-11998","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/11998","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/users\/24"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/comments?post=11998"}],"version-history":[{"count":21,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/11998\/revisions"}],"predecessor-version":[{"id":12130,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/11998\/revisions\/12130"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/86"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/media?parent=11998"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Restrictions on use<\/h2>\n
Set up procedure<\/h2>\n
\r\nmodule load apps\/binapps\/nextflow\/25.10.4 # also loads Java openJDK 25.0.1\r\n<\/pre>\n
Running Nextflow<\/h2>\n
~\/scratch\/.nextflow<\/code> directory structure. Be aware that if you use multiple pipelines or different versions of the same pipeline, which download dependencies, this scratch location can become large and you may want to clean up<\/a> periodically.<\/p>\nscratch<\/code>. Nextflow will download and create a lot of data while running a workflow and launching several jobs, therefore storing these data in your $HOME<\/code> directory is not ideal.<\/p>\n$NXF_UOM_CONFIG<\/code> after loading the Nextflow module. If you are using your own config file, please ensure that you do NOT use a local profile, as nextflow will not use slurm and will run the pipeline from where it was launched, usually the login node.<\/p>\nmodule show apps\/binapps\/nextflow\/25.10.4<\/code>.<\/p>\nRunning Nextflow from a login node<\/h3>\n
ssh username<\/em>@login3-csf3.itservices.manchester.ac.uk<\/code> to go to login node 3.<\/p>\nmodule load apps\/binapps\/nextflow\/25.10.4 # load the module\r\n\r\nnextflow -bg run <pipeline<\/em>> -c <nextflow_config_file<\/em>> \\\r\n -profile <profile_1,profile_2,...<\/em>> [arg...] &> NXF_OUT.log\r\n<\/pre>\n
\n
-bg<\/code>:\n.nextflow.pid<\/code> file created in the directory from which you launched nextflow. See also: Monitoring_a_pipeline’s_progress<\/a> and Cancelling a pipeline run<\/a>.<\/p>\n<\/li>\n<pipeline<\/em>><\/code>:\n\/path\/to\/main.nf<\/code>.
\nFor a pipeline from an online repository<\/a>, you can give the project URL or a short version if supported e.g. nf-core pipelines.<\/p>\n<\/li>\n-c <nextflow_config_file<\/em>><\/code>:\n$NXF_UOM_CONFIG<\/code>.<\/p>\n<\/li>\n-profile <profile_1,profile_2,...<\/em>><\/code>:\n-c<\/code> option.<\/p>\n$NXF_UOM_CONFIG<\/code> Nextflow configuration file we currently have the csf3himem<\/code> profile defined. This will instruct Nextflow to launch all jobs to the himem<\/code> Slurm partition. This is chosen as the himem<\/code> partition allows both single and multicore jobs and has large memory, which is a typical requirement for bioinformatics pipelines.<\/p>\n<\/li>\n[arg..]<\/code>:\n--<\/code>) in contrast to the generic nextflow options which start with a single hyphen (e.g. -bg<\/code>). Each pipeline may have its own different options defined. For published pipelines these are usually described in the pipeline’s documentation.<\/p>\n<\/li>\n&> NXF_OUT.log<\/code>:\nNXF-OUT.log<\/code> and returns you to the command prompt. <\/p>\n<\/li>\n<\/ul>\nExample test runs<\/h3>\n
RNAseq pipeline from nf-core<\/h4>\n
\r\nnextflow -bg run nf-core\/rnaseq -c $NXF_UOM_CONFIG \\\r\n -profile test,singularity,csf3himem --outdir output-rnaseq &> NXF_OUT.log\r\n<\/pre>\n
test<\/code> from the -profile<\/code> arguments and add the required arguments for the pipeline at the end e.g. for inputs, outputs, settings. These arguments will be defined in the pipeline documentation.<\/p>\n-r<\/code> option:<\/p>\n\r\nnextflow -bg run nf-core\/rnaseq -r 3.22.2 -c $NXF_UOM_CONFIG \\\r\n -profile test,singularity,csf3himem --outdir output-rnaseq &> NXF_OUT.log\r\n<\/pre>\n
wf-human-variation workflow from Oxford Nanopore EPI2ME Labs<\/h4>\n
scratch<\/code>, and from inside that directory download and unpack the demo dataset: <\/p>\n\r\nwget https:\/\/ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com\/wf-human-variation\/hg002%2bmods.v1\/wf-human-variation-demo.tar.gz\r\ntar -xzvf wf-human-variation-demo.tar.gz\r\n<\/pre>\n
\r\nnextflow run epi2me-labs\/wf-human-variation \\\r\n -bg \\\r\n -c $NXF_UOM_CONFIG \\\r\n -profile singularity,csf3himem \\\r\n --bam 'wf-human-variation-demo\/demo.bam' \\\r\n --ref 'wf-human-variation-demo\/demo.fasta' \\\r\n --bed 'wf-human-variation-demo\/demo.bed' \\\r\n --sample_name 'DEMO' \\\r\n --snp \\\r\n --sv \\\r\n --mod \\\r\n --phased &> NXF_OUT.log\r\n<\/pre>\n
Running a specific version of Nextflow<\/h3>\n
NXF_VER<\/code> variable at the start of the command:<\/p>\n\r\nNXF_VER=23.10.1 nextflow -bg run nf-core\/rnaseq -r 3.22.2 -c $NXF_UOM_CONFIG \\\r\n -profile test,singularity,csf3himem --outdir output-rnaseq &> NXF_OUT.log\r\n<\/pre>\n
~\/scratch\/.nextflow<\/code> directory structure and run from there. <\/p>\nMonitoring a pipeline’s progress<\/h3>\n
nextflow log<\/code> from the launch directory to see the status and metadata of the current and past pipeline runs launched from the specific directory.<\/p>\nNXF_OUT.log<\/code> file like:
\ncat NXF_OUT.log<\/code>
\nOr if you want to get live updates as the log file grows:
\ntail -f NXF_OUT.log<\/code><\/p>\n.nextflow.log<\/code> file, in the same manner as described above.<\/p>\nnf-<\/code> characters. Usuful commands are:<\/p>\n\r\nsqueue # see pending and running jobs\r\nsacct -X -S now-2hours # see all jobs submitted within the past 2hours, including completed\r\n<\/pre>\n
-with-trace<\/code> option<\/a> when launching nextflow. This will create a tab-delimited csv-like file, that contains a list of all pipeline tasks, where the SLURM jobID is the 3d column. To view it, from the nextflow launch directory, try:<\/p>\ncolumn -t -s $'\\t' output\/execution\/trace.txt<\/code><\/p>\n$launchDir\/.nextflow\/work<\/code>, that contains all the task log files and outputs generated.<\/p>\nnextflow log [options]<\/code> command to filter and analyse pipeline data<\/a> and even get an html report or timeline (exported in $launchDir\/.nextflow\/output\/execution<\/code>) when the pipeline has finished. <\/p>\nCancelling a pipeline run<\/h3>\n
\n
kill PID<\/em><\/code>, where the PID is the Nextflow Process ID; the number stored in the .nextflow.pid<\/code> file located in the directory you launched the pipeline from.<\/p>\nkill $(cat .nextflow.pid)<\/code>.<\/p>\n<\/li>\n<\/ol>\nResuming a pipeline run<\/h3>\n
\n
-resume<\/code> cli option to the original command<\/li>\n<\/ul>\nFurther Info<\/h2>\n
Official documentation<\/h3>\n
Official courses<\/h3>\n
Example tutorials for piplelines<\/h3>\n
\nhttps:\/\/www.nextflow.io\/docs\/edge\/tutorials\/data-lineage.html<\/a><\/p>\nVSCode integration<\/h3>\n