10. Nextflow configuration¶

A key Nextflow feature is the ability to decouple the workflow implementation by the configuration setting required by the underlying execution platform.

This enables portable deployment without the need to modify the application code.

10.1 Configuration file¶

When a workflow script is launched, Nextflow looks for a file named nextflow.config in the current directory and in the script base directory (if it is not the same as the current directory). Finally, it checks for the file: $HOME/.nextflow/config.

When more than one of the above files exists, they are merged, so that the settings in the first override the same settings that may appear in the second, and so on.

The default config file search mechanism can be extended by providing an extra configuration file by using the command line option: -c <config file>.

10.1.1 Config syntax¶

A Nextflow configuration file is a simple text file containing a set of properties defined using the syntax:

1

name = value

10.1.2 Config variables¶

Configuration properties can be used as variables in the configuration file itself, by using the usual $propertyName or ${expression} syntax.

1
2
3

propertyOne = 'world'
anotherProp = "Hello $propertyOne"
customPath = "$PATH:/my/app/folder"

10.1.3 Config comments¶

Configuration files use the same conventions for comments used in the Nextflow script:

1
2
3
4
5
6

// comment a single line

/*
   a comment spanning
   multiple lines
 */

10.1.4 Config scopes¶

Configuration settings can be organized in different scopes by dot prefixing the property names with a scope identifier or grouping the properties in the same scope using the curly brackets notation. This is shown in the following example:

1
2
3
4
5
6
7

alpha.x  = 1
alpha.y  = 'string value..'

beta {
    p = 2
    q = 'another string ..'
}

10.1.5 Config params¶

The scope params allows the definition of workflow parameters that override the values defined in the main workflow script.

This is useful to consolidate one or more execution parameters in a separate file.

1
2

params.foo = 'Bonjour'
params.bar = 'le monde!'

1
2
3
4
5

params.foo = 'Hello'
params.bar = 'world!'

// print both params
println "$params.foo $params.bar"

nextflow run params.nf

Bonjour le monde!

nextflow run params.nf --foo Hola

Hola le monde!

10.1.6 Config env¶

The env scope allows the definition of one or more variables that will be exported into the environment where the workflow tasks will be executed.

1
2

env.ALPHA = 'some value'
env.BETA = "$HOME/some/path"

Save the above snippet as a file named my-env.config. Then save the snippet below in a file named foo.nf:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

process FOO {
    debug true

    script:
    '''
    env | egrep 'ALPHA|BETA'
    '''
}

workflow {
    FOO()
}

Finally, execute the following command:

nextflow run foo.nf -c my-env.config

BETA=/home/user/some/path
ALPHA=some value

10.1.7 Config process¶

Process directives allow the specification of settings for the task execution such as cpus, memory, container, and other resources in the workflow script.

This is useful when prototyping a small workflow script.

However, it’s always a good practice to decouple the workflow execution logic from the process configuration settings, i.e. it’s strongly suggested to define the process settings in the workflow configuration file instead of the workflow script.

The process configuration scope allows the setting of any process directives in the Nextflow configuration file. For example:

1
2
3
4
5

process {
    cpus = 10
    memory = 8.GB
    container = 'biocontainers/bamtools:v2.4.0_cv3'
}

The above config snippet defines the cpus, memory and container directives for all processes in your workflow script.

The process selector can be used to apply the configuration to a specific process or group of processes (discussed later).

The syntax for setting process directives in the configuration file requires = (i.e. assignment operator), whereas it should not be used when setting the process directives within the workflow script.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

process FOO {
    cpus 4
    memory 2.GB
    time 1.hour
    maxRetries 3

    script:
    """
    your_command --cpus $task.cpus --mem $task.memory
    """
}

This is especially important when you want to define a config setting using a dynamic expression using a closure. For example, in a workflow script:

1
2
3

process FOO {
    memory { 4.GB * task.cpus }
}

And the equivalent in the configuration file, if you choose to set it there:

1
2
3
4
5

process {
    withName: FOO {
        memory = { 4.GB * task.cpus }
    }
}

Directives that require more than one value, e.g. pod, in the configuration file need to be expressed as a map object.

1
2
3

process {
    pod = [env: 'FOO', value: '123']
}

Finally, directives that are to be repeated in the process definition, in the configuration files need to be defined as a list object. For example:

1
2
3
4

process {
    pod = [[env: 'FOO', value: '123'],
           [env: 'BAR', value: '456']]
}

10.1.8 Config Docker execution¶

The container image to be used for the process execution can be specified in the nextflow.config file:

1
2

process.container = 'nextflow/rnaseq-nf'
docker.enabled = true

The use of unique "SHA256" Docker image IDs guarantees that the image content does not change over time, for example:

1
2

process.container = 'nextflow/rnaseq-nf@sha256:aeacbd7ea1154f263cda972a96920fb228b2033544c2641476350b9317dab266'
docker.enabled = true

10.1.9 Config Singularity execution¶

To run a workflow execution with Singularity, a container image file path is required in the Nextflow config file using the container directive:

1
2

process.container = '/some/singularity/image.sif'
singularity.enabled = true

The following protocols are supported:

• library:// download the container image from the Singularity Library service.
• shub:// download the container image from the Singularity Hub.
• docker:// download the container image from the Docker Hub and convert it to the Singularity format.
• docker-daemon:// pull the container image from a local Docker installation and convert it to a Singularity image file.

1
2

process.container = 'nextflow/rnaseq-nf'
singularity.enabled = true

nextflow run script7.nf

10.1.10 Config Conda execution¶

The use of a Conda environment can also be provided in the configuration file by adding the following setting in the nextflow.config file:

1

process.conda = "/home/ubuntu/miniconda2/envs/nf-tutorial"

You can specify the path of an existing Conda environment as either directory or the path of Conda environment YAML file.