Skip to content

Change output directory and file names for bwa index#9947

Open
pmoris wants to merge 15 commits intonf-core:masterfrom
pmoris:bwa-index-dir
Open

Change output directory and file names for bwa index#9947
pmoris wants to merge 15 commits intonf-core:masterfrom
pmoris:bwa-index-dir

Conversation

@pmoris
Copy link
Contributor

@pmoris pmoris commented Feb 9, 2026

  • Changes the output directory's name from bwa to ${prefix}.
  • Makes prefix default to meta.id instead of fasta.baseName.
  • Add suffix and default it to fasta.baseName.
  • Changes output filenames to ${suffix} .{amb,ann,bwt,pac,sa}

This improves the following situations:

  1. When index files for multiple references are processed simultaneously in a single process (e.g. fastq screen), there are no longer identically named bwa directories being staged (-> could be avoided by using stageAs in the process input). 2) Makes it easier to publish the resulting output directories when processing multiple reference genomes sequentially (otherwise the bwa dirs would overwrite each other; this required adaptations in modules.config to avoid like: path: { "${params.outdir}/reference/${meta.id}" }).

See discussion here: https://nfcore.slack.com/archives/CJRH30T6V/p1770372215776389

A previous fix was proposed, but never merged: #7793

Two final considerations to make:

  1. the current fix keeps things similar for downstream processes (i.e. the actual index files are still named the same as the input fasta file via fasta.baseName), but perhaps it could also be useful to do things the other way around: set the filename based on prefix/meta.id, so that it becomes possible to rename the files.
  2. I resorted to repeating the task.prefix / meta.id to capture the output directory. If there's a tidier solution for this, I'm all ears.

PR checklist

Closes #XXX

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the module conventions in the contribution docs
  • If necessary, include test data in your PR.
  • Remove all TODO statements.
  • Broadcast software version numbers to topic: versions - See version_topics
  • Follow the naming conventions.
  • Follow the parameters requirements.
  • Follow the input/output options guidelines.
  • Add a resource label
  • Use BioConda and BioContainers if possible to fulfil software requirements.
  • Ensure that the test works with either Docker / Singularity. Conda CI tests can be quite flaky:
    • For modules:
      • nf-core modules test <MODULE> --profile docker
      • nf-core modules test <MODULE> --profile singularity
      • nf-core modules test <MODULE> --profile conda
    • For subworkflows:
      • nf-core subworkflows test <SUBWORKFLOW> --profile docker
      • nf-core subworkflows test <SUBWORKFLOW> --profile singularity
      • nf-core subworkflows test <SUBWORKFLOW> --profile conda

- Changes the output directory's name from `bwa` to `${prefix}`.
- Makes prefix default to `meta.id` instead of `fasta.baseName`.
- Changes output filenames to `${fasta.baseName} .{amb,ann,bwt,pac,sa}`

This improves the following situations:

1) When index files for multiple references are processed simultaneously in a
single process (e.g. fastq screen), there are no longer identically named `bwa`
directories being staged (-> could be avoided by using stageAs in the process input).
2) Makes it easier to publish the resulting output directories when processing
multiple reference genomes sequentially (otherwise the bwa dirs would overwrite
each other; this required adaptations in modules.config to avoid like:
`path: { "${params.outdir}/reference/${meta.id}" }`).

Co-authored-by: maxulysse <maxulysse@users.noreply.github.com>
@pmoris pmoris requested a review from maxulysse as a code owner February 9, 2026 15:17
pmoris and others added 4 commits February 9, 2026 16:43
Use baseName of fasta file in case neither prefix nor meta.id are set.

Co-authored-by: maxulysse <maxulysse@users.noreply.github.com>
Correctly specifies second element of output tuple as a path
By changing prefix into a non-def variable, it becomes accessible in
the output section (see https://www.nextflow.io/docs/latest/reference/syntax.html#variable-declaration).

Co-authored-by: maxulysse <maxulysse@users.noreply.github.com>
Improves nested ternary operator to create prefix by
making fallback logic clearer and explicitly transforming
meta.id into a string.
@pmoris
Copy link
Contributor Author

pmoris commented Feb 11, 2026

I think this PR is more or less ready to be merged now, but as mentioned in Slack, the linter keeps failing. However, it was (at least locally) also failing for me on the master commit I started from (eb19663), and also on the current most recent commit (b94c791).

How should we proceed here?

╭─ [✗] 2 Module Tests Failed ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│              ╷                                    ╷                                                                                                                                                                   │
│ Module name  │ File path                          │ Test message                                                                                                                                                      │
│╶─────────────┼────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╴│
│ bwa/index    │ modules/nf-core/bwa/index/meta.yml │ correct_meta_outputs: Module meta.yml does not match main.nf. Outputs should contain: {'index': [['meta', 'bwa']], 'versions_bwa': [['${task.process}', 'bwa',    │
│              │                                    │ 'bwa 2>&1 | sed -n "s/^Version: //p']]} Run nf-core modules lint --fix to update the meta.yml file.                                                               │
│ bwa/index    │ modules/nf-core/bwa/index/meta.yml │ correct_meta_topics: Module meta.yml does not match main.nf. Topics should contain: {'versions': [['${task.process}', 'bwa', 'bwa 2>&1 | sed -n "s/^Version:      │
│              │                                    │ //p']]} Run nf-core modules lint --fix to update the meta.yml file.                                                                                               │
│              ╵                                    ╵                                                                                                                                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

@maxulysse maxulysse mentioned this pull request Feb 12, 2026
17 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants