Skip to content

Print workflow outputs on run completion#6875

Merged
pditommaso merged 2 commits intomasterfrom
print-workflow-outputs
Mar 25, 2026
Merged

Print workflow outputs on run completion#6875
pditommaso merged 2 commits intomasterfrom
print-workflow-outputs

Conversation

@bentsherman
Copy link
Copy Markdown
Member

@bentsherman bentsherman commented Feb 27, 2026

Spun off from #6574

This PR updates OutputDsl to print the workflow outputs at the end of a run.

By default it prints a human-readable summary. You can set -output-format json to output JSON instead.

It has no effect when there is no output block

When combined with -q, it is useful for printing a comprehensive pipeline output that can be passed to downstream operations

@bentsherman bentsherman requested a review from a team as a code owner February 27, 2026 16:03
@netlify
Copy link
Copy Markdown

netlify Bot commented Feb 27, 2026

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 8fc849a
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69c415da423db000085ba920

@pditommaso pditommaso force-pushed the master branch 2 times, most recently from d9fa5cd to d752bc2 Compare February 28, 2026 13:10
@pditommaso
Copy link
Copy Markdown
Member

This could be useful in several scenarios. Could not make sense to print it by default in human readable manner and use an output format flag to switch to json, similarly to module run?

@bentsherman
Copy link
Copy Markdown
Member Author

Yes I would be happy to always print something for the output.

The -o option is already used by the run command to specify the output directory, so we would probably call it -output-format

@bentsherman
Copy link
Copy Markdown
Member Author

@jorgee if I add an -output-format option to the run command then you could reuse it in module run to control the output format like we were saying

@bentsherman bentsherman added this to the 26.04 milestone Mar 11, 2026
@jorgee
Copy link
Copy Markdown
Contributor

jorgee commented Mar 13, 2026

@jorgee if I add an -output-format option to the run command then you could reuse it in module run to control the output format like we were saying

This will fit perfect for module run.

@bentsherman bentsherman force-pushed the print-workflow-outputs branch from 142f8dd to 31e5f90 Compare March 20, 2026 15:04
@bentsherman bentsherman changed the title Add -print-output option to print workflow outputs JSON Print workflow outputs on run completion Mar 20, 2026
@bentsherman
Copy link
Copy Markdown
Member Author

bentsherman commented Mar 20, 2026

Updated to print human-readable output by default.

This is what it looks like for the output-dsl.nf test:

@bentsherman
Copy link
Copy Markdown
Member Author

Refining the output:

Outputs:

  /home/bent/projects/nextflow-io/nextflow/tests/checks/output-dsl.nf/results

  samples:
    - {id: sample9, fastqc: log/sample9.fastqc.log, bam: null, bai: null, quant: quant/sample9}
    - {id: sample4, fastqc: log/sample4.fastqc.log, bam: null, bai: null, quant: quant/sample4}
    - {id: sample5, fastqc: log/sample5.fastqc.log, bam: null, bai: null, quant: quant/sample5}
    - {id: sample15, fastqc: log/sample15.fastqc.log, bam: null, bai: null, quant: quant/sample15}
    - {id: sample8, fastqc: log/sample8.fastqc.log, bam: null, bai: null, quant: quant/sample8}
    - {id: sample11, fastqc: log/sample11.fastqc.log, bam: null, bai: null, quant: quant/sample11}
    - {id: sample1, fastqc: log/sample1.fastqc.log, bam: null, bai: null, quant: quant/sample1}
    - {id: sample14, fastqc: log/sample14.fastqc.log, bam: null, bai: null, quant: quant/sample14}
    - {id: sample6, fastqc: log/sample6.fastqc.log, bam: null, bai: null, quant: quant/sample6}
    - {id: sample3, fastqc: log/sample3.fastqc.log, bam: null, bai: null, quant: quant/sample3}
    - ... (90 more items)

  summary:
    - summary_report.html
    - summary_data/data.json
    - summary_data/fastqc.txt

@pditommaso
Copy link
Copy Markdown
Member

pditommaso commented Mar 20, 2026

it looks nice! is it aligned with modules output? or the other way around?

@bentsherman
Copy link
Copy Markdown
Member Author

It is now 😄

$ nextflow -q module run test/my-test-module --meta.id 42 --id 42 --text input.txt --reads sample.fastq 
Outputs:

  /home/bent/projects/sketches/module-system-example/results

  txt:
    - {id: '42'}
    - out-42.txt

  id: '42'

$ nextflow -q module run test/my-test-module --meta.id 42 --id 42 --text input.txt --reads sample.fastq -output-format json
{
    "txt": [
        {
            "id": "42"
        },
        "/home/bent/projects/sketches/module-system-example/results/out-42.txt"
    ],
    "id": "42"
}

@pditommaso
Copy link
Copy Markdown
Member

Cool, i'll play hard with modules this w-e

@pditommaso
Copy link
Copy Markdown
Member

Code Review

A few issues worth addressing:

  1. replaceAllreplace in OutputDsl.normalizeOutput
    replaceAll(outputDir + '/', '') treats outputDir as a regex. Cloud URIs like s3://bucket.name/path will misbehave since . is a regex wildcard. Use replace() (literal string replacement) instead.

  2. Default outputFormat='text' always overrides config
    In CmdRun, outputFormat defaults to 'text' (truthy). ConfigBuilder checks if( cmdRun.outputFormat ) — which is always true, so the CLI default always wins even when the user didn't pass -output-format. Other CLI options use null as default to avoid this. Consider defaulting to null in CmdRun and falling back to 'text' at the usage site in OutputDsl.apply().

  3. No validation for -output-format values
    Invalid values like -output-format xml silently fall through to text format. Consider validating early with a helpful error message.

  4. No unit tests for DumpHelper new methods
    The new normalize(), prettyPrintYaml(), prettyPrintJson() public methods have no unit tests. Would improve confidence in edge cases (null values, nested structures, Path serialization).

  5. Typo in Session.groovy
    Line ~153: "Output format for worklfow outputs""workflow".

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman bentsherman force-pushed the print-workflow-outputs branch from e6e7eb5 to 660c8d0 Compare March 25, 2026 15:54
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman bentsherman requested a review from jorgee March 25, 2026 18:37
@pditommaso pditommaso merged commit b8752e4 into master Mar 25, 2026
26 checks passed
@pditommaso pditommaso deleted the print-workflow-outputs branch March 25, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nextflow module run fails with DataflowBroadcast cast error when process has multiple outputs

3 participants