Skip to content

Only send finalState if needed#1306

Open
taylordowns2000 wants to merge 4 commits intomainfrom
only-send-final-state-if-needed
Open

Only send finalState if needed#1306
taylordowns2000 wants to merge 4 commits intomainfrom
only-send-final-state-if-needed

Conversation

@taylordowns2000
Copy link
Member

@taylordowns2000 taylordowns2000 commented Mar 15, 2026

Short Description

When a workflow ends with a single leaf step whose dataclip is already persisted, send final_dataclip_id instead of final_state in run:complete to avoid re-uploading state Lightning already has.

Closes #1309 and best served after OpenFn/lightning#4531 is merged.

Implementation Details

  • Replaced lastDataclipId with leafDataclipIds[] on RunState, populated in step-complete only for steps with no downstream edges.
  • In run-complete, if there's exactly one leaf dataclip that wasn't withheld, send final_dataclip_id; otherwise fall back to sending final_state as before.
  • Updated lexicon types, lightning-mock, and tests accordingly.

QA Notes

Branching workflows (multiple leaves) should still send final_state.

  • Single-leaf workflows where the dataclip was withheld should also still send final_state.
  • Single-leaf, non-withheld workflows should send only final_dataclip_id.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

  • I have used Claude Code
  • I have used another model
  • I have not used AI

You can read more details in our
Responsible AI Policy

Release branch checklist

Delete this section if this is not a release PR.

If this IS a release branch:

  • Run pnpm changeset version from root to bump versions
  • Run pnpm install
  • Commit the new version numbers
  • Run pnpm changeset tag to generate tags
  • Push tags git push --tags

Tags may need updating if commits come in after the tags are first generated.

@github-project-automation github-project-automation bot moved this to New Issues in Core Mar 15, 2026
@taylordowns2000 taylordowns2000 marked this pull request as ready for review March 16, 2026 14:32
@taylordowns2000 taylordowns2000 changed the title only send finalState if needed Only send finalState if needed Mar 16, 2026
@taylordowns2000 taylordowns2000 marked this pull request as draft March 16, 2026 15:20
@taylordowns2000 taylordowns2000 marked this pull request as ready for review March 17, 2026 10:23
@josephjclark
Copy link
Collaborator

@taylordowns2000 I'm going to put out a worker patch today but I'm going to exclude this as we're waiting on the Lightning PR. I'll get this reviewed properly with a clear head tomorrow.

@taylordowns2000
Copy link
Member Author

that's perfect @josephjclark , thank you! if you do get the chance, a review before Thursday morning (730am Paris time) would be great as it would allow @stuartc and I to review your review when we talk about OpenFn/lightning#4531

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New Issues

Development

Successfully merging this pull request may close these issues.

Don't duplicate final state payload

2 participants