feat: Fast Lanes — dedicated worker capacity for named queues by stuartc · Pull Request #4515 · OpenFn/lightning

stuartc · 2026-03-09T14:37:32Z

Description

This PR adds Fast Lanes support to Lightning — dedicated worker capacity for
named queues, guaranteeing low-latency execution for sync workloads.

Closes #4498

This is a tracking PR that collects the Lightning-side implementation:

Add runs.queue column and set queue at run creation #4500 — Add runs.queue column and set queue at run creation (PR Add runs queue column for Fast Lanes support #4514)
Thread queue preferences through the run claim path #4501 — Thread queue preferences through the run claim path (PR Thread queue preferences through the claim path (#4501) #4516)

Related worker-side issues:

Worker: Add --queues option and slot group configuration parsing kit#1288 — Add --queues CLI option and slot group configuration parsing
Implement per-group workloops and queue-aware claiming kit#1289 — Implement per-group workloops and queue-aware claiming

What's included

Data model (#4500)

runs.queue string column (NOT NULL, default "default")
Composite partial index on (state, queue, inserted_at) for active runs
Queue assignment at run creation:
- Sync webhook (webhook_reply: :after_completion) → "fast_lane"
- All other triggers (webhook/cron/kafka) → "default"
- Manual runs and retries → "manual"
Validation restricts queue to default, fast_lane, or manual

Claim path (#4501)

claim message gains a queues parameter (ordered preference chain)
Filter mode (no *): strict pinning, only named queues returned
Preference mode (with *): all queues eligible, named ones prioritized
Backward compatible: omitting queues defaults to ["manual", "*"]
Queue filtering/ordering in shared Queue.claim/4 benefits both FifoRunQueue and RoundRobinQueue
sanitise_queues/1 validates input and falls back to defaults for malformed payloads

Validation steps

Run mix ecto.reset to apply the split migrations cleanly
mix test — all tests pass
Verify sync webhook triggers produce runs with queue: "fast_lane"
Verify cron/async webhook triggers produce runs with queue: "default"
Verify manual runs and retries produce runs with queue: "manual"
Verify filter mode: a worker claiming with ["fast_lane"] only receives fast_lane runs
Verify preference mode: a worker claiming with ["manual", "*"] gets manual runs first, then FIFO
Verify backward compat: a worker claiming without queues gets the default ["manual", "*"] behavior

Additional notes for the reviewer

Phase 2 cleanup (removing the priority column, Remove priority column from runs table #4502) is out of scope
and will happen after Fast Lanes is deployed and stable
The priority column is kept and still written for manual/retry runs
(:immediate) as a backward-compat safety net during rollout
The implementation uses COALESCE(array_position(...), wildcard_pos) instead
of CASE WHEN for queue preference ordering — semantically equivalent but
cleaner for variable-length preference chains
RoundRobinQueue (Thunderbolt) changes are out of scope — that's thunderbolt#627,
which this PR unblocks

AI Usage

I have used Claude Code
I have used another model
I have not used AI

Pre-submission checklist

I have performed an AI review of my code (we recommend using /review
with Claude Code)
I have implemented and tested all related authorization policies.
(e.g., :owner, :admin, :editor, :viewer)
I have updated the changelog.
I have ticked a box in "AI usage" in this PR

codecov · 2026-03-09T14:50:52Z

Codecov Report

❌ Patch coverage is 96.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 89.56%. Comparing base (d8eca73) to head (efd00df).

Files with missing lines	Patch %	Lines
lib/lightning/runs/queue.ex	90.90%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4515      +/-   ##
==========================================
+ Coverage   89.54%   89.56%   +0.01%     
==========================================
  Files         425      425              
  Lines       20307    20325      +18     
==========================================
+ Hits        18184    18204      +20     
+ Misses       2123     2121       -2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

* feat: add queue column to runs table for Fast Lanes support (#4500) Add a `queue` VARCHAR column to the `runs` table (NOT NULL, default "default") with a composite partial index on (state, queue, inserted_at) for active runs. Queue assignment rules: - Webhook triggers with webhook_reply: :after_completion → "fast_lane" - All other triggers (webhook/cron/kafka) → "default" - Manual runs and retries → "manual" * refactor: split migration, move queue routing to WorkOrders Split the combined add-column + concurrent-index migration into two separate migrations (transactional and non-transactional). Move queue determination logic from Run.for(%Trigger{}) into WorkOrders.build_for where all other routing decisions live. Unify both Run.for/2 clauses to use put_if_provided/3. Add tests for invalid queue rejection.

* feat: add queue column to runs table for Fast Lanes support (#4500) Add a `queue` VARCHAR column to the `runs` table (NOT NULL, default "default") with a composite partial index on (state, queue, inserted_at) for active runs. Queue assignment rules: - Webhook triggers with webhook_reply: :after_completion → "fast_lane" - All other triggers (webhook/cron/kafka) → "default" - Manual runs and retries → "manual" * refactor: split migration, move queue routing to WorkOrders Split the combined add-column + concurrent-index migration into two separate migrations (transactional and non-transactional). Move queue determination logic from Run.for(%Trigger{}) into WorkOrders.build_for where all other routing decisions live. Unify both Run.for/2 clauses to use put_if_provided/3. Add tests for invalid queue rejection. * feat: thread queue preferences through the claim path (#4501) Workers can now pass a `queues` parameter in their claim request to filter or prioritize runs by queue. Filter mode (no wildcard) returns only runs from named queues. Preference mode (with `*`) orders named queues first while still returning all runs. Defaults to ["manual", "*"] for backward compatibility. * test: add queue filtering and preference tests for claim path Cover filter mode, preference mode, default behavior, and malformed input fallback across Queue, FifoRunQueue, Runs, Services.RunQueue, and WorkerChannel. Update existing claim tests to new 3-arity signatures. * chore: fix worker_name typespec and remove dead test code

Runs on the fast_lane queue now bypass project concurrency checks in the claim query, ensuring sync webhook responses are never blocked by in-progress cron/async runs.

Queue.claim was stripping all ORDER BY clauses before applying queue preference ordering. This broke Thunderbolt's round-robin scheduler which relies on project_id ordering in the base query. Now saves and restores the caller's ordering after queue preference.

Map priority 0 (immediate) to "manual" and priority 1 (normal) to "default" instead of setting all rows to "default".

….22.1 Update @openfn/ws-worker from 1.17.0 to 1.22.1, which includes the --workloops CLI option for splitting worker capacity into independent slot groups with queue preferences (e.g., "fast_lane:1 manual>*:4"). RuntimeManager changes: - Add workloops field to Config struct, passed as --workloops flag - Remove default capacity (nil instead of 5) so the worker uses its own defaults unless explicitly configured - Add WORKER_WORKLOOPS env var to bootstrap config - capacity and workloops guards only emit when set; if both are configured the worker itself reports the conflict

Replace case expression with function clause pattern matching to bring complexity under Credo's threshold.

What changed in lib/lightning/work_orders.ex: - retry/3 → passes Keyword.get(opts, :queue, "manual") to enqueue_retry, so single retries default to "manual" - retry_many([%RunStep{}], opts) → injects queue: "default" into opts before calling individual retry() calls (bulk retry from a specific job) - retry_many([%WorkOrder{}], opts) → unchanged; it enqueues Oban jobs which call enqueue_many_for_retry with default "default" - enqueue_many_for_retry/3 → accepts optional queue param (default "default") for bulk retries from start - enqueue_retry/3 and new_retry_run/8 — now accept and thread through the queue value Result: ┌────────────────────────┬───────────┐ │ Path │ Queue │ ├────────────────────────┼───────────┤ │ Manual run (click Run) │ "manual" │ ├────────────────────────┼───────────┤ │ Single retry │ "manual" │ ├────────────────────────┼───────────┤ │ Bulk retry from start │ "default" │ ├────────────────────────┼───────────┤ │ Bulk retry from job │ "default" │ └────────────────────────┴───────────┘

taylordowns2000

it's great. thank you. i've made a small change (bulk retries shouldn't get same priority as manual runs) which you're welcome to revert if it's more complicated than i thought. otherwise please merge at your leisure. i'm now moving on to the TB review.

github-project-automation bot added this to Core Mar 9, 2026

github-project-automation bot moved this to New Issues in Core Mar 9, 2026

stuartc marked this pull request as ready for review March 12, 2026 13:59

stuartc added 6 commits March 18, 2026 08:35

feat: initialize Fast Lanes branch

2eecea2

feat: exempt fast_lane runs from concurrency limits

19f2461

Runs on the fast_lane queue now bypass project concurrency checks in the claim query, ensuring sync webhook responses are never blocked by in-progress cron/async runs.

fix: backfill runs queue column from priority during migration

efd00df

Map priority 0 (immediate) to "manual" and priority 1 (normal) to "default" instead of setting all rows to "default".

stuartc force-pushed the fastlanes branch from 389227f to efd00df Compare March 18, 2026 08:54

stuartc requested a review from taylordowns2000 March 18, 2026 13:39

stuartc and others added 4 commits March 18, 2026 15:44

fix: reduce cyclomatic complexity in RuntimeManager.Config.to_arg

e23dd9a

Replace case expression with function clause pattern matching to bring complexity under Credo's threshold.

Merge remote-tracking branch 'origin/main' into fastlanes

e6ae0dd

space removal

d50e652

taylordowns2000 approved these changes Mar 18, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Fast Lanes — dedicated worker capacity for named queues#4515

feat: Fast Lanes — dedicated worker capacity for named queues#4515
stuartc wants to merge 11 commits intomainfrom
fastlanes

stuartc commented Mar 9, 2026 •

edited

Loading

Uh oh!

codecov bot commented Mar 9, 2026 •

edited

Loading

Uh oh!

taylordowns2000 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

stuartc commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What's included

Data model (#4500)

Claim path (#4501)

Validation steps

Additional notes for the reviewer

AI Usage

Pre-submission checklist

Uh oh!

codecov bot commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

taylordowns2000 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stuartc commented Mar 9, 2026 •

edited

Loading

codecov bot commented Mar 9, 2026 •

edited

Loading