Skip to content

Worker: Add --queues option and slot group configuration parsing #1288

@stuartc

Description

@stuartc

Summary

Add a new --queues CLI option to the worker that lets operators divide
worker capacity into slot groups, each with its own queue preference chain.
This issue covers configuration parsing and validation only — the runtime
behavior (per-group workloops) is a separate issue.

Background

Currently the worker has a single --capacity flag (default 5) that controls
the total number of concurrent runs. All runs are claimed from a single global
queue with no differentiation.

The fast lanes feature requires workers to dedicate specific slots to specific
queues. For example: 1 slot pinned to sync webhook work, 4 slots for general
work.

Configuration syntax

--queues "<preference>:<count> <preference>:<count> ..."
Delimiter Meaning
, Queue preference separator (left = highest preference)
* Wildcard: accept runs from any queue
:N Slot count for this preference group
(space) Slot group separator

Examples:

# 1 slot pinned to fast_lane, 4 slots preferring manual then anything
--queues "fast_lane:1 manual,*:4"

# 5 generic slots (pure FIFO across all queues)
--queues "*:5"

# 2 fast lane (strict), 3 with manual preference
--queues "fast_lane:2 manual,*:3"

Environment variable: WORKER_QUEUES

Slot group data structure

Each parsed group should produce something like:

interface SlotGroup {
  queues: string[];     // e.g., ["fast_lane"] or ["manual", "*"]
  maxSlots: number;     // e.g., 1
}

Backward compatibility

  • --queues and --capacity are mutually exclusive — specifying both is
    an error
  • If neither is provided, default to --capacity 5 behavior
  • When --queues is not provided, derive from --capacity:
    manual,*:<capacity> (e.g., capacity 5 → "manual,*:5")
  • This preserves current behavior where manual/immediate runs get preference

Validation rules

  • Each slot group must have count >= 1
  • Queue names must be non-empty strings (alphanumeric + underscore)
  • * can only appear as the last element in a preference chain
  • Total slot count across all groups becomes the effective capacity
  • --queues and --capacity are mutually exclusive

Protocol type update

Update ClaimPayload in the lexicon types to include the optional queues
field:

interface ClaimPayload {
  demand?: number;
  worker_name?: string;
  queues?: string[];  // NEW: queue preference chain
}

Testing

  • Parse "fast_lane:1 manual,*:4" → two slot groups with correct queues and counts
  • Parse "*:5" → one slot group with ["*"] and 5 slots
  • Parse "fast_lane,manual,*:1 *:4" → correct multi-preference chain
  • Reject "*,fast_lane:1" (wildcard not at end)
  • Reject ":0" or ":abc" (invalid counts)
  • Reject conflicting --queues and --capacity flags
  • Default derivation: capacity 5 with no --queues[{ queues: ["manual", "*"], maxSlots: 5 }]

Metadata

Metadata

Assignees

Type

No type

Projects

Status

In review

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions