Skip to content

edge: add Top-of-Book parser framework#3522

Open
armcconnell wants to merge 1 commit intomainfrom
edge/1-parser-framework
Open

edge: add Top-of-Book parser framework#3522
armcconnell wants to merge 1 commit intomainfrom
edge/1-parser-framework

Conversation

@armcconnell
Copy link
Copy Markdown
Contributor

Summary

  • Introduces the edge package with a pluggable Parser interface, a parser registry, and the first implementation: a decoder for the DoubleZero Top-of-Book feed v0.1.0 wire format.
  • Parser is stateful: it learns instrument definitions from refdata messages and uses them to enrich marketdata (quote/trade) messages with symbol, price/qty exponents, and derived floats. Marketdata arriving before its definition is held in a bounded per-instrument buffer (most-recent-wins, capped at 1000 instruments) and flushed the moment the definition lands, so a cold-start reader still produces meaningful output at the first refdata cycle.
  • Wire decoder is a hand-written positional reader over the fixed little-endian format using `encoding/binary` and a small sticky-error reader helper. No third-party deps, no code generation.
  • Emits generic `Record` values; intentionally decoupled from output sinks, which land in the next PR.

Part 1 of 4 — edge feed parser stack

  1. this PR — edge: add Top-of-Book parser framework ← start here
  2. edge: add JSON/CSV/Unix-socket output sinks
  3. edge: add feed runner, manager, HTTP API, and CLI
  4. edge: add integration tests, synthetic publisher, and devnet guide

Each PR compiles and tests standalone. Feature is off-by-default until PR 3 lands; early PRs are dormant code.

Testing Verification

  • Unit tests cover wire decode for each message type, bounded-buffer overwrite semantics, cap-reached drops, and refdata-triggered flush.
  • Also exercised against real testnet Top-of-Book traffic on a QA node — processed 1M+ records end-to-end without buffer growth or parse errors.

Size note

~1000 lines of production code, above doublezero's ~500-line guideline. The stack already splits the feature into 4 PRs; the parser itself doesn't subdivide further without creating non-functional intermediate states. Bulk of the diff is the wire decoder (~340 LOC, straight-line positional reads) and the parser state machine (~600 LOC), both of which benefit from being reviewed together.

Introduces the edge feed parser package with a pluggable Parser
interface, a registry for wiring parsers by name, and the first
implementation: a decoder for the DoubleZero Top-of-Book feed
v0.1.0 wire format.

The parser is stateful: it holds instrument definitions from refdata
messages and uses them to decode quote/trade messages in marketdata
messages. Marketdata messages arriving before their instrument
definition are held in a bounded per-instrument buffer
(maxBufferedInstruments = 1000, most-recent-wins) and flushed the
moment the definition lands, so a cold-start reader still produces
meaningful output at the first refdata cycle.

The wire decoder in topofbook_wire.go is a straightforward
positional reader over the fixed-layout little-endian format:
header validates magic bytes and frame length; each app message
dispatches on msg_type to one of seven body decoders; unknown
types are skipped rather than erroring. A sticky-error wireReader
helper keeps the decoder free of intermediate error checks.

The parser is intentionally decoupled from output — it emits
generic Record values that a separate sink layer (landed in a
follow-up PR) consumes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant