Skip to content

edge: add JSON/CSV/Unix-socket output sinks#3523

Draft
armcconnell wants to merge 1 commit intoedge/1-parser-frameworkfrom
edge/2-sinks
Draft

edge: add JSON/CSV/Unix-socket output sinks#3523
armcconnell wants to merge 1 commit intoedge/1-parser-frameworkfrom
edge/2-sinks

Conversation

@armcconnell
Copy link
Copy Markdown
Contributor

Part 2 of 4 — edge feed parser stack

  1. edge: add Top-of-Book parser framework #3522 — edge: add Top-of-Book parser framework ← start here
  2. this PR — edge: add JSON/CSV/Unix-socket output sinks
  3. edge: add feed runner, manager, HTTP API, and CLI (coming)
  4. edge: add integration tests, synthetic publisher, and devnet guide (coming)

Draft until #3522 lands. Reviewers welcome to read for context; I'll mark ready-for-review after rebasing onto main.

Summary

  • Introduces the `OutputSink` interface and three implementations: JSON-lines file, CSV file (with header row auto-inferred from the first record's shape), and Unix-socket broadcast with drop-on-slow-consumer semantics so a stuck reader can't backpressure the parser.
  • `NewSink` factory keyed on format + path; paths beginning with `unix://` route to the socket sink, others to file sinks.
  • CSV sink pivots the parser's flat `Fields` map onto a stable column set so heterogeneous record types (quotes, trades, instrument definitions, heartbeats) line up in one sheet.
  • The parser still knows nothing about sinks — these are pure consumers of `[]Record` produced by the parser from edge: add Top-of-Book parser framework #3522.

Testing Verification

  • Unit tests cover each sink: CSV header inference + column-pivot, JSON line format, socket broadcast to multiple connected clients, and slow-consumer drop behavior.

Introduces the OutputSink interface and three implementations:

  - JSONFileSink  — JSON lines to a file
  - CSVFileSink   — CSV with a header row inferred from the first
    record; rows pivot the parser's flat Fields map onto stable
    columns so every record type lines up in one sheet
  - SocketSink    — broadcast to all clients connected to a Unix
    domain socket, with drop-on-slow-consumer semantics so a stuck
    reader can't backpressure the parser

NewSink is a factory keyed on Format ("json"/"csv") and Path. Paths
starting with "unix://" route to SocketSink; everything else is a
regular file.

Sinks take []Record from the parser (introduced in the previous
commit). The parser still knows nothing about sinks — this package
stays a decode library.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant