Skip to content

Engine throws uncaught exception if logs don't send #1310

@josephjclark

Description

@josephjclark

Maybe

Trying to understand this stack trace which caused a worker exist

[RTE] ❯ pool: finished task in worker 1960
[RTE] ℹ Error occurred during compilation
[SRV] ℹ e6d86202-d371-4033-adc4-61f59b345e52 :: sent workflow-error :: OK :: 105ms
[SRV] ✘ 8a1c978e-bfb4-4332-9488-972faf24b7a0 :: run:log :: ERR: [run:log] timeout
file:///app/packages/ws-worker/dist/start.js:703
        report(new LightningTimeoutError(event));
               ^

LightningTimeoutError: [run:log] timeout
    at Object.callback (file:///app/packages/ws-worker/dist/start.js:703:16)
    at file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:109:71
    at Array.forEach (<anonymous>)
    at Push.matchReceive (file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:109:54)
    at Object.callback (file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:140:12)
    at Channel.trigger (file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:460:12)
    at Push.trigger (file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:156:18)
    at Timeout.<anonymous> (file:///app/node_modules/.pnpm/phoenix@1.7.10/node_modules/phoenix/priv/static/phoenix.mjs:143:12)
    at listOnTimeout (node:internal/timers:588:17)
    at process.processTimers (node:internal/timers:523:7) {
  reportedToSentry: true
}

Node.js v22.22.0
  • In the engine execute.ts, the await in await log() on error is superflous. Those engine events aren't async (we don't wait for events to be return)
  • It looks like the workflow throws an error, and then blows up trying to send a log line
  • If the await log in the engine throws, it'll kill the engine I think
  • But then, that wouldn't throw. The throw is coming out of the worker
  • We do see the engine log Error occurred during compilation to stdout. And presumably the worker received that log event and is trying iprocess it
  • maybe because the run technically didn't start, the throw on that log event is somehow not getting caught?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions