Allow simultaneous inputs on AIChat

[deathbyknowledge]

Fixes #24: AIChatAgent now handles multiple messages arriving simultaneously by queuing them and processing one at a time.

Problem

When multiple chat messages arrived before the first response completed, the onChatMessage → _reply cycles would race, causing:

• Interleaved streaming responses
• Potential message corruption
• Unpredictable behavior

Solution

Added a request queue that serializes chat message processing:

• _chatRequestQueue — Array of pending work functions
• _chatRequestInFlight — Guard to prevent concurrent processing
• _enqueueChatRequest() — Adds work to queue, kicks off drain if idle
• _drainChatRequestQueue() — Processes queue items sequentially

Changes

• Queue mechanism for serializing onChatMessage calls
• Tool result continuations now use the queue instead of waitForStream() + 500ms timeout hack
• Tool approval continuations similarly refactored
• Removed dead _streamCompletionPromise / _streamCompletionResolve code (no longer needed)
• Queue cleared on destroy() and CF_AGENT_CHAT_CLEAR

[changeset-bot]

🦋 Changeset detected

Latest commit: 94b0601

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@cloudflare/ai-chat	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/cloudflare/agents@934

npm i https://pkg.pr.new/cloudflare/agents/@cloudflare/ai-chat@934

npm i https://pkg.pr.new/cloudflare/agents/@cloudflare/codemode@934

npm i https://pkg.pr.new/cloudflare/agents/hono-agents@934

commit: 94b0601

[deathbyknowledge]

/bonk review pretty please

[ask-bonk]

remote: Permission to cloudflare/agents.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/cloudflare/agents/': The requested URL returned error: 403

github run

[ask-bonk]

Bonk workflow failed. Check the logs for details.

View workflow run

[ask-bonk]

I now have a thorough understanding of the PR. Let me write up my review.

---

PR Review: Allow simultaneous inputs on AIChat

Summary

This PR replaces the _streamCompletionPromise/_streamCompletionResolve mechanism with a request queue (_chatRequestQueue + _enqueueChatRequest + _drainChatRequestQueue) that serializes all onChatMessage → _reply cycles. This is a solid architectural improvement.

What's Good

1. The core queue design is clean and correct. The _enqueueChatRequest / _drainChatRequestQueue pattern is a well-understood serialization approach. The while loop with shift() in _drainChatRequestQueue naturally handles items added during processing.

2. Eliminates the 500ms timeout hack. The old code had a fragile setTimeout(resolve, 500) fallback when _streamCompletionPromise was null (a known race). The queue approach is strictly better because it serializes by construction — no timers needed.

3. Consistent application. All three call sites (initial chat request, tool result continuation, tool approval continuation) use the same _enqueueChatRequest path, making the control flow uniform and easier to reason about.

4. Proper cleanup. Queue is cleared on destroy() and CF_AGENT_CHAT_CLEAR, matching the lifecycle of other state.

Issues

1. Fire-and-forget _enqueueChatRequest — uncaught rejections if DO hibernates (Medium)

packages/ai-chat/src/index.ts:433

The chat request handler now calls this._enqueueChatRequest(...) and immediately returns. The _drainChatRequestQueue method is async but its returned promise is never awaited or passed to ctx.waitUntil(). The comment says "the queue drains via ctx.waitUntil" but I don't see that actually happening.

In the old code, the handler returned the result of this._tryCatchChat(...), which meant the Durable Object's WebSocket handler awaited the full reply. Now, the onMessage handler returns undefined while _drainChatRequestQueue runs in the background. If the DO hibernates or the isolate is evicted mid-drain, queued work silently disappears.

The same issue applies to tool result and tool approval continuations — they were already fire-and-forget via .then(), so this isn't a regression there, but the initial chat request path is a regression.

Suggestion: Either:

• Have _enqueueChatRequest call this.ctx.waitUntil(drainPromise) so the runtime knows work is outstanding, or
• Return the drain promise from the onMessage handler for the initial chat request case.

2. _drainChatRequestQueue is called but not awaited (Low)

packages/ai-chat/src/index.ts:739

if (!this._chatRequestInFlight) {
  this._drainChatRequestQueue(); // async, not awaited
}

This is intentional (fire-and-forget kickoff), but combined with issue #1, it means there's no back-pressure. If the caller doesn't hold the promise, the only thing keeping the drain alive is the event loop. Consider storing the drain promise so destroy() can await it for graceful shutdown.

3. Stale connection closure in queued callbacks (Low-Medium)

packages/ai-chat/src/index.ts:433-478

The queued callback captures connection from the enclosing onMessage scope. If multiple messages arrive from different connections, each queued callback closes over its respective connection. This is correct for preserving the sender's identity, but worth noting: if a connection disconnects while its request is still queued, the agentContext.run will execute with a stale connection reference. The abortSignal won't fire because cancellation requires an explicit CF_AGENT_CHAT_REQUEST_CANCEL message.

This isn't a bug per se (the old code had the same behavior with a single in-flight request), but the queue makes it more likely since requests can wait longer. Consider checking connection.readyState before executing a queued callback, or wiring up the abort signal to connection close.

4. Missing changeset (Low)

The changeset bot flagged this — this is a behavioral change to @cloudflare/ai-chat (fixes a race condition) and should have a patch changeset.

Nits

• The → in the JSDoc for _enqueueChatRequest (onChatMessage → _reply cycle) was changed from -> in the diff. Either is fine, but keep it consistent with the rest of the codebase (the rest uses -> or plain English).

Missing Test Coverage

There are no tests for the actual queueing behavior — i.e., verifying that two concurrent chat requests are processed sequentially rather than interleaved. The existing client-tools-continuation.test.ts exercises the continuation path but doesn't send overlapping requests. A test that sends two CF_AGENT_USE_CHAT_REQUEST messages back-to-back and asserts both responses complete in order would be valuable.

Verdict

The approach is correct and the code is clean. The main concern is issue #1 — the initial chat request is now fire-and-forget where it wasn't before, which could cause silent work loss on hibernation. If that's addressed (even just a ctx.waitUntil), this is good to merge with a patch changeset.

github run

[elithrar]

/bonk review pretty please

[ask-bonk]

I read the PR diff, changeset file, and key sections of packages/ai-chat/src/index.ts (the queue implementation, destroy method, cancel logic, and onMessage handler) along with the existing test files to understand the full context, then wrote up a review confirming the queue approach is correct while flagging that the ctx.waitUntil comment is misleading since no such call exists, making the drain fire-and-forget and potentially lossy on hibernation.

github run

[threepointone]

May I please have some tests

[threepointone]

should we update the ai-chat example for this? what's the ux like? also is this behaviour alsways desirable / should it be a flag?

[deathbyknowledge]

May I please have some tests

Donezo. Added one to check response order of new messages and one to make sure the queue empties on clear