1/4 Replace realtime websocket transport with WebRTC#16805
Open
aibrahim-oai wants to merge 20 commits intomainfrom
Open
1/4 Replace realtime websocket transport with WebRTC#16805aibrahim-oai wants to merge 20 commits intomainfrom
aibrahim-oai wants to merge 20 commits intomainfrom
Conversation
Move the realtime model transport implementation to WebRTC while keeping the core session/event interface intact for the TUI layer. Co-authored-by: Codex <noreply@openai.com>
Install cmake on Linux and macOS Bazel runners so audiopus_sys build scripts can build bundled Opus. Co-authored-by: Codex <noreply@openai.com>
Link audiopus_sys against the Bazel opus module so remote Bazel builds do not depend on host cmake. Co-authored-by: Codex <noreply@openai.com>
Teach the shared test helper to accept realtime /calls POSTs, answer SDP offers, and relay scripted events over a data channel while logging incoming RTP audio packets as synthetic append requests. Update the one stale handshake-path assertion to the /realtime/calls URL. Co-authored-by: Codex <noreply@openai.com>
Ignore the workspace `opus` dependency in cargo-shear for core_test_support because Rust imports that package as `audiopus`. Co-authored-by: Codex <noreply@openai.com>
Include nested response helper modules in the core_test_support Bazel target so the new realtime WebRTC test server source is visible to macOS Bazel and argument-comment-lint jobs. Co-authored-by: Codex <noreply@openai.com>
Explicitly include first-level response helper modules in the core_test_support Bazel target so realtime_webrtc_server.rs is available to macOS builds and lints. Co-authored-by: Codex <noreply@openai.com>
Use the workspace opus crate name directly and clone the request sender before moving it into the data-channel callback so the RTP track handler can still enqueue decoded packets. Co-authored-by: Codex <noreply@openai.com>
Preserve item_id on v1 audio deltas and annotate opaque None arguments in calls URL tests so argument-comment lint passes. Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
aibrahim-oai
changed the title
This was referenced Apr 4, 2026
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
- switch realtime transport and test server to opus-rs - drop native opus bazel and cmake plumbing Co-authored-by: Codex <noreply@openai.com>
- use valid 24 kHz mono PCM audio in realtime tests - keep websocket/WebRTC request sequencing aligned with transport behavior Co-authored-by: Codex <noreply@openai.com>
- replace silent PCM realtime test fixtures with a deterministic tone - avoid codec paths optimizing away audio in WebRTC test flows Co-authored-by: Codex <noreply@openai.com>
Stop the realtime test WebRTC server from hanging after the client closes. Teach the scripted request loop to exit when the data channel or peer connection closes so compact-remote tests can unwind instead of timing out at shutdown. Co-authored-by: Codex <noreply@openai.com>
Keep the first realtime audio frame from being dropped during WebRTC startup. Also give the realtime test server a tiny flush window before it closes the data channel so the last scripted events land reliably. Co-authored-by: Codex <noreply@openai.com>
Install the rustls crypto provider in the realtime WebRTC test server so the app-server realtime tests stop panicking, and update the core handshake assertion to match the current realtime calls URL. Co-authored-by: Codex <noreply@openai.com>
Collaborator
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8207ff6ead
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sample_rate: 24_000, | ||
| num_channels: REALTIME_AUDIO_CHANNELS, | ||
| samples_per_channel: Some(samples_per_channel as u32), | ||
| item_id: None, |
Contributor
There was a problem hiding this comment.
Do not hardcode item_id to None when emitting RealtimeEvent::AudioOut from RTP. core/src/realtime_conversation.rs only tracks output audio for truncation when item_id is present; frames without it are ignored, so V2 speech-start interruption cannot compute audio_end_ms or send conversation.item.truncate reliably.
Useful? React with 👍 / 👎.
Comment on lines
+759
to
+765
| let packet = match track.read_rtp().await { | ||
| Ok((packet, _)) => packet, | ||
| Err(err) => { | ||
| let _ = tx_message.send(Err(ApiError::Stream(format!( | ||
| "failed to read realtime WebRTC audio packet: {err}" | ||
| )))); | ||
| return; |
Contributor
There was a problem hiding this comment.
track.read_rtp() errors are always forwarded as fatal ApiError. On normal connection teardown, this can race with Closed notifications and make next_event() return an error instead of clean end-of-stream. That misclassifies graceful closes as failures and can trigger unnecessary error handling in callers.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack position: 1/4
Stack:
This PR: