Add image hard gate to bypass router for image requests#4203
Add image hard gate to bypass router for image requests#4203aashna wants to merge 1 commit intomicrosoft:mainfrom
Conversation
When a chat request contains an image attachment, bypass the model router entirely and use the heuristic-based path (_resolveWithoutRouterModel) which already handles vision-capable model selection via _applyVisionFallback. The router model is text-only and cannot reason about image content, so sending image requests through the router would produce unreliable routing decisions. Instead, the existing vision fallback heuristic is the correct path. Changes in automodeService.ts: - Check hasImage(chatRequest) before deciding to use router - When image detected and router configured, emit automode.imageHardGate telemetry event and fall through to heuristic path - usingRouterModel = routerConfigured && !requestHasImage Changes in routerDecisionFetcher.ts: - Add optional hasImage parameter to getRoutedModel() - Pass has_image: true in request body when set (defense-in-depth for server-side gating in auto-intent-service) Tests: - Add test: should send has_image flag when provided - Add test: should not send has_image when not provided
There was a problem hiding this comment.
Pull request overview
This PR adds an "image hard gate" to the AutomodeService so that chat requests containing image attachments bypass the router model entirely and use the existing heuristic-based path (_resolveWithoutRouterModel/_applyVisionFallback). The router is text-only and would produce unreliable routing decisions for image requests. As a secondary change, a hasImage flag is added to the RouterDecisionFetcher.getRoutedModel() API as a defense-in-depth mechanism to inform the server-side router of image presence.
Changes:
automodeService.ts: IntroducesrequestHasImagecheck before routing; when an image is detected and the router is configured, emits anautomode.imageHardGatetelemetry event and routes via the heuristic path instead.routerDecisionFetcher.ts: Adds an optionalhasImageparameter togetRoutedModel()that includeshas_image: truein the POST request body when set.routerDecisionFetcher.spec.ts: Adds two tests for the newhasImageparameter behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/platform/endpoint/node/automodeService.ts |
Adds image hard gate logic: detects images in the request, bypasses router, and emits telemetry |
src/platform/endpoint/node/routerDecisionFetcher.ts |
Adds optional hasImage parameter to getRoutedModel() to pass has_image in the request body |
src/platform/endpoint/node/test/routerDecisionFetcher.spec.ts |
New tests for has_image flag behavior in getRoutedModel() |
Two issues were identified:
-
Dead code in the defense-in-depth path (moderate): The
hasImageparameter added togetRoutedModelis never passed astruefrom production code. The hard gate (usingRouterModel = routerConfigured && !requestHasImage) guarantees that_resolveWithRouterModel— the only caller ofgetRoutedModel— is reached only whenrequestHasImageisfalse. As a result,has_image: trueis never included in any real router request body. For the stated defense-in-depth to function,requestHasImageneeds to be threaded through_resolveWithRouterModeland passed togetRoutedModel. The two new tests inrouterDecisionFetcher.spec.tsverify this unreachable scenario. -
Missing test coverage for the image hard gate in
automodeService.spec.ts(moderate): The existingresolveAutoModeEndpointtest suite covers several bypass scenarios (inline chat, terminal, no router URL), but no test verifies the new image hard gate path: a panel chat request with an image reference that hasmimeTypestarting with"image/"should bypass the router fetch and emit theautomode.imageHardGatetelemetry event.
|
I believe we've decided this one isn't needed as we will fallback to the client's image safety logic which seems to be working well |
When a chat request contains an image attachment, bypass the model router entirely and use the heuristic-based path (_resolveWithoutRouterModel) which already handles vision-capable model selection via _applyVisionFallback.
The router model is text-only and cannot reason about image content, so sending image requests through the router would produce unreliable routing decisions. Instead, the existing vision fallback heuristic is the correct path.
Changes in automodeService.ts:
Changes in routerDecisionFetcher.ts:
Tests: