Vertex AI Model Garden: attachment capability lookup fails due to provider mismatch

[simonferquel-clanker]

Problem

pkg/model/provider/vertexai/modelgarden.go routes non-Gemini models by creating an openai.Client with the original ModelConfig (Provider = "google"). When this client calls oaistream.ConvertMessages, it passes modelcaps.NewID("google", modelName), e.g. "google/anthropic.claude-3-5-sonnet-20241022-v2:0".

models.dev does not have entries under the "google" provider for Anthropic/Meta/Mistral models — they are listed under their own providers ("anthropic", "meta", etc.). The lookup therefore always misses, and vision/PDF attachments are silently dropped.

Example

A Vertex AI Model Garden config with publisher: anthropic and model: claude-3-5-sonnet-20241022 should resolve capabilities from anthropic/claude-3-5-sonnet-20241022, not google/claude-3-5-sonnet-20241022.

Possible fixes

• In the Model Garden client setup, remap the ModelConfig.Provider to match the publisher before constructing the underlying openai.Client or anthropic.Client
• Add a capability-override mechanism in the agent config (e.g. explicit modality declarations)
• Add Vertex AI model aliases to a local lookup table mapping Vertex model IDs to their models.dev equivalents

Context

Identified during review of PR #2738 (fix: pass fully-qualified provider/model ID to modelcaps.Load).