fix(integrations): google-genai: reworked gen_ai.request.messages extraction from parameters
#5275
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
… if mime_type and file_uri are present (Cursor comment)
…i-report-image-inputs
Semver Impact of This PR🟢 Patch (bug fixes) 📋 Changelog PreviewThis is how your changes will appear in the changelog. New Features ✨
Bug Fixes 🐛Integrations
Litellm
Other
Documentation 📚
Internal Changes 🔧Release
Other
🤖 This preview updates automatically when you update the PR. |
…i-report-image-inputs
| if isinstance(function_response, dict): | ||
| tool_call_id = function_response.get("id") | ||
| tool_name = function_response.get("name") | ||
| response_dict = function_response.get("response") or {} | ||
| # Prefer "output" key if present, otherwise use entire response | ||
| output = response_dict.get("output", response_dict) | ||
| else: | ||
| # FunctionResponse object | ||
| tool_call_id = getattr(function_response, "id", None) | ||
| tool_name = getattr(function_response, "name", None) | ||
| response_obj = getattr(function_response, "response", None) or {} |
There was a problem hiding this comment.
I've seen this .get() vs getattr pattern a lot in our AI integrations. Feels like introducing a helper function that would try both at once would potentially deduplicate a lot of code.
Not something that needs to be done in this PR, mostly thinking out loud.
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
…AI messages Add transform_content_part() and transform_message_content() functions to standardize content part handling across all AI integrations. These functions transform various SDK-specific formats (OpenAI, Anthropic, Google, LangChain) into a unified format: - blob: base64-encoded binary data - uri: URL references (including file URIs) - file: file ID references Also adds get_modality_from_mime_type() helper to infer content modality (image/audio/video/document) from MIME types.
…rmats Replace inline_data and file_data dict handling with the shared transform_content_part function. Keep Google SDK object handling and PIL.Image support local since those are Google-specific.
Add dedicated transform functions for each AI SDK: - transform_openai_content_part() for OpenAI/LiteLLM image_url format - transform_anthropic_content_part() for Anthropic image/document format - transform_google_content_part() for Google GenAI inline_data/file_data - transform_generic_content_part() for LangChain-style generic format Refactor transform_content_part() to be a heuristic dispatcher that detects the format and delegates to the appropriate specific function. This allows integrations to use the specific function directly for better performance and clarity, while maintaining backward compatibility through the dispatcher for frameworks that can receive any format. Added 38 new unit tests for the SDK-specific functions.
Replace generic transform_content_part with the Google-specific transform_google_content_part function for better performance and clarity since we know Google GenAI uses inline_data and file_data formats.
| if hasattr(part, "file_data") and part.file_data: | ||
| file_data = part.file_data | ||
| file_uri = getattr(file_data, "file_uri", None) | ||
| mime_type = getattr(file_data, "mime_type", None) | ||
| if file_uri and mime_type: | ||
| return { | ||
| "type": "blob", | ||
| "mime_type": mime_type, | ||
| "file_uri": file_uri, | ||
| } |
There was a problem hiding this comment.
Bug: The function _extract_part_content creates a non-standard format for file_data parts, using {"type": "blob", "file_uri": ...} instead of the expected {"type": "uri", "uri": ...}.
Severity: HIGH
Suggested Fix
Update _extract_part_content to produce the standardized format for file_data. It should return a dictionary with {"type": "uri", "uri": file_data.file_uri, "modality": ..., "mime_type": ...}. This can be achieved by reusing or replicating the logic from transform_google_content_part. The associated tests should also be updated to assert the correct format.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.
Location: sentry_sdk/integrations/google_genai/utils.py#L323-L332
Potential issue: When processing a `genai_types.Part` object containing `file_data`, the
`_extract_part_content` function generates a dictionary with `{"type": "blob",
"file_uri": ...}`. This is inconsistent with the standardized format `{"type": "uri",
"uri": ...}` used for dictionary-based parts. This inconsistency causes downstream
issues, specifically preventing the `redact_blob_message_parts` function from redacting
the URI, as it expects a `content` key for parts of type `blob`. The `modality` field is
also missing from the generated dictionary.
Did we get this right? 👍 / 👎 to inform future reviews.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
| "type": "blob", | ||
| "mime_type": mime_type, | ||
| "file_uri": file_uri, | ||
| } |
There was a problem hiding this comment.
Inconsistent output format between dict and Part object handling
Medium Severity
When handling file_data from Part objects, the code returns {"type": "blob", "file_uri": ...} but when handling the same data in dict format via transform_google_content_part, it returns {"type": "uri", "uri": ...}. Additionally, the Part object handling is missing the modality field that the standardized format includes. This inconsistency affects file_data handling (using "type": "blob" instead of "type": "uri", and "file_uri" instead of "uri"), inline_data handling (missing modality), and File object handling (same issues as file_data).
Description
Previously we only extracted only text parts were extracted. Now the full range of possibilities are covered.
Issues
Closes https://linear.app/getsentry/issue/TET-1638/redact-images-google-genai