Agent Engine + ADK + Gemini Live freezes for long responses

[mpuxted]

Environment details

• OS type and version:
• Python version: python --version
• pip version: pip --version
• google-cloud-aiplatform version: pip show google-cloud-aiplatform

This issue is replicable on the current latest versions of ADK (1.23.0) and google-cloud-aiplatform (1.135.0), but I have created a copy of the bidi streaming colab that can replicate the issue. Due to colab dependency requirements, I had to pin some libraries to earlier versions

Steps to reproduce

1. Open this gist
2. Click Open in Colab
3. I believe my latest run will still show, which means you can scroll to the bottom and hit play on the last audio to see the result of the bug (audio cut off)

Otherwise, you should be able to run the colab yourself by just entering in a project ID and a bucket name.

Once it is deployed, just ask "What are all the sites?" and the majority of the time it should only send 4s~ audio instead of the full list of sites (usually cutting out part-way through a site name).

The UI for the colab is a bit buggy, but you should be able to replicate the issue nonetheless.

I think the only thing required to make this happen are:

1. A large prompt
2. A tool that returns a large amount of data

For good measure, I have thrown in a 5 second delay into the tool, to closer simulate my real example where we call a slow API, however I think this delay can be removed and the bug will still remain.

Workarounds

I know of no fix for this solution, but one thing of interest is if you say "carry on" to the AI, it will continue from where it left off, so it is aware that it has given the whole list.

I've tried the following:

1. Setting the scaling settings on agent engine to be the maximum
2. Prompting it to "always say the entire list of sites"
3. Hosting the ADK agent in Cloud Run instead of agent engine