Multimodal

The ADK Dev UI supports camera but not screen sharing. In this post I build a custom client that swaps getUserMedia for getDisplayMedia, writes a 16kHz AudioWorklet from scratch, and sends 1 FPS JPEG screen snapshots to the same WebSocket endpoint. The agent now sees the application instead of the user’s face.

I want to build a live AI assistant that can hear a user and see their screen at the same time, then talk back in real time. This post walks through getting Google’s ADK bidirectional streaming demo running locally with mic, camera, and voice out — and documents the things that tripped me up.

I tried to recreate a Google AI Studio proof-of-concept in code. The first path failed on authentication; the second produced a working bidirectional streaming prototype; the third pushed it toward a realistic scenario.