Screen-Sharing

2026

In-app Live Assistant : Part 2 --- Building the Screen-Sharing Version

6 April 2026·9 mins

AI Experiments Google-Adk Vertex-Ai Gemini Javascript Screen-Sharing Websocket Multimodal Ai-Agents

The ADK Dev UI supports camera but not screen sharing. In this post I build a custom client that swaps getUserMedia for getDisplayMedia, writes a 16kHz AudioWorklet from scratch, and sends 1 FPS JPEG screen snapshots to the same WebSocket endpoint. The agent now sees the application instead of the user’s face.

In-App Live Assistant

13 February 2026·3 mins

Open-Source Ai-Agents Multimodal Streaming Voice Screen-Sharing Xapi Learning-Measurement

This is a project I am actively pursuing: an open-source “in-app live assistant” pattern that other product teams can embed.

The core idea is simple: if the assistant can listen and see the current UI, it can give concrete next-step guidance instead of generic chatbot answers.