HIDAgent: A Platform-Agnostic Hardware Bridge for Personal Computer Use Agents (Video Preview)

HIDAgent: A Platform-Agnostic Hardware Bridge for Personal Computer Use Agents Jeffrey P Bigham UIST 2026 Video Preview for a UIST 2026 paper Computer Use Agents powered by performant VLMs are increasingly capable of visually interpreting GUIs and executing actions like humans do. However, their method of controlling UIs remains tethered to system-specific APIs or VNC connections, limiting agents to specific platforms and open OS environments. This paper introduces HIDAgent, a platform-agnostic hardware bridge that allows agents to observe any device via HDMI capture and control it via emulated HID mouse and keyboard events—all over a single physical connection. Built for less than $30 using off-the-shelf components, HIDAgent enables agents to operate across mobile, desktop, and even legacy or “closed” systems without local software installation. We demonstrate the flexibility of this approach through six diverse use case prototypes, ranging from cross-device automation to accessibility. By decoupling the agent from the host OS, HIDAgent facilitates research into a future where personal agents are mobile and independent of the devices they control. Web:: https://programs.sigchi.org/uist/2026/program/content/262858