the goal was to make the agent’s computer feel local. when you take over, clicking, typing, and scrolling should feel natural, even though the computer is running in a cloud sandbox.
every click has to reach that sandbox, and the result has to stream back to your browser. we’ve been cutting the input and streaming delay we add along the way. in the benchmark below, replicas measured 127.4 ms median click-to-visible response, including the network trip.
it got to the point where i could get a kill in slither.io, watch tft, and hit 128 wpm on monkeytype, with replicas itself running locally in the same cloud sandbox. here’s the video:
we started by moving video and audio to webrtc and giving input a more direct path to the sandbox. then we worked on everything around it: preventing old frames from piling up, sharing encoding work, and recovering connections without interrupting a healthy stream.
we then rewrote the capture process and benchmarked versions in C, Go, and Rust. across six local runs per language, all three delivered about 60 visible updates per second. Rust had the lowest measured input latency, while C used slightly less CPU and memory. we chose Go because i knew it better and could maintain it more easily. the small performance differences didn’t justify taking on a language i was less comfortable changing.
the bigger change was controlling when we captured the screen and how much of it we processed. the old path used ffmpeg to capture and encode whole frames. our Go implementation reuses unchanged parts of the screen and cuts work while the desktop is idle, with the heavy pixel conversion and encoding still handled by native libraries. we also reduced unnecessary browser rendering work and fixed cases where a quiet or hidden desktop was mistaken for a struggling connection.
we also worked on how agents use the browser. they can now group actions into a short script, find controls by their labels, and check the result before continuing. in one recorded comparison, the same model signed in, configured an environment, and sent a workspace message with 22 browser calls instead of 45. both attempts succeeded; the new one finished in 4m 7s instead of 5m 45s. those times include different provisioning waits.
an agent using the computer also reached #1 on agar.io.
to compare the streams, we opened the same animated page in fresh replicas, cursor, and devin sessions. fps here means new animation frames shown each second. repeated copies of the same frame don’t count. we also measured the time between new frames and how long a click took to appear. all runs used one linux chrome client with software graphics.
| Metric | Replicas 0.3.9 | Cursor | Devin |
|---|---|---|---|
| New frames shown per second (FPS) ↑ | 55.7 | 49.8 | 18.0 |
| p95 gap between new frames ↓ | 29.3 ms | 36.6 ms | ≈182.8 ms |
| Median click-to-visible response ↓ | 127.4 ms | 126.4 ms | 339.6 ms |
replicas streamed at 1920×1080 (1080p), cursor at 1920×1200, and devin at 1600×1200. cursor was carrying more pixels.
replicas delivered 12% more frames per second than cursor and the shortest p95 frame gap of the three. median click latency was effectively tied with cursor and about 62% lower than devin in this test.
our first measurement code could slow devin’s viewer. we fixed that, checked the frame counts against its own viewer statistics, and reran all 40 clicks. every click matched a visible response.
see the benchmark setup
1080p stream, computer-use 0.3.9.
1920×1200 desktop, control enabled.
1600×1200 desktop, native counters visible.
i want taking over an agent’s computer to feel normal. being able to play slither in it is a nice side effect.