video2sim

Film a tabletop with your phone and say what the robot should do. video2sim turns the clip into a MuJoCo scene for the SO-101 arm: SAM 3 segments every object in every frame, the objects are measured and placed, and the task is set up in simulation.

Repository on GitHub

Demo

Input: a 10-second phone clip. Command: “pick up cup and put on the napkin”.
Output: SAM 3 instance masks for the cup (red) and napkin (blue) in all 24 frames.

Screenshots

video2sim web app showing 48 of 48 masks segmented, with an overlay grid of frames
The local web app after a run: 48/48 masks (2 objects × 24 frames), segmented on a Colab GPU in about a minute.
Contact sheet of 24 frames with the cup masked red and the napkin masked blue
Every frame, including the cup cut off at the edge and overlapping the napkin.

How it works

  1. Upload a short phone video and type a command.
  2. Frames are extracted and the object names are read from the command.
  3. SAM 3, running on a Colab GPU, instance-segments every frame. The browser talks to it through Colab's private proxy, so nothing is exposed publicly.
  4. A vision model identifies the objects; the SAM 3 masks give their positions and sizes.
  5. A MuJoCo scene is built with the SO-101 arm, the objects and the goal, then rendered.