Embodied Robotics Training
Scan-to-simulation pipelines for robot validation and deployment
For most of their history, robots have been programmed move by move. An engineer specifies each motion in advance, and the machine executes a script it cannot deviate from. That approach is brittle. It holds only as long as the world matches the plan, and the world rarely does.
Embodiment changes the premise. When an AI is given a body, it gains senses, sight, hearing, proprioception, a spatial model of its surroundings, and with them the ability to learn the way living things do: by perceiving a situation, acting, observing the result, and adjusting. The engineering question shifts from how to specify every motion to how to give an agent enough experience to develop judgment of its own. Experience is the bottleneck, and the physical world is a poor place to gather it. It runs in real time, it is expensive to instrument, and failure can destroy equipment or hurt people.
Simulation removes that bottleneck, and not only by being safe. It can run faster than real time. An embodied agent can live through thousands of hours of scenarios in minutes, encountering rare events, edge cases, and dangerous conditions far more often than it ever would on site. Each run leaves the agent with synthetic experience: perceptual data, memories, and decision traces tied to specific real-world actions. This is how an agent builds intuition for a task before it has performed it once in reality, and how it accumulates the volume of experience that learned behavior requires.
The fidelity of that experience is the whole game, which is where our work lives. We build high-fidelity digital twins of real environments, scanning facilities with LiDAR and photogrammetry and converting the data through 3D Gaussian splatting and OpenUSD into real-time worlds in NVIDIA Omniverse and Isaac Sim. The closer the simulated light, geometry, and physics come to reality, the more cleanly what an agent learns in the twin transfers to the machine on the floor. A training ground for the senses is only as good as the senses it can faithfully reproduce.
This sits at the center of a conviction that has organized my work for years: creative visualization is research infrastructure. The same capture and rendering techniques that build believable worlds for film and performance now build the worlds where machines learn to perceive, decide, and act.