SIBNIK turns VR teleoperation of real hardware into structured training data for robot learning — collecting diverse demonstrations at near-zero marginal cost per scene variation.
Modern imitation-learning policies (ACT, Diffusion Policy, VLAs) are only as good as the demonstrations they're trained on — and collecting those demonstrations on real hardware is slow, expensive, and hard to scale across scene variation.
A human teleoperates a real robot arm through a VR headset and controllers, giving policies demonstrations grounded in true physical dynamics — not sim-only approximations.
A software layer handles interaction, kinematics, and scene variation; the robot handles reality. That split is what makes scaling demonstration collection tractable.
Every session is captured in LeRobot format — multi-camera RGB, joint states, and actions — ready to drop into ACT, SmolVLA, or other imitation-learning pipelines.
Joint limits, slew-rate limiting, a deadman switch, and a watchdog sit between the VR input and the arm at all times.
The current pipeline runs end-to-end on a Franka Panda 7-DOF arm.
A custom VR front end drives the Franka Panda with full kinematics, inverse kinematics, and an intuitive manipulation gizmo for precise control.
A custom MuJoCo bridge over ZeroMQ streams commands to the physical robot in real time.
Two static RealSense cameras record synchronized RGB views alongside joint state and action data.
Every recorded episode is replayed open-loop as a mandatory gate before it's counted toward a training set.
Episodes are assembled into LeRobot-format datasets and used to fine-tune SmolVLA / ACT policies, starting from a target of 50 demonstrations per task.
A Unitree G1–style humanoid, running live in your browser. Walk it, turn it, and grab a hand to pose its arm. Five cameras — head, both wrists and two tracking views — stream what it sees, the same multi-view setup SIBNIK records as demonstration data.
The near-zero marginal cost of generating a new scene variation in software is the whole point: it's what lets demonstration collection scale the way robot learning needs it to. The Franka Panda is the proving ground — humanoid robotics is the target embodiment.
PhD student in robotics at the University of Waterloo, working out of RoboHub. Roughly three years in industry as a VR/XR developer, spanning VR, multiplayer, ML integration, and advanced rendering. SIBNIK is built independently — outside of lab time and lab equipment.
Interested in the approach, collaborating, or just want to talk robot learning.