*Equal contribution · Improbable AI Lab, MIT
Joga is a full-stack system for humanoid dribbling, shooting, passing, and receiving. A torso-mounted actuated neck gives the robot active vision, and data-driven alignment of perception and dynamics preserves the speed and precision learned in simulation when the policies run on hardware.
2.31 m/s
peak dribbling speed on hardware
10 min
of hardware data per alignment model
5/5
real-world dribble-to-pass and dribble-to-shoot transitions
Exhibiting human-like agility in object interaction is a long-standing goal in humanoid robotics. Soccer offers a demanding case study, requiring dynamic coordination of perception, locomotion, and foot-ball contacts. Reinforcement learning in simulation is a promising approach for learning agile object interaction because it can directly optimize task outcomes, but sim-to-real gaps in perception and dynamics make transfer to reality challenging. We introduce Joga (Joint Optimization of Gaze and Action), a full-stack system for humanoid dribbling, shooting, passing, and receiving that perceives the ball through an RGB camera mounted on a custom 2-DoF actuated neck. Our approach combines hardware design with data-driven model alignment to close the sim-to-real gap. The torso-mounted actuated neck enables rapid and stable active vision, while a learned perception alignment model learns the distributional bias and noise of onboard ball observations to achieve the precision needed for sharp turns and dribbling at high speeds. For dribbling, we further learn a whole-body residual action model from 10 minutes of hardware data and use the corrected simulator to finetune the policy, improving its speed. On hardware, Joga reaches a peak dribbling speed of 2.31 m/s while maintaining more ball controllability than domain-randomization and linear-perception-model baselines. The learned skills can be chained, transitioning from controlled dribbling to shooting or passing, and from receiving an incoming pass to dribbling. Together, these results show how limited real-world data and targeted modeling can preserve agile humanoid object interaction learned entirely in simulation.
Real-world skills
On a Unitree G1, each skill policy jointly controls the whole body and the neck, coordinating where the robot looks with how it moves. Dribbling repeatedly redirects a moving ball while locomoting; shooting and passing impart directed impulses; receiving brings an incoming ball under control so it can be kept for subsequent play. At deployment the robot relies only on its onboard camera and proprioception.
The problem
Reinforcement learning lets whole-body strategies emerge from task objectives such as “dribble forward at 2 m/s” or “pass 30° right at 5 m/s,” and parallel simulation makes the required trial and error practical. But the simulator is not the robot. The ball must stay in view as the robot accelerates and turns, and even when it is visible, onboard detections carry bias and noise that depend on both the ball's position and the robot's motion. On the actuation side, parallel-linkage mechanisms introduce configuration-dependent coupling and reflected inertia, and shared power limits can make available torque depend on whole-body demand.
Domain randomization can tolerate these discrepancies, but overly broad randomization favors slow, conservative strategies that work across many simulated systems rather than the one real robot. Joga instead improves the fidelity of the nominal system using limited real-world data, and keeps targeted randomization only for remaining variability in terrain, ball, and scene.
How can we preserve the speed, precision, and fluidity learned in simulation when a humanoid plays soccer in the real world?
The system
A custom torso-mounted 2-DoF neck carries an RGB camera for rapid, stable gaze changes during fast locomotion.
Learn the state-dependent bias, noise, and dropout of onboard ball detection from paired detector and motion-capture data.
Learn joint-position command corrections from 10 minutes of hardware dribbling, then finetune the dribbling policy in the corrected simulator.
Finetune passing and shooting from states visited during dribbling so skills hand off mid-stride.
The neck mounts directly to the torso and is driven by two DYNAMIXEL XC330-M288-T servos. It deliberately avoids external linkages, gears, and bearings: more complex transmissions introduce backlash, compliance, slippage, and calibration error, which slow the neck down and widen the sim-to-real gap. The simple design gives a large range of motion and speed for tracking the ball through robot and ball motion, and after a simple system identification it needs no neck domain randomization during policy training.
Perception alignment
The policy observes the ball position back-projected from the detector's 2D bounding box through a pinhole model. Despite intrinsic and hand-eye calibration, this estimate carries systematic, state-dependent errors that simulation lacks: depth is biased because the camera is not a perfect pinhole, noise grows with range and extrinsic mismatch, and the detector drops out at a range-dependent rate.
The Perception Alignment Model (PAM) replaces perception domain randomization entirely. Working in bounding-box space, so it mirrors the real vision pipeline, it predicts a heteroscedastic Gaussian over the box residual plus a visibility probability, conditioned on the true ball position, base angular velocity, neck and waist joint velocities, projected gravity, and the ball position in the base frame. PAM is trained on 10 minutes of paired onboard detections and motion-capture labels, then frozen and sampled at every control step during policy training.
The learned error structure is far from isotropic: depth bias is about +5 cm at dribbling range and changes sign beyond roughly 2 m, noise grows with both range and ego-motion, and dropout rises from a few percent at close range to over 30% at 2.5 m.
| Policy | Success |
|---|---|
| DR | 3/4 |
| Linear | 1/4 |
| PAM | 4/4 |
To stress-test the benefit, we train policies in a base simulator and deploy them in a target simulator with a large perception gap: a miscalibrated pinhole model, a 5% focal-length error, a principal-point shift, lens distortion, and a small extrinsic offset. The PAM policy nearly matches an oracle trained directly in the target environment, while DR and linear-model policies degrade as the course gets harder.
| Policy | Easy (45°) | Medium (90°) | Hard (180°) |
|---|---|---|---|
| DR | 77.5 ± 0.5 | 53.0 ± 0.4 | 29.9 ± 0.5 |
| Linear | 76.9 ± 0.3 | 54.5 ± 0.5 | 33.5 ± 0.3 |
| PAM | 86.7 ± 0.4 | 69.2 ± 0.2 | 57.4 ± 0.4 |
| Oracle | 89.4 ± 0.3 | 70.9 ± 0.3 | 58.0 ± 0.3 |
Dynamics alignment
Accurate perception is not enough if the simulator mispredicts the motion produced by the policy's commands. Actuation discrepancies on a humanoid can depend on the configuration, motion, and commands of many joints at once, so the Whole-Body Unsupervised Actuator Network (WB-UAN) extends the joint-local residual of Contact-UAN to a single network conditioned on the whole body. It predicts a correction to the joint-position target:
δqt = μ(ot), st+1 = fsim(st, qtbase + δqt).
The simulator's rigid-body dynamics, contact solver, and integration are kept intact; only the actuator input is corrected. WB-UAN observes a causal history of absolute joint positions, velocities, and uncorrected targets across the entire body. It is fit with PPO on short replays of 10 minutes of hardware dribbling, each initialized with Contact-UAN's contact-consistent reset, so it needs neither torque labels, motion capture, nor gradients through the simulator. The dribbling policy is then finetuned in the frozen, corrected simulator; only the policy runs on the robot.
Replaying recorded hardware commands in simulation: WB-UAN rollouts (teal) track the hardware reference (white) more closely than the simulator without a residual (orange).
| Model | Train | Test |
|---|---|---|
| No residual | 0.0879 | 0.0885 |
| Contact-UAN | 0.0495 | 0.0472 |
| WB-UAN | 0.0379 | 0.0495 |
| Policy | Success | Dir. error (°) |
|---|---|---|
| DR | 3/5 | −18.1 ± 4.5 |
| Linear | 4/5 | −16.3 ± 12.0 |
| PAM | 4/5 | −15.6 ± 13.7 |
| PAM + WB-UAN | 5/5 | −14.5 ± 3.4 |
| Policy | Success | Avg. (m/s) | Peak (m/s) |
|---|---|---|---|
| PAM | 5/11 | 1.35 ± 0.20 | 1.56 |
| PAM + WB-UAN | 5/5 | 1.71 ± 0.16 | 2.31 |
With corrected actuator dynamics, the dribbling policy is the most stable and precise on sharp turns and pushes both sustained and peak ball speed well above the PAM-only policy, while remaining easier for a blind operator to control.
Skill chaining
A passing or shooting policy trained from a stationary reset may succeed in isolation yet fail when invoked during dribbling, with the robot mid-stride and the ball already moving. We roll out the dribbling policy in simulation, collect a bank of robot and ball states (including velocities, so gait phase and ball motion are preserved), and finetune the passing and shooting policies from states sampled uniformly from this bank. The dribbling policy stays fixed, and at deployment control simply switches to the striking policy from whatever state the robot and ball are in.
In simulation, finetuning over the dribbling state distribution reduces missed strikes, falls, and strike speed and heading errors. On hardware, chaining alone reaches 4/5 on dribble-to-pass and 3/5 on the harder dribble-to-shoot; combined with PAM and WB-UAN, both reach 5/5.
Switching from dribbling to shooting in simulation: a striking policy trained from random initial states (left) versus one finetuned with distributional skill chaining (right).
| Policy | Dribble → pass | Dribble → shoot |
|---|---|---|
| Chaining only | 4/5 | 3/5 |
| Chaining + PAM + WB-UAN | 5/5 | 5/5 |
More on hardware
Questions & answers
Joga stands for Joint Optimization of Gaze and Action: each policy controls the neck and the body together. It also nods to “joga bonito,” Portuguese for “play beautifully,” a soccer philosophy of creativity, joy, and flair.
The robot must keep the ball in view from long-range approaches down to close-range foot contact, including while accelerating and turning. A torso-mounted 2-DoF neck with direct-drive servos changes gaze quickly and reduces jitter, and avoiding linkages and gears removes backlash and compliance that would otherwise need to be randomized in simulation.
The real pipeline goes image → detector box → pinhole back-projection. Modeling residuals on the box (center for bearing, log-size for range) and back-projecting reproduces that interface exactly, so a residual in box size maps directly to a relative depth error, just as it does on the robot.
A linear baseline conditions only on range with affine per-axis moments and constant dropout. It captures the mean depth bias but not the motion- and pose-dependent structure, so it overestimates depth noise at close range and stays flat against ego-motion. PAM conditions jointly on ball geometry and robot motion and learns a nonlinear distribution over box errors and visibility.
Contact-UAN learns joint-local corrections from each actuator's own tracking-error history. WB-UAN keeps Contact-UAN's contact-consistent replay and unsupervised fitting, but a single network sees absolute joint positions and commands across the whole body, so each joint's correction can depend on other joints. This better fits recorded hardware trajectories and yields faster, more controllable dribbling.
Only the skill policies, the onboard ball detector, and proprioception. PAM and WB-UAN are simulator-side models used during training, and motion capture is used only to collect PAM training labels.
