LIDEA: Human-to-Robot Imitation Learning via Implicit Feature Distillation and Explicit Geometry Alignment

Yifu Xu1*, Bokai Lin2*, Xinyu Zhan1, Hongjie Fang1, Yong-Lu Li1,2, Cewu Lu1,2, Lixin Yang1,2†
1Shanghai Jiao Tong University 2Shanghai Innovation Institute

†Co-advising

Paper Code Video
Teaser Figure

Overview of LIDEA. LIDEA bridges the embodiment gap between human hands and robot arms from two complementary aspects: (Top) implicit 2D feature distillation utilizes a transitive feature bridge to align human and robot representations; (Bottom) explicit 3D geometry alignment filters embodiment-specific geometries and fills a virtual gripper to construct a geometry-aligned point cloud.

Abstract

Scaling up robot learning is hindered by the scarcity of robotic demonstrations, whereas human videos offer a vast, untapped source of interaction data. However, bridging the embodiment gap between human hands and robot arms remains a critical challenge. Existing cross-embodiment transfer strategies typically rely on visual editing, but they often introduce visual artifacts due to intrinsic discrepancies in visual appearance and 3D geometry.

To address these limitations, we introduce LIDEA (Implicit Distillation and Explicit Geometric Alignment), a novel framework that enables direct policy learning from human videos. In the 2D visual domain, LIDEA employs a dual-stage transitive distillation pipeline that aligns human and robot representations in a shared latent space. In the 3D geometric domain, we propose an embodiment-agnostic alignment strategy that explicitly decouples embodiment from interaction geometry, ensuring consistent 3D-aware perception.

Extensive experiments empirically validate LIDEA through two complementary protocols evaluating data efficiency and cross-embodiment robustness. Results show that human data substitutes up to 80% of costly robot demonstrations, and the framework successfully transfers unseen patterns from human videos for out-of-distribution generalization. Our code and dataset will be made public.

Method

Pipeline Architecture

The LIDEA Framework. (Left) Stage ① establishes semantic equivalence by distilling features from human observations to pseudo-robot counterparts. Stage ② then trains the real-robot encoder to match the pseudo-robot representations, achieving a shared latent space where EH ≈ EP ≈ ER. (Right) To construct a canonical 3D observation space, embodiment-specific geometries are filtered from the unprojected point clouds. A virtual gripper is then filled into the scene, yielding hybrid 3D observations. (Center) The 3D visuomotor policy fuses these aligned dense 2D features and sparse 3D tokens to predict continuous actions.