Yifei Yao

Learning Embodied Agents from Experience

Research project 2026 · In progress

How can an open model learn to act from a frontier agent’s experience? I am studying the transfer of Astra’s robot interaction experience to Qwen3.8-27B through staged SFT and reinforcement learning, connecting visual grounding, short manipulation skills, and long-horizon tasks.

Building on EBench and SimplEvo, this project studies how experience collected through interaction can become a student model’s own ability to observe, act, and correct mistakes.

Multi-stage SFT and embodied VQA RL experiments are underway. Short-skill rollout and scoring have passed initial smoke tests; their training benefit and transfer to long-horizon tasks remain under investigation.

From seeing to acting

A look at the data and task design, using recorded simulator frames and reference labels.

01 / Embodied VQA

Where is the target, relative to the hand?

One scene supports questions about image location, relative height, and where to move. The model receives RGB views and measured robot state; the simulator supplies reference labels for training.

Overhead view of two robot arms, an apple beside the right hand, and a fruit bowl on the left. — Scene view · pick up the apple and place it in the bowl

Right wrist camera: the apple appears at the top centre, above the open gripper fingers. — Right wrist view · 3 × 3 image regions

In which region is the centre of the apple?

Reveal simulator reference

Top. For this image-region question, matching the reference earns a reward of 1; a different region earns 0. These are reference labels, not predictions from the student.

A second question: how high is the hand?

How high are the right fingertips above the top of the apple?

Reference: 3–10 cm above. The simulator measures approximately 5.2 cm. This question uses robot-relative geometry, rather than the apple’s position in the image.

02 / VQA reinforcement learning

Turn spatial questions into training feedback

Sample answersThe student proposes several answers for the same observation.
Check against the sceneSimulator labels score spatial accuracy, beyond the answer’s format.
Update and re-evaluateRL uses the scored answers; agent replay helps retain tool-use behavior.

VQA reward and closed-loop task success are evaluated separately. Better answers alone do not establish better robot execution.

03 / Short skills toward longer tasks

Break “put the apple in the bowl” into testable skills

Local objectives make it possible to test alignment, grasping, and placement separately before studying their composition over a full task.

Demonstration replay before grasping: the right hand approaches the apple on the table. — **Align.** Approach the target and hold the hand in position. The skill checks positioning while discouraging movement of the object.

Demonstration replay with the apple held by the right gripper after grasping. — **Grasp & lift.** Lift the object and keep it held. The skill checks sustained holding, rather than a gripper-close command alone.

Demonstration replay at placement: the right hand opens over the fruit bowl. — **Place.** Release the object at the destination and let it settle. The skill checks the final placement, release, and rest.

Illustrative frames from one recorded demonstration replay. Each skill is a separate training objective; these frames do not show an RL-trained student completing the task. Long-horizon RL is ongoing work.

← All publications