CoRL RoboMem Workshop 2026

GEMA

Generative Memory
for Robotic Manipulation

Zhiyan Li1Pembe Gizem Ozdil1Ben Zandonati2Ruojin Cai1Hang Le1Tomás Lozano-Pérez2Leslie Pack Kaelbling2Yilun Du1,†Weirui Ye2,†
1 Harvard University2 Massachusetts Institute of Technology† Equal advising

GEMA interprets historical evidence as task-specific textual and visual references for robot control under partial observability.

Paper Explore the method ↓
FROM EXPERIENCE TO REFERENCE
Recorded robot observation of an object on a tabletop01 / OBSERVED EXPERIENCE
Generated visual reference showing an offset placement target02 / GENERATED REFERENCE
THE NEXT REQUEST

“Place it 10 cm to the right
of its original position.”

Historical evidenceTask-specific guidance
01 / MOTIVATION · HUMAN COGNITION

Inspired by how humans
remember and imagine.

Our starting point is a cognitive perspective on human memory: remembering is a constructive process.

Constructive accounts of episodic memory suggest that people can recombine elements of past experience to imagine possible future events. This motivates GEMA’s central idea: use retained experience to construct task-specific guidance for future action.

For a robot acting under partial observability, retrieving the past alone may not specify what to do next. Placing a cube 10 cm to the right of its original location, for example, requires a target that was never observed.

Inspired by this account of human cognition, GEMA separates historical evidence from its interpretation for control. A pretrained multimodal language model turns that evidence into textual and visual references; a downstream controller executes them, while a task agent assesses observed outcomes.

02 / THE FRAMEWORK

A memory that works
with the controller.

Retained experience and the current request meet in generative memory. A task agent coordinates reference generation, execution, and progress assessment.

New observations → memory
01 / RETAINED EXPERIENCE
Initial red cube position in the artist workshop sceneRobot approaching the red cubeGripper aligned over the red cubeGripper closing around the red cube
Observe → approach → graspSampled observations from the same episode
CURRENT REQUEST

Place the cube 10 cm to the robot’s right of its original position.

Robot-base frame · world −Y
02 / GENERATIVE MEMORYInterpret the past
03 / TASK REFERENCE
Generated target image: the red cube placed 10 cm to the robot’s right of its original position in the same workshop scene
A new target from past experience
05 / EXECUTION2×
Cube Transform · 10 cm robot-right
04 / DOWNSTREAM CONTROLExecute with live observations
AGENTIC

GEMA-Astra

Text and image references guide an agentic controller.

WORLD ACTION MODEL

GEMA-DreamZero

FL2V visual-goal conditioning, with text through cross-attention.

VLA POLICY

GEMA-Pi05

Memory-derived text enters the policy’s instruction prompt.

03 / EVALUATION DESIGN

Two ways to test
memory in action.

Controlled episodes isolate specific memory demands. Continuous query streams test whether that memory remains useful as the scene evolves.

A

Task-Based evaluation

9 tasks · 100 episodes per task

Transform and Resume test remembered positions and drawer associations. Seven MIKASA variants test novelty comparison, sequence reproduction, and cue retention.

B

Continuous-Query (Con-Q)

Persistent scenes · No resets between queries

Single-view LIBERO sequential memory scene01 / LIBERO

Sequential memory

Remember event order and resume interrupted tasks.

Query types
  • Store / retrieve / transferMove objects between the table and drawers, or between drawers.
  • Move to platePlace an object on the fixed plate.
  • Return to a prior locationReturn the plate object to its pre-placement table spot or source drawer.
  • Interrupt / resumePause a drawer task, finish a plate-and-return subtask, then resume.
Example query“Return the butter from the plate to the drawer it came from.”
LIBERO spatial memory scene with drawers and objects02 / LIBERO

Spatial memory

Recover historical locations and object–drawer associations.

Query types
  • Store / retrieve / relocateMove objects among six drawers, table spots, and the plate; transfers may cross cabinets.
  • Return to a prior locationRestore an object to the drawer or table position it occupied before a plate task.
  • Locate a historically vacated drawerUse the drawer an object left in a specified earlier query.
  • Nested interruption / resumptionComplete inner subtasks before continuing suspended parent tasks.
Example query“Move the blue bowl to the drawer the red bowl just vacated.”
RoboCasa semantic memory scene with robot and order trays03 / ROBOCASA

Semantic memory

Retain, update, and restore historical order contents.

Query types
  • Add / removeAdd one item to an order, or remove it to the returns area.
  • Transfer / cancelTransfer an item between orders, or clear one order to returns.
  • RestockMove a returned item back to its own stock position.
  • Restore a past orderRecover an order’s contents after a specified earlier query, preserving the other order.
Example query“Restore order A to its contents after query 2, keeping order B unchanged.”
04 / RESULTS

Evaluate the action.
Measure the memory.

Task success and query survival capture complementary aspects of memory-dependent control.

Mean task success

Unweighted mean across nine tasks (%)

100 evaluation episodes per task. DreamZero is the controller-only baseline.

Per-task results and reporting details
Task success (%). ± indicates standard deviation over five seeds where reported.
TaskMemoryVLALong-DPGMPDreamZeroGEMA-DreamZero