The robot forgot which drawer it used
Figure: Hu et al., Tokyo University of Science · Research
Aug 19, 2026, 9:05 AM ETResearch
Zhiqiang Hu, Shouren Huang and Masatoshi Ishikawa at Tokyo University of Science have built GaussMemory, which swaps the 2D token buffers that vision-language-action models normally use for a persistent 3D gaussian scene memory — and trains the policy to work out which gaussians are worth tracking carefully.
The problem: Put an apple in one of two identical drawers. Close it. Go and do something else. Come back and retrieve the apple.
Nothing in the current camera frame contains the answer, so a policy that acts on what it can see has to guess, and it guesses right about half the time. This is the ordinary structure of long-horizon manipulation, not an adversarial edge case: most useful tasks involve putting something somewhere and coming back for it later.
How it works: A feed-forward splatting encoder — MVSplat — turns each observation into 8,192 live gaussians. Gaussian Grouping with DINO features sorts them into per-object sets, so the memory has handles on things rather than a soup of primitives, and the persistent store holds up to 32,000.
The part that does the work is a single cross-attention module the authors call Unified Memory Attention, which reads the memory to predict the next action and writes updates back through the same computation. Because both directions run through one differentiable path, what the task needs shapes what gets stored: if the next step requires the plate, attention on the plate rises, and that signal flows back into tracking the plate more precisely.
By the numbers:
- The gap over an identical policy with no memory widens as the horizon does: +4.0 points on LIBERO Spatial, +8.6 on Goal, +11.7 on Long-10.
- Replacing the 3D gaussians with 2D tokens and changing nothing else drops Long-10 from 94.1% to 90.2%.
- LIBERO Goal 96.8% against MemoryVLA's 96.4%; Long-10 94.1% against 93.4%; ahead on 9 of the 10 individual Long-10 tasks.
- On VLABench, +5.2% on Track 1 and +6.0% on Track 6 over π₀-FAST.
- Compared against ten published VLA baselines, including OpenVLA, π₀, CogACT, SpatialVLA and 4D-VLA. Accepted to IROS 2026.
Yes, but: The headline comparisons are fractions of a point on a benchmark that is close to saturated — 96.8 against 96.4 is not a result anyone should lean on, and MemoryVLA still wins Spatial and Object at 98.4%, where recognising what you are looking at matters more than remembering where it went.
Everything here is simulation. LIBERO and VLABench both run in simulators, and the gap between a gaussian memory built from clean synthetic observations and one built from a real camera in a real kitchen is exactly the gap that has embarrassed this class of method before.
The big picture: The interesting claim is not the leaderboard position, it is that memory should be selective and that the task should be what makes it selective. Existing 3D memory systems record everything with equal care — the grasp target and the wall behind it get the same treatment — because the update rules were hand-written and had no way to know which was which.
Splatting is a convenient substrate for this mostly because it is explicit. You can point at a subset of primitives and say: track these, discard those. That is much harder to do inside a neural field, and it is the sort of property that decides which representation a field standardises on.
