Evaluating world models by the responses they predict
Shared Physical Representations
Across Sensors and Action Orders
1New York University · 2Carnegie Mellon University · 3Columbia University
What alignment leaves open
Multimodal world models are often judged by whether different sensors produce similar representations. We judge them by the physical responses they predict. Asked this way, evaluation splits into three questions, and answering one does not answer the others.
Do two sensors place the same entity at nearby points in representation space?
proximity, not predictionDo audio and acceleration, fitted apart, predict the same surface's responses?
4.5× gap · 19/19 surfacesDo learned dynamics hold when familiar actions arrive in an unseen order?
shared steps win · 3/3 seedsDo two partial observations predict better together than either one alone?
NMSE 0.41 → 0.32Overview
A sensor-specific compiler maps each observation to a coordinate in a response chart. A shared response model, the executor, turns that coordinate and a query into a predicted response. Sensors are compared through these predictions rather than their coordinates: same-entity predictions should lie closer together than wrong-entity pairs, and closer than each lies to the population prediction.
Across sensors · Cluster Haptic
A three-axis machine scans 118 surfaces while varying direction, speed and normal force. Separate compilers map audio and acceleration into one response chart, with no surface identifier and no cross-modal matching loss, and each comparison pairs the two sensors on disjoint panels of scan conditions. The chart and response model were frozen before any compiler was fitted, and they predict 67 further scan conditions that neither panel contains. Pick any of the 19 test surfaces: its two predictions agree more closely than they do with a wrong surface. Toggle one in and they part.
Same-surface and mean wrong-surface distance for each of the 19 test surfaces, ranked by same-surface distance on a log scale. Same is smaller for all 19, with a median ratio of 5.2. Dpop is normalized differently from NMMSE.
NMMSE against measured repeat-1 responses on the 67 bridge queries, with the population prediction at 1. All four branches beat it. Acceleration, which shares its sensing modality with the target, is more accurate under both panel assignments.
Across action orders · live simulation in your browser
A unit mass on a spring, a dashpot and a plastic slider. The slider remembers: once a pulse yields it, the rest position moves, so A then B does not settle where B then A does. No fitting or development program places A next to B; both orders are kept for testing. Run them and watch. This is the paper's simulator, integrated live at Δt = 0.005.
Across action orders · two executors, one chart
Both executors read the same rank-three response coordinate and learn from the same 17 fitting programs, with 17,923 and 17,844 parameters. The shared-step model applies one residual update per time step, so pulse B acts on the state that A left behind. The whole-program model reads the ordered action word with a GRU and outputs the post-action trajectory in one pass. Judged by fitting error, the whole-program model wins. On both unseen transitions the ranking reverses, in all three initializations.
Three initializations per executor at 30,000 updates, checkpoints chosen on development error. Small marks are single runs and bars their means. NMSE divides by one query-only predictor per split, so raw MSE keeps the same order. The shared-step model also trains on per-step transitions, so this comparison covers its recursive structure and its local supervision together.
Test oracle NMSE of the shared-step executor trained for the fusion analysis, at five update budgets and relative to its own zero-coordinate prediction. The initial 1,200-update model misses the predefined 0.80 threshold; the 30,000-update model, frozen for fusion, reaches 0.18.
Complementary evidence
Free decay starts from a small elastic release, so it never reaches the yield force. Settled hysteresis has zero velocity, so damping drops out of it. Two GRU compilers turn each record into a Gaussian factor over the response coordinates, and a product of experts adds both to a standard-normal prior. Samples from the fused belief then run through a frozen shared-step executor under both orders.
| kstiffness | cdamping | Fyyield force | |
|---|---|---|---|
| Free decay | blind | ||
| Hysteresis | blind | ||
| Fused |
A blind cell varies by exactly zero over that parameter. All three parameters still move the test response: adjacent grid levels of k, c and Fy change it by RMS 0.095, 0.039 and 0.090.
The paper
Shared Physical Representations Across Sensors and Action Orders.
Abstract
Multimodal world models are often evaluated by whether different sensors produce similar representations. However, similar representations do not necessarily imply that the models make the same physical predictions, or that those representations can be reused when actions are combined in a new order. We study both questions through the physical responses predicted by a model. We first use the Cluster Haptic dataset to ask whether audio and acceleration can independently recover the behavior of the same surface from different observations. Predictions from the two sensors are substantially closer for the same surface than for different surfaces, with a 4.5× gap on average, while both also outperform an average-surface prediction. We then show that this agreement alone does not determine how familiar actions should compose. In a controlled elastoplastic system, shared step dynamics fit observed programs less accurately than a whole-program predictor but generalize better to unseen action orders, with the ranking reversing on both held-out transitions across three independent initializations. Fusing free-decay and hysteresis observations further improves prediction, with diagonal Gaussian beliefs yielding the lowest errors. Together, these results distinguish cross-sensor consistency, multimodal fusion, and generalization to new action orders as separate questions in evaluating multimodal physical representations.
@article{tan2026beyond,
title = {Beyond Multimodal Alignment: Shared Physical Representations
Across Sensors and Action Orders},
author = {Tan, Kaizhen and Xu, Xin and Tao, Siru and Li, Yixiao
and Hong, Hanzhe and Feng, Yang and Du, Heqing},
journal = {arXiv preprint},
year = {2026}
}