World models predict the future frames of physical interaction.
They can generate training data, initialize policy models, and serve as simulators for embodied AI.
Dexterous Tactile World Model
Predict future egocentric human manipulation videos with whole-hand touch
observed frame 0
left gloveright glove
pressure finger flexion
observed: RGB, hand pose, touch
predicted: 36 frames at once
0
12
28
44
48
chunk 1
chunk 2
3
Predict the future with touch
Force is invisible.
How much force the hand applies, and how it changes over time, decides how an interaction unfolds.
Dexterous touch measures them directly.
Whole-hand tactile gloves now record where and how firmly both hands press.
Can world models use dexterous touch to better predict the future?
Dexterous Tactile World Model
We embed each hand's tactile reading and add it to the video tokens at that hand's location, through a zero-initialized residual in a pretrained video diffusion transformer.
Observations t ≤ K
Task instruction
Video diffusion transformer, 30 blocks
tactile residual added in 5 of the 30 blocks
Embedeth = Whsth + frame(t) + hand(h)
Place
footprint g at the hand
footprint g at the handAdd as a residualu ← u + proj(LN(e)) g
Predicted frames t > K
TrainK ∈ {4, …, 44}, flow-matching loss to ground truth
InferK = 12, predict 36 frames at once
Better visual quality, hand location and motion
TouchWorldFeelWorldVT-WMDTWM (ours)
First predicted chunk, frames 13 to 28. Dashed boxes: ground-truth hand regions.
Better generalization to unseen objects & tasks
Why touch helps?
Touch history correlates with future manipulation. The observed frames look alike, but the observed touch evolves differently, and so do the manipulations that follow.
force holds steady
observed frame 12
grasp holdsfuture frame 44force decays
observed frame 12
hands releasefuture frame 44left handright hand
Changing touch → the interaction is about to change
A decaying touch leads to a release 2.5× as often (16%), a rising touch 1.8× as often (12%).
A rising force usually marks a short push, press or pinch in progress, which ends within a second and the hand lets go.
Training with touch helps, even without it at inference
Training with touch teaches the model to tell visually similar frames with different contact states. Even without touch at inference, it applies this learned representation to estimate current contact, which also reduces its uncertainty. An uncertain model averages over possible futures and produces blur.
Vision-only baselineTouch in training onlyTouch in training and inference (DTWM)
The benefit increases over the prediction horizon
Vision-only baselineDTWM (ours)
BibTeX
@article{zeng2026dtwm,
title = {Dexterous Tactile World Model},
author = {Zeng, Ziyao and Sun, Xiatao and Wang, Hao and Pan, Yueyang and Yu, Zhengxiang
and Yang, Fengyu and Liu, Tianyu and Fan, Zhiwen and Rakita, Daniel},
journal = {arXiv preprint arXiv:2609.34286},
year = {2026}
}