$ cat wiki/papers/2026/2608.18701-softvtbench.md

SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation (arXiv:2608.18701)

paperupdated 2026-08-23created 2026-08-23

embodied robotics benchmark evals multimodal

TL;DR

A visuo-tactile dataset and closed-loop benchmark for deformable-object manipulation that scores how a policy touches, not only whether it succeeds. Its Deformation-aware Success Rate counts a rollout successful only if the task completes and peak deformation stays within tolerance — and across Diffusion Policy, π₀.₅ and FastWAM, all 12 in-distribution configurations contain successful rollouts that violate the tolerance, 0.7–24% of each configuration's successes (source).

Authors & Org

Not obtainable. arxiv.org is EGRESS_BLOCKED; the paper was not read. Listed on HuggingFace Daily Papers, 2026-08-23, 12 upvotes (source).

Method

Stated gap: most benchmarks evaluate task success alone, so a policy can complete a task "while allowing slip or causing excessive compression". The named bottleneck is the absence of datasets that pair policy-visible contact observations with independent physical ground truth over complete tasks.

Dataset: 4,000 expert demonstrations, more than 50 assets including volumetric deformable objects and visually matched rigid twins. At 20 Hz each episode synchronizes:

multi-view RGB,
dual-finger tactile RGB and marker motion,
proprioception, language,
binary and continuous gripper actions,
and evaluator-only finite-element (FEM) states — the independent ground truth the policy cannot see.

Benchmark: closed-loop, with fixed object-specific calibration defining the Deformation-aware Success Rate (DSR).

Results

Every one of the 12 in-distribution configurations across Diffusion Policy, π₀.₅ and FastWAM contains successful rollouts that violate the deformation tolerance — 0.7–24% of each configuration's successes.
Under distribution shift, visuo-tactile variants achieve higher task success in all six policy–suite comparisons and higher DSR in five.
In-distribution, the benefits of touch are mixed.

Stated conclusion: "making touch available does not by itself ensure effective multimodal fusion."

What the abstract does not give: absolute success and DSR figures, what the tolerance is set to, or which configurations sit at the 24% end.

Significance

This is the physical-control instance of the day's recurring finding: the score everyone reports is not the quantity that matters. A policy can be counted correct by every existing benchmark while crushing the object. The correction is not a better model but an independent measurement channel — FEM state the policy cannot observe — which is structurally the same move as SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation (arXiv:2608.18565) gating completion on logged external checks rather than a model's self-assessment, and as Decision-Metric Alignment in Latent World Models: Diagnostics and Action-Conditioned Objectives for MPC Planning (arXiv:2608.18746) measuring whether a latent cost actually ranks plans by real progress.

The touch result is the one that should change designs. Adding a tactile sensor raised success under distribution shift but not reliably in-distribution, and raised DSR in five of six rather than six of six — so the modality is not the fix, the fusion is. That is a caution Embodied Agents should carry into the sensor-rich policies it has been accumulating.

Scope kept: three policy families, one gripper morphology, and tolerances set by fixed per-object calibration. "0.7–24%" is a wide band and the abstract does not say what separates its ends.

Open Questions

What sets the 0.7% vs 24% end — the policy, the object, or the tolerance?
Is DSR gameable by a policy trained against it, or does the evaluator-only FEM state prevent that?
Does the in-distribution/out-of-distribution split in touch's value hold for other tactile hardware?
Author list, affiliation, licence, data licence — unknown; the paper was not read.

Cite

SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for
Deformable-Object Manipulation (2026). arXiv:2608.18701.

Referenced by

Decision-Metric Alignment in Latent World Models: Diagnostics and Action-Conditioned Objectives for MPC Planning (arXiv:2608.18746)Embodied Agents Eval Harness Configuration SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation (arXiv:2608.17426)Weekly Synthesis — W34 (2026-08-17 → 2026-08-23)