What are we discussing when we discuss dexterous-hand simulation?#

Open the original research page (Chinese): What are we discussing when we discuss dexterous manipulation simulation?

The original layout, six-layer framework, 38-paper explorer, 19-repository comparison and full report are preserved. Its October supplement adds 14 relevant works and one background reference. The September snapshot is byte-identical to the author’s local file. Original Obsidian links still open local notes; paper and repository sources remain public. This English page is a shorter reading companion.

2026-11-03 focused addition: sharing policies and dynamics across demonstrations. FlashDexRetarget, ReDexT, Dex-SuperMimic and the Weave integration extend the existing DONIC and SuperMimic coverage. These outside works do not change the original 38-paper or 19-repository counts and are not DexLab replications.

October 2026 additions#

Start with DiffContact, RoboJEPA and Rollback, then use DiffGIPC for coupled derivatives, HandsOn for failure stages and SuperMimic for learned-model gradients. The entries below inherit the original six layers; they are research leads, not DexLab replications. They do not change the September screening counts. Relevance scores from the monthly review are 4/4 for these 14 works and 3/4 for the PDE reference, not quality or performance rankings. The first three have full-reading notes; the other three priorities received targeted text/figure checks, and eight supporting works received abstract/available-text screening.

Original layer	October work and useful question	Boundary to retain
Task and contact	HandsOn, BiDex-100: diagnose contact stages and expand task coverage.	Task counts do not establish fair cross-body comparisons. HandsOn’s agent-protocol exceptions must remain explicit; its real-world clips are qualitative.
Body and actuation	RynnWorld-Latent: share a latent action representation across bodies.	Target-body decoding, actuation limits and delays remain separate requirements; this is an action-interface contribution, not actuator calibration.
Physics and contact solving	DiffContact, DiffGIPC: distinguish useful relaxed sensitivities from exact derivatives and preserve coupled contact responses.	Check native-forward cost after updates. DiffGIPC’s directional checks cover declared points/windows; representation changes also affect reported speedups.
Sensing and observation	DeXplicit, Slot-JEPA, SWIM: make prediction interfaces reflect objects and action-relevant state.	Prediction metrics do not measure contact force or replace tactile-history calibration; SWIM needs state supervision.
Data and policy	RoboJEPA, SuperMimic, LeWAM, Dex-One2Many, DONIC: align prediction with planning, test learned gradients and expand demonstrations.	Separate prediction, action response and native task benefit. RoboJEPA changes horizon and resolution together; tracking is not autonomous manipulation; one video still needs reconstruction and assets.
Calibration and real benefit	Rollback: compare alternative actions from the same decision context.	Verify hidden-state restoration and repeat variability. Adaptation-used curling trajectories are not unseen validation; the 48-H200 system is not a minimal setup.

diff-soup remains a separate PDE engineering reference, without qualified contact/friction evidence. See the full October discussion for versioned claims and proposed DexLab comparisons. All selected works were first submitted in October; the review window was 2026-10-02 through 2026-11-02, and this supplement was published on 2026-11-03.

We are asking how a controlled virtual environment can help us study what a hand can do, why it fails, and which findings deserve validation on a real robot. The simulated system includes motion, actuation, contact, sensing, control clocks and task conditions. Engine names, images and throughput each answer only part of that question.

This guide starts with picking up a bottle, rotating it within the hand and putting it down. It develops six layers of questions, five research approaches and three experimental starting points. Dexterous manipulation here includes multi-finger or bimanual grasping, finger transitions, tool use and deformable-object handling. Parallel-gripper research serves as adjacent evidence where its contact or sensing mechanisms are relevant.

Start with a hand rotating a bottle#

The apparently continuous motion contains different experimental requirements:

Touch the bottle: collision geometry should place contact at the correct surfaces.
Hold it: fingers must provide supporting forces and resisting moments that the actuators can actually produce.
Rotate it within the hand: some contacts persist while others slide, roll or transfer between fingers. Object rotation relative to the palm must be distinguished from moving the whole wrist.
Put it down and release: new support should carry the object, and opening the fingers should end their attachment to it.

“It did not fall,” “its orientation changed” and “in-hand manipulation succeeded” therefore need separate acceptance criteria. Initial table support, virtual attachment and independent hand support must also be disclosed. A failed trial identifies an unmet requirement; attributing its cause takes controlled comparisons.

Six layers: decide what simulation should help with#

Layer	Question	Evidence to retain
Task and contact	Is the goal grasping, finger transfer, tightening or a specified process tolerance?	Initial conditions, contact stages, success windows, disturbances and failure types.
Body and actuation	Where can the hand move, and what forces can it produce?	Collision geometry, mass and inertia, joint/actuator coupling, torque limits, gains and delays.
Physical evolution	How do contact, friction and deformation affect the next state?	Contact positions and normals, forces and moments, penetration, slip, timestep and solver budget.
Sensing and observation	What does the policy know, and can hardware supply it?	Camera/tactile generation, occlusion, noise, calibration, sampling rates and timestamps.
Data and policy	Does simulation plan, train, augment demonstrations or reject failed actions?	Valid-trajectory yield, object/contact coverage, training inputs, interaction counts and total cost.
Calibration and validation	Which predictions survive independent checks, and which transfer to hardware?	Numerical references, held-out conditions, real measurements, failures and uncertainty.

These layers organize this guide; they are not a unified algorithm proposed by a particular paper. They connect through a loop: the policy selects an action from observations, the controller and actuators turn it into physical input, and the evolving environment produces the next observation. The task and scorer define success for the entire loop.

For example, a joint-position target becomes torque through a controller responding to tracking error. It differs from commanding torque or directly writing joint positions. Changing gains, limits or update rates can change contact even with the same target sequence. Record physics steps, control updates and sensor samples separately. The MuJoCo actuation model illustrates the distinction between actuator input and output.

Contact detection, contact response and numerical solution#

Make “good contact simulation” into three inspectable questions:

Detection: which surfaces meet, and what are the contact positions, normals and distances? Inspect visual meshes and collision shapes separately.
Response: what normal forces, sliding friction, torsional moments or deformation can these contacts support?
Numerical solution and time integration: how are coupled motion and contact responses computed and advanced to the next state? What timestep, stopping tolerances and computational budgets apply?

This is a functional distinction; an implementation may couple these operations. MuJoCo documents collision detection, the contact model and the constraint solver separately. Software comparisons should also distinguish engines, task frameworks, solver algorithms and integrators; see DexLab’s configuration taxonomy.

A more thorough solve first establishes a more thorough solution of the selected model. If that model lacks a moment direction needed by the task, additional iterations alone cannot supply it. Numerical sensitivity to timestep and accuracy for real materials also require separate checks.

Which part is differentiable?#

A gradient answers a local question: how would a small action or parameter change affect the next state or task loss? A differentiability claim should identify the differentiation variables, the modules involved and the treatment of contact transitions.

Before a finger reaches the bottle, a small motion may leave the bottle unaffected. A larger motion enters contact and starts pushing it. A local derivative along the current branch does not automatically describe the effect of crossing that boundary.

An if/else does not make every calculation inside its branches non-differentiable. Conversely, a contact solver that supplies gradients does not establish smooth contact-pair selection, collision-feature changes or arbitrary long trajectories. Smoothing, fixing a contact branch and defining custom derivatives can also produce gradients with different meanings.

Evaluate forward motion, local derivatives and the benefit of updating actions with those derivatives separately. Finite differences provide a local check; perturbation scale matters near nonsmooth boundaries, and one check cannot certify every contact mode. Warp’s differentiation documentation describes automatic-differentiation limitations and Jacobian, finite-difference and gradient-checking tools. Engine- and version-specific coverage still requires its own verification.

Validate touch, images and physical state separately#

A fingertip resultant force, pressure distribution, gel deformation and optical tactile image describe different levels of information. A contact-force array does not automatically constitute a calibrated tactile sensor. Attractive camera images do not establish that a policy used vision in a closed loop.

Choose observations by asking what the task needs to know: holding may require slip detection, insertion may require the direction of misalignment, and finger transitions may require contact history. The same current pose need not imply the same internal tactile state. HydroShear models stick-slip and path-dependent shear. Results reported with grippers and GelSight Minis do not directly establish multi-finger contact-transfer capability.

Taccel, in contrast, includes robots, objects and tactile sensors in rigid/deformable contact simulation. These approaches invest in different parts of the model. Compare task-relevant signals and computational costs, alongside their images.

Disclose whether object ground truth is training-only, whether deployment uses estimates or direct ground truth, and whether the tactile model feeds back into physical motion. Restoring an intermediate state also requires the sensor, controller and physical history states that the implementation depends on.

How researchers use simulation#

Simulation supports repeated attempts, failure replay and controlled interventions; it can also predict a short action horizon online. A comparable benchmark freezes task distributions, interfaces, budgets and scoring rules. The five approaches below can be combined, with their costs counted together.

Approach	Simulation’s main role	What remains to be established
Detailed contact and deformation	Compute task-critical contact, gel or object deformation.	Material/sensor calibration and whether extra computation improves useful outcomes.
Aligned observation representations	Generate force, depth or shear representations that can be aligned with real signals.	The real-sensor mapping and whether discarded information matters.
Data generation and filtering	Transform demonstrations, execute new trajectories and reject failures.	Whether data adds contact strategies and recovery opportunities rather than only frames.
Retargeting and curricula	Turn human motion into executable robot behavior and gradually remove assistance.	Actuation constraints, performance after assistance removal and deployment observations.
Simplified simulation plus real data	Learn reusable coarse skills.	Real touch, fine control, recovery, and the required real data and human time.

This table synthesizes the papers below. It is neither a list of five implemented DexLab capabilities nor a method ranking. Added simulation complexity earns its cost by improving a measurable failure.

Sharing policies and dynamics across demonstrations#

Picking up, rotating and putting down a bottle can reuse control experience on one hand. Different references may share dynamics, while changes in geometry, mass, friction or articulation require suitable conditioning and state representations. This connects the data/policy, physics and validation layers.

A shared policy selects actions from execution state and reference motion. A shared world model learns transitions from state and actual actions; differentiating imagined rollouts can then train the policy. Shared policy training also works with model-free FlashSAC or PPO. Simulator throughput, experience reuse and differentiable optimization are separate possible gains.

Work and checked source	What is shared	Evidence boundary
FlashDexRetarget, v2	Multi-reference FlashSAC; each hand’s actor–critic shares parameters across demonstrations, conditioned on geometry, distances and future references.	A joint-training precedent without a SuperMimic-style differentiable WM. Its CHORD comparison uses PPO, not the current FlashCHORD recipe.
DONIC, v1	A shared bimanual tracker using PPO, BC, contact-role rewards and successful-execution recycling.	Up to 3,420 training references; one frozen checkpoint evaluates unseen trajectories and objects. This differs from cumulative conversion during training.
ReDexT, official project	A feedback residual policy corrects IK commands, with frozen execution or optional shared adaptation.	Each hand has its own policy shared over trajectories. Target-collection adaptation is not zero-shot generalization.
Dex-SuperMimic, fixed snapshot	Learned-model rollouts and policy updates for Wuji cube reorientation and pen spinning.	Three task configurations do not establish one jointly trained checkpoint or CHORD-scale reference coverage.
rethinking-supermimic, fixed snapshot	Weave / Weave XHand references and policy updates through a learned model.	Implemented data/training entry points; the checked README gives no success-rate or speed benchmark for the added hand–object tasks.

The FlashDexRetarget project reports 90% conversion on 50 XHand references in 29 GPU-hours and over 600 of 1,000 converted within 60 GPU-hours. These are author-reported cumulative conversions, not final frozen-checkpoint success. Different hands, resets, thresholds and cost accounting prevent a direct ranking.

A possible SuperCHORD research question#

Multi-reference policies already have precedents. Combining CHORD wrench-space guidance with a shared hand–object WM and reference-conditioned policy is a proposal; this targeted search establishes neither priority nor a DexLab training result.

Compare independent per-reference training, joint FlashSAC and joint training with model-gradient policy updates. Match rewards before separately testing contact guidance. One hand with compatible object representations and uniform reference sampling is a reasonable starting point, not a frozen or executed protocol.

Record total GPU-hours, simulator transitions, cumulative conversions, final-checkpoint training coverage, unseen-trajectory performance and unseen-object performance separately. Check forward prediction, action response near contact and native-simulator benefit after updates; low prediction error alone does not establish useful gradients. Read FlashDexRetarget first, DONIC/ReDexT for generalization, then the two SuperMimic implementations for model-based optimization.

Checked on 2026-11-03 against the linked papers, official project pages and fixed repository snapshots. This was targeted reading, not a new full-paper review or reproduction campaign.

Seven representative works and their questions#

Work and source	Main contribution	Questions to follow while reading
BODex · v3	Bilevel optimization for dexterous grasp synthesis, evaluated in simulation and real grasping.	Which loads and objects define stability? Does evaluation extend to finger transfers and continuous manipulation?
Taccel · v2	High-performance rigid/deformable contact and visual tactile simulation.	Which sensor counts, meshes, parallelism and hardware underlie throughput?
HydroShear · v2	History-dependent shear and stick-slip tactile observations.	What real data calibrates it? Is the task a gripper or multi-finger task, and how is its initial state prepared?
DexMimicGen · v2	Bimanual dexterous trajectory generation from a small set of human demonstrations.	How much new object, contact-mode and recovery coverage results? What real-data cost remains?
DexMachina · v1	Virtual object controllers of decreasing strength teach policies to reproduce demonstrated object effects.	When is assistance removed? What actuation and wrist conditions govern hand comparisons?
DexScrew · v2	Finger gaits learned in simplified simulation support real demonstration collection and tactile-policy learning.	How are progress and complete success distinguished? Who performs real collection and initial assembly?
Dex-X · v3	Simulated interaction supplies tactile supervision missing from human videos for visual tactile policies.	Which tasks test the simulated teacher and real deployment? What observations, motion references and generalization conditions apply?

These works address grasp generation, contact and touch, demonstration augmentation or policy transfer. They are not seven directly rankable physics engines. The summaries use the linked versions’ official abstracts and existing research notes. The questions identify conditions to inspect when reading or reproducing, without claiming that the papers omit them.

Keep units and denominators attached to numbers: task progress is not complete-task success; frames are not independent demonstrations; aggregate parallel steps are not a single environment’s real-time rate; a real success rate for one task does not cover every task. Paper experiments, released code and actual reproduction supply different evidence.

Test these distinctions against three DexLab cases#

Case	Existing evidence	Interpretation boundary
Wuji two-finger sphere pinch	Under the same controller, changing point contact to synthetic torsional contact reduced maximum displacement in a two-second hold from 2.734 mm to 1.025 mm, passing the original 2 mm threshold.	The contact model changed acceptance for this configuration. Parameters are not measured fingertip materials; slow drift remains, and neither solver failure nor in-hand rotation success was established.
Sponge wiping	In the fixed configuration, assisted holding passed 3/3 trials; removing assistance passed 0/3.	This supports assistance dependence under those conditions, without establishing general failure of the underlying solver.
Cloth manipulation	Single-hand pinch, lift and release passed. The complete folding record reached 12.55% maximum edge strain against a predeclared 5% limit.	Executable local actions and complete-task acceptance are separate. Other folding requirements also remain unmet.

Report separately whether records are trustworthy, computation agrees with the selected model, physical references hold, the task succeeds and hardware validates the result. See the failure audit for detailed attribution and the model-evaluation protocol for policy/backend acceptance.

Where to start experimentally#

First check holding and release. Choose one hand, a simple object and a fixed wrist; specify load and holding window. Qualify the positive control, then add interpretable controls such as open fingers, disabled contact or low friction. Record native contact, object motion and actuator saturation. Diagnose failed basic support before interpreting a finger-transition failure. The existing Wuji case provides an offline-rescorable starting point.

Then ask which tactile information helps. Under matched task conditions, compare no touch, fingertip forces, geometric/shear representations and more detailed tactile models. Check signals and event timing before comparing policy benefits at matched budgets; retain rendering, learning and real-calibration costs. This is a proposed study, not a completed tactile comparison on this site.

Finally ask which real cost simulation reduces. When comparing demonstration augmentation, functional retargeting and real-data completion, fix test objects and task requirements. Record real demonstration counts, collection time, simulated interactions and held-out performance separately. Without hardware tests, numerical and task checks can proceed while transfer benefits remain unvalidated.

For any route, write an experiment card before running: goal and initial state, models and versions, action/observation interfaces, clocks, success/failure criteria, controls, budget and raw-record location. Changes to tasks, parameters or acceptance need a new condition record. Continue with the Newton topic guide, experiment directory and diagnostic method.

Sources and maintenance scope#

This public guide adapts the research note “What are we discussing when we discuss dexterous manipulation simulation?”, created on 2026-09-10 and supplemented on 2026-09-14. It preserves the six-layer framework, research approaches and evidence boundaries. On 2026-11-02, the cited official abstracts, documentation and published DexLab cases were checked; HydroShear and Dex-X links were updated to v2 and v3 respectively. This did not repeat the original note’s complete literature screening or carry old-version details into newer versions without verification.

On 2026-11-03, the author’s original self-contained HTML was published at their request, retaining its September sources and verification scope, with October work appended separately. This page remains the shorter companion. Outside papers are not DexLab replications; proposed studies still require separately frozen protocols. The original snapshot SHA-256 is 2c41d2ee32f8094298d2398e3950d5b749a3030a62a07e25128df6d709823c00.