Dataset & setup
HRDexDB (SNU VCLab) provides ~2.1K real dexterous grasping sequences across 100+ scanned objects and multiple hand embodiments, with synchronized robot joint signals and 6D object tracking from a 23-camera rig. We use the two robot embodiments:
xArm6 + Wonik Allegro V5. Hand commands and measured joint states recorded at ~97 Hz in radians; arm joints at ~130 Hz.
xArm6 + Inspire F1 with 12 finger joints, 6 actuated + 6 linkage-coupled (URDF mimic joints). Raw counts converted to radians with the vendor mapping.
Per-video-frame 6D poses (~33 Hz) of the scanned object mesh, transformed into the robot base frame via the released camera-to-robot calibration.
We evaluate 120 episodes covering 35 objects (up to two scenes per object and hand). Every stream carries wall-clock timestamps on a shared epoch clock, so arm, hand, and object signals are time-aligned exactly, then resampled to a uniform 100 Hz control clock.
Method: dynamic replay, no cheating
Each episode is rebuilt as a Newton scene: the fixed-base xArm6+hand articulation imported from the
HRDexDB URDFs, the scanned object as a free rigid body at the first ground-truth pose, and a
support plane at the object's initial resting height. The raw scans are frequently open shells
(non-watertight), so the object collides as a watertight CoACD convex decomposition
(ModelBuilder.approximate_meshes, ≤16 hulls, original scan kept as a visual-only shape) with
mass, center of mass, and inertia integrated analytically over the watertight parts. The simulation runs
SolverMuJoCo (implicitfast, Newton solver, elliptic cones) at 400 Hz, and a 100 Hz
control loop writes the recorded joint trajectory into Control.joint_target_q — nothing else
touches the simulation.
Which signal should drive the robot?
The dataset stores both commanded and measured joint signals for the hands (the arm's commands are Cartesian end-effector poses, so the arm always follows measured joint positions). A pilot on five objects per hand compared the two as PD target sources:
| Hand | Target source | Episodes | ADD RMSE median [cm] | lift match |
|---|---|---|---|---|
| Allegro V5 | recorded commands | 12 | 5.8 | 67% |
| Allegro V5 | measured joints | 12 | 5.6 | 42% |
| Inspire F1 | recorded commands | 7 | 3.0 | 50% |
| Inspire F1 | measured joints | 7 | 3.1 | 62% |
Both sources track similarly in ADD; recorded commands match the lift outcome markedly better on the Allegro (67% vs 42%) and slightly worse on the Inspire. We use commands as the headline — they are what the real controller received, including the intentional squeeze beyond contact that measured joint states hide.
CMA-ES calibration
Default gains and material parameters are not documented for this hardware, so we calibrate seven scalar parameters per hand with CMA-ES (log-space, population 8), minimizing the mean object ADD RMSE over five training episodes (banana, apple, baseball, book, beige brush — one scene each). Candidates are evaluated population-parallel in a multi-world Newton model with per-world gains, friction, mass, and armature. Allegro V5: 22 generations, 176 rollout sets, training ADD RMSE 5.4 → 5.3 cm · Inspire F1: 22 generations, 176 rollout sets, training ADD RMSE 2.9 → 2.8 cm.
| Parameter | Search range | Allegro V5 (tuned) | Inspire F1 (tuned) |
|---|---|---|---|
arm_ke | [500, 20000] | 9020 | 590 |
arm_kd | [10, 500] | 297.8 | 228.7 |
hand_ke | [5, 500] | 40.24 | 5.692 |
hand_kd | [0.1, 20] | 8.344 | 0.419 |
friction | [0.2, 2.5] | 2.429 | 1.655 |
object_mass | [0.03, 1] | 0.05124 | 0.2036 |
joint_armature | [0.001, 0.1] | 0.07537 | 0.02514 |
How much of the improvement is real? Winner's curse on a chaotic objective
Grasp outcomes are contact-chaotic: re-running the same episode with the same parameters in a fresh solver instance shifts ADD RMSE by up to ~10% (and can flip a marginal lift). Selecting the best of ~160 noisy evaluations therefore biases the reported optimum low. We revalidate the frozen tuned parameters with fresh rollouts on the training episodes:
| Hand | CMA-ES reported best [cm] | Tuned, revalidated [cm] | Defaults [cm] |
|---|---|---|---|
| Allegro V5 | 5.3 | 6.1 | 5.6 |
| Inspire F1 | 2.8 | 2.5 | 3.1 |
The two hands behave differently. For the Inspire F1, the tuned parameters survive revalidation and transfer to held-out objects — a genuine calibration gain. For the Allegro V5, the revalidated optimum is indistinguishable from the defaults: with 16 independently driven fingers, grasp outcomes are chaotic enough that the CMA-ES "improvement" is mostly selection noise. Within this parameter family, Allegro replay fidelity is limited by contact chaos, geometry approximation, and dataset calibration error rather than by poorly chosen gains.
Results
Aggregate object-tracking fidelity across all evaluated episodes (calibration outliers excluded; holdout = objects never seen by the tuner). ADD is the mean vertex distance between the simulated and ground-truth object pose — it penalizes both translation and rotation without symmetry ambiguity.
| Hand | Params | Episodes | ADD RMSE median [cm] | ADD RMSE mean [cm] | p90 [cm] | holdout median [cm] | pos RMSE median [cm] | rot RMSE median [°] | lift match |
|---|---|---|---|---|---|---|---|---|---|
| Allegro V5 | default | 68 | 8.3 | 9.3 | 14.8 | 8.8 | 7.2 | 41.5 | 81% |
| Allegro V5 | tuned | 68 | 8.3 | 9.8 | 14.9 | 9.2 | 7.5 | 38.0 | 79% |
| Inspire F1 | default | 52 | 4.9 | 5.9 | 11.6 | 5.2 | 4.6 | 21.0 | 79% |
| Inspire F1 | tuned | 52 | 4.3 | 5.1 | 9.8 | 4.8 | 4.3 | 17.8 | 85% |
Per-object breakdown
Representative episodes
Overlay videos
Headless ViewerGL renders with the ground-truth object pose overlaid as a translucent green
ghost (per-shape opacity from PR #3053).
The solid object is simulated; the ghost is where the real object actually was.
Honest limitations
- Grasp outcomes are chaotic. Millimeter-scale differences at contact onset flip episodes between firm grasp and slip. Repeated rollouts within one solver instance are nearly bit-stable, but fresh solver instances shift metrics by up to ~10% and can flip marginal lifts (4 episodes × 8 runs: within-instance ADD std ≤ 0.70 cm, 2 episode(s) with non-deterministic lift outcome). Median metrics over many episodes are the right lens, not single episodes.
- Geometry gap. URDF collision meshes of the hands and CoACD convex decompositions of the scanned objects approximate the real contact geometry — convexification rounds off concavities that matter for marginal grasps; fingertip compliance (rubber pads, tactile skins) is not modeled.
- Unknown object inertia. Object masses are not released with the dataset; a single tuned mass scale per hand is a crude stand-in for per-object mass.
- Calibration and tracking noise. Object 6D tracking and camera-to-robot calibration carry errors of up to a few centimeters in places; two episodes were auto-flagged as calibration outliers by the palm-distance check.
- Mimic-joint approximation. The Inspire F1 linkage is modeled through URDF mimic constraints rather than the real four-bar mechanics.
Reproduce
All code lives on the eric/hrdexdb-replay branch of
eric-heiden/newton under scripts/hrdexdb/.
git clone -b eric/hrdexdb-replay https://github.com/eric-heiden/newton cd newton && uv sync --extra examples && uv pip install huggingface_hub cma imageio-ffmpeg git clone https://github.com/snuvclab/HRDexDB ~/repos/HRDexDB # robot URDFs cd scripts/hrdexdb uv run python download_all.py # fetch episodes (no videos) uv run python replay.py --hand allegro_v5 --object banana --scene 2 uv run python tune.py --hand allegro_v5 # CMA-ES calibration uv run python evaluate.py --hand allegro_v5 --tag tuned --params results/tuned_params_allegro_v5_cmd.json uv run python render.py --hand allegro_v5 --object banana --scene 2 # overlay video