manosim in Newton: replaying human grasps from HUG

manosim ships MANO hand models as MJCF, built for Human Universal Grasping (HUG). Its README montage shows those hands grasping objects in MuJoCo. This report brings the same thing up in Newton: the unmodified manosim MJCF driven by real 10 Hz MANO trajectories from HUG-Bench recordings, against the metric-scale scanned objects the humans actually grasped, with every clip rendered headless through ViewerGL. Getting there needed three fixes in Newton's MJCF importer and turned up four further ways an MJCF that runs happily in MuJoCo does not yet run in Newton — including one that silently imports a critically damped drive as an undamped one.

Generated 2026-07-28 · Newton 1.5.0.dev0 at a586d5c7 (branch eric/manosim-mjcf, forked from upstream main) · manosim 75c77c0 · data: kevinywu/hug-bench + kevinywu/1m-hugs · NVIDIA RTX PRO 6000 Blackwell (MIG 1g.24 GB) · headless ViewerGL on Xvfb.

The montage

6 independent scenes, each one scanned HUG-Bench object resting on a ground plane while the manosim MANO hand replays a human's recorded approach and closure, then lifts. Nothing here is scripted animation: the hand is a dynamic articulation driven only by joint targets and a wrist wrench, and every object is a passive rigid body that moves only because something touched it.

Left to right, top to bottom: picnic basket (large_2), bundled socks (medium_4), cabbage (large_4), lock (small_2), headphones (large_1), salt shaker (small_4). Only the visual meshes are drawn; the convex collision hulls sit a few millimetres away from them and are hidden, since drawing both puts two surfaces in the depth buffer that z-fight. Rendered with ViewerGL(headless=True) → get_frame() → H.264, then tiled with ffmpeg. Each clip reports its own outcome below; borderline grasps do flip between runs, since float32 contact solving on the GPU is not bit-reproducible.

Individual grasps

Whole-scene replay

The montage isolates one object at a time. This variant keeps a scene as it was scanned — all five objects at their recorded positions on the table — and replays the hand through it. It is the more honest picture of the data, and it also exposes the dataset's main limitation: HUG-Bench stores one resting pose per object and no per-frame object tracking, so a replay is only valid up to the moment the human first picks something up. Every window below is therefore taken around the earliest grasp in its recording.

Scene test/small_2: hacky sack, lock, match box, pepper shaker and tape measure at their scanned positions, with the recorded hand reaching in for the match box.

Is any of this actually simulated?

A fair question to ask of any replay video: is the hand really picking the object up through contact, or is something in the driver moving it? Three checks, all of which the pipeline runs.

Take the contact away and nothing happens

The strongest test is an ablation. Each successful grasp is run twice through the identical protocol; the only difference in the second run is that hand-object collision pairs are filtered out, so the hand sweeps straight through. If the object still rose, something other than contact was lifting it.

Every one of the 8 objects lifts when the hand can touch it and 8 of them do not move at all when it cannot (largest residual 0.0 mm). Median 795 hand-object contact points are carrying the object at the moment the lift starts.

The object is a passive rigid body with a free joint. Nothing writes to its state, no constraint attaches it to the hand, and the hand is never teleported — only Control.joint_target_q for the fingers and Control.joint_f for the wrist are written, both before the solver step. The lift is contact and friction, or it is nothing.

What the physics actually touches is not what you see

The apparent penetration in these clips is real, and it is a property of the released assets rather than of the solver. HUG-Bench ships a watertight visual.obj for looks and a separate CoACD convex decomposition for collision, and only the decomposition takes part in contact. Where they disagree, the fingers correctly stop on a surface that is not being drawn.

Measured over all 60 objects, the collision hull's worst-case departure from the visual surface has a median of 2.98 mm, a 90th percentile of 9.7 mm, and reaches 79.51 mm on the picnic basket; 17 objects exceed 5 mm. That is the budget within which a finger can look wrong while being right.

Where the sudden movements come from

The hand is not one of them. Its position matches the pose it is being commanded to step for step: over a full rollout the largest frame-to-frame wrist movement is 24.63 mm against a commanded 24.44 mm on the picnic basket, 13.78 against 13.77 on the lock, and 14.07 against 14.06 on the headphones. The wrist is following, not jumping — and the commanded step is simply how fast the person's own hand was moving, at half playback rate.

The objects are a different matter. Two clips show one jolting rather than being smoothly picked up. Frame-to-frame object displacement is 5 mm for a well-behaved lift — exactly the 0.15 m/s of the standardized lift at 30 fps — and stays there for most objects. The picnic basket reaches 94 mm in one frame. It is also the object with the worst hull-versus-visual gap (80 mm) and the lowest density in the release (34.7 kg/m³, about a twentieth of water, for a wicker basket). A hand that appears to be several centimetres clear makes contact with an invisible hull and knocks a body that is far too light. Both halves of that come from the asset, not the simulation.

What had to change in Newton

MJCF slide + ball joints did not import

manosim's wrist is the standard MJCF idiom for an actuated floating base — three slide joints followed by a ball, each carrying its own damping, armature and <position> actuator:

<body name="mano_body_0" pos="0 0 0">
  <joint name="wrist_tx" type="slide" axis="1 0 0" limited="false" damping="5.0"/>
  <joint name="wrist_ty" type="slide" axis="0 1 0" limited="false" damping="5.0"/>
  <joint name="wrist_tz" type="slide" axis="0 0 1" limited="false" damping="5.0"/>
  <joint name="wrist_rot" type="ball" limited="false" damping="5.0" armature="0.01"/>

Newton's importer treated ball as terminal. It discarded the three slide axes it had already parsed and emitted a 3-DOF JointType.BALL, then tripped over its own DOF bookkeeping:

ValueError: JOINT_DOF attribute 'mujoco:dof_passive_damping' has invalid DOF index 3 (joint has 3 DOFs)

The fix rebuilds that combination as a free joint carrying the authored per-DOF properties, keeps a lone ball joint mapping to JointType.BALL, and rejects any other ball combination with an actionable message instead of a DOF-index error. Two supporting changes came with it: add_joint_free() gained optional per-DOF axis configs so importers can preserve damping and armature, and the merged-joint DOF map now records a span rather than a single index — without which a <position joint="ball" gear="0 1 0"> actuator set gains on only the first of the ball's three DOFs. A third, independent bug surfaced while testing: the lone-ball import path passed armature and friction to add_joint_ball() but not damping, so every MANO finger joint was silently losing its authored damping="0.01".

The unmodified manosim MJCF now imports as 16 bodies, 16 joints (1 free + 15 ball), 51 DOFs and 67 coordinates, with the wrist's damping 5.0, armature 0.01 and all six actuator gains intact. Seven regression tests cover it. Six fail with the source fix stashed — five with the original DOF-index ValueError, one because two ball joints in a body used to be silently accepted; the seventh is the guard that a lone ball joint still imports as a ball joint, which by construction passes either way. The full MJCF importer module (268 tests) still passes.

Free joints cannot be actuated, and nothing says so

Mapping the wrist to a free joint is kinematically exact but has a consequence: SolverMuJoCo emits mjJNT_FREE for free joints, and MuJoCo does not allow actuators to target those. The six wrist actuators manosim authors are therefore imported, stored, and never produce force. The hand simply sags: commanded to hold station at 30 cm with kp=300, it drooped 213 mm and settled on the ground.

The importer now warns when an MJCF actuator targets a joint Newton represents as free. The replay drives the wrist explicitly instead, through Control.joint_f, which is honoured on free joints — verified by cancelling gravity with joint_f[2] = m·g and watching the hand hover to within 0.05 mm over a second. That gives a 6-DOF PD wrench with gravity feed-forward, standing in for the arm that carried the hand in the recording, and it tracks the recorded wrist to a median 2.16 mm.

MJCF dampratio is not converted

<position kp="300" dampratio="1.0"/> means "critically damp this drive"; MuJoCo compiles it to kv = 2·dampratio·√(kp·dof_M0). Newton carries kp across as joint_target_ke but leaves joint_target_kd at zero, so an MJCF that is stable in MuJoCo arrives in Newton with an undamped drive. Reconstructing it needs the diagonal of the generalized mass matrix, which newton.eval_mass_matrix provides; for this hand that is 0.4 kg on the wrist translations and ~1e-3 kg·m² per finger joint. A first attempt that used joint_armature as a stand-in for the effective inertia under-damped the wrist by a factor of 60 and blew the hand up.

Model edits must precede the solver

SolverMuJoCo bakes gains, force ranges and damping into its MuJoCo spec at construction, and notify_model_changed(ModelFlags.JOINT_PROPERTIES) does not rebuild the actuators. A sweep over joint_effort_limit from 0.02 to 0.3 N·m produced bit-identical results — the setting was simply not reaching the solver — which read as "force limiting does not help" when in fact it was the single most important fix in the whole pipeline. Every model edit now happens before the solver is built.

Contact buffers overflow silently

A watertight CoACD decomposition (up to 15 convex parts per object) plus a 16-shape hand generates far more contacts than an estimate taken from the initial state allocates: a five-object scene reports 9,104 contacts at rest, and a hand closing on an object several times more again. Both SolverMuJoCo's nconmax and CollisionPipeline's contact capacity default to a value derived from the initial state, and neither overflow raises — nconmax drops constraints, the pipeline drops contacts, and the scene explodes several steps later where the cause is no longer visible. Both buffers are now sized from a measured resting contact count with 6× headroom, because a closing hand produces several times the contacts of the same scene at rest.

A ball-joint position drive flutters, and the damping that should stop it is dropped

With contacts, gravity and drive all removed, the hand is exactly static — max|q̇| == 0.0, so Newton's articulation dynamics are sound. Switch the imported drive back on, with its target set to the hand's own current pose, and a residual of 7e-6 rad/s appears on the first step and grows to divergence in about 0.23 s. It is pose-dependent (identity is fine), roughly independent of the time step, and unaffected by gains.

The mechanism is a property of MuJoCo's ball-joint transmission. Actuator length is the nonlinear map ⟨log(q), gear⟩, but the moment is hard-coded to the constant gear vector — the moment is not the gradient of the length. Three gear = eᵢ actuators therefore produce a generalized stiffness

K = kp · Jr⁻¹(φ),   φ = log(q_joint)     antisym(K) = (kp/2)·[φ]×

which is non-symmetric. A circulatory stiffness like that is a classical flutter source: its eigenvalues are kp and kp·(h·cot h ± i·h) with h = |φ|/2, so the system has a right-half-plane root unless the total damping d satisfies

d² > m·kp·(θ/2)·tan(θ/2)      ⟺      ζ ≳ θ/4

At φ = 0, Jr⁻¹ = I and K is symmetric — which is precisely why the model is bit-exactly static at the identity pose and unstable at every recorded one. Measuring the stiffness by central differences on qfrc_actuator reproduces K = kp·Jr⁻¹(φ) to 1e-10 relative and antisym(K) = (kp/2)[φ]× exactly; 33–67% of the commanded torque is perpendicular to the actual rotation error.

Both terms of the criterion are measurable here. The worst finger joint in the recorded poses reaches θ = 1.24 rad (71°) with effective inertia 1.0e-3 kg·m², predicting a critical damping of 0.115, or 0.137 once corrected for the discrete time step. Sweeping the real 15-joint hand over a 2.7 s recorded window agrees to within 6%:

The criterion is not academic: it decides results. Running the whole study at the marginal value 0.1 rather than 0.15, with everything else identical, costs 3 rollouts to instability (57/60 stable against 60/60) and 2 successful grasps (14 lifted against 16).

Only damping removes the unstable eigenvalue. Armature rescales the growth rate as 1/√m, gain reduction as √kp, and a smaller time step not at all — the eigenvalue is continuous-time (+49.4/s in the dt → 0 limit). Passive joint_damping and actuator kv are interchangeable; only their sum matters. A standalone reproduction — one ball joint, fixed base, no gravity, no contacts, target equal to the current pose — is at scripts/repro_ball_drive.py.

Rigid fingers need a force limit

The most consequential finding, and the one that turned "everything explodes" into "everything is stable". A MANO fit is a fit to a soft hand: the recorded joint angles place the fingers up to a centimetre inside the object being held. Replay those angles against a rigid mesh and the position drive sees a joint error it can never close out, and answers with roughly 120 N of grip on a 10 g match box. Saturating each finger drive at 0.4 N·m — about 4 N at the fingertip, a firm human squeeze — turns that into a bounded force. Every rollout in this report is stable; before the limit, most ejected their object across the room.

Still open upstream

Three of the findings above are fixed on the branch; three are reported here but not yet addressed, and they are worth stating precisely because each fails silently:

• Convert <position dampratio> into joint_target_kd. parse_actuators reads only the kv attribute, so a critically damped MJCF drive imports as an undamped one. The value is already stashed in mujoco.actuator_biasprm and is resolved for CTRL_DIRECT actuators, but the JointTargetMode.POSITION path reads joint_target_kd directly and drops it. Even a warning when a <position dampratio=…> imports with joint_target_kd == 0 would have saved most of the debugging in this report.
• Consider a native quaternion drive for ball joints. Decomposing into three gear = eᵢ MuJoCo actuators inherits MuJoCo's non-gradient moment arm and the flutter it implies. A drive computing τ = kp·log(q_target·q⁻¹) − kd·ω is unconditionally restoring at any joint angle.
• Make buffer overflow loud. nconmax and the collision pipeline's contact capacity both print a line and carry on with dropped constraints. An opt-in strict mode that raises would turn a confusing downstream explosion into an immediate, correct diagnosis.

Is the data actually saying what we think?

Two geometric assumptions sit under every number above, and both were checked rather than assumed.

MANO pose → Newton joint coordinates

The dataset gives a wrist transform and 15 axis-angle rotations per frame; the hand needs 67 generalized coordinates. Writing them and running forward kinematics should place every joint on its recorded 3D landmark. Across all 14699 recorded frames in 12 scenes it does, to a median of 1.89 mm.

The one outlier — up to 40 mm, always at the thumb base — is not a convention error but a real limitation: aria_data.pkl re-optimises the MANO shape parameters every frame (β spread 0.28–0.46), while the simulated hand has one fixed geometry. A fixed-shape sim hand cannot follow a hand whose bone lengths change, and the thumb carpometacarpal joint is where that shows most.

Where the objects are

HUG-Bench gives each object a watertight visual.obj, a CoACD decomposition and a world_position — but no orientation. Placement is only correct if the mesh is already world-aligned and merely centred. Testing that properly meant back-projecting the masked depth of 900 benchmark frames into the world frame and comparing against the mesh placed by pure translation: median one-sided Chamfer 5.15 mm, median silhouette IoU against the released object masks 0.79. Sweeping 90°, 180° and random reorientations, and keeping only the 506 (object, rotation) pairs where the rotation is geometrically observable at all, 0 beat the identity placement. ICP recovers a median 6.8° but moves the surface by only 0.87 mm — it is sliding along symmetry axes — and leaves 8.3 mm of residual translation, which is the real accuracy of world_position.

Results

Every object with a recorded grasp, taken through the same protocol: settle on the ground, replay the human's first recorded approach and closure at half speed, then a standardized 18 cm vertical lift and a 0.6 s hold. Success means the object is still off the ground at the end.

Green = lifted and held. Configuration: capsule hand, 8 substeps at 30 fps (dt = 1/240), finger drive saturated at 0.4 N·m, finger damping 0.15, object contact stiffness 2500, SolverMuJoCo with the Newton solver, implicitfast integrator and an elliptic friction cone. Whole study: 9 minutes.

manosim ships a second hand — mesh_hand, one convex hull per bone instead of a capsule, closely matching the MANO surface — and recommends the capsule version for contact work. Rerunning the whole study with it, everything else identical, gives 60/60 stable and 16/60 lifted: the same headline numbers as the capsule hand. They are not the same grasps, though — only 12 objects succeed under both, 4 succeed with capsules alone and 4 with hulls alone. Which of two very similar collision proxies you pick decides about a third of the outcomes, which is a fair measure of how marginal these grasps are once the hand is rigid.

What I would tell the next person

• The manosim assets need no MANO registration. The built MJCFs and bone meshes are committed to the repo; registration is only needed to re-fit a new hand shape. Likewise the HUG-Bench object assets and hand recordings download anonymously.
• Import the MJCF, do not hand-edit it. Rewriting the wrist as a <freejoint/> is a tempting workaround for the importer bug and it silently discards the authored wrist damping and armature, which matters for stability.
• Budget contacts from a measurement, not a guess, and treat any "exceeded MJWarp limit" line as a hard failure rather than a warning — it is the most common cause of an explosion whose real trigger is 50 steps earlier.
• Force-limit anything driven from human motion capture. Kinematic fits are not contact-consistent; a position drive replaying them will always command penetration.
• Check what the drive does at its own target before blaming contacts. Setting the target to the current pose and watching |q̇| is a two-minute test that would have saved several hours here.
• Verify that a model edit actually reached the solver. Two independent mechanisms discard them silently: gains and force ranges are baked into the MuJoCo spec at construction, and notify_model_changed only pushes what its flag covers. Read the value back off solver.mjw_model rather than trusting the write.
• A <position> actuator on a ball joint is not a passive PD controller. Away from identity its stiffness is non-symmetric and it needs a damping ratio above roughly a quarter of the joint angle just to stay bounded. A drive computing τ = kp·log(q_target·q⁻¹) − kd·ω directly would be unconditionally restoring; decomposing into three gear = eᵢ actuators is not.
• The dataset is a grasp dataset, not a manipulation dataset. One resting pose per object and no object tracking means only the first grasp in each recording is replayable against the scanned scene. Later grasps in the same recording look identical in the hand data and aim at empty space.

Reproducing

# Newton with the MJCF importer fix
git clone git@github.com:eric-heiden/newton.git && cd newton
git checkout eric/manosim-mjcf

# manosim assets (prebuilt MJCF, no MANO registration needed)
git clone https://github.com/KevinyWu/manosim.git

# HUG-Bench object assets (~98 MB) and hand recordings (~30 MB per scene,
# streamed out of the scene tars rather than downloading all 47 GB)
hf download kevinywu/1m-hugs --repo-type dataset --include "val/**" --local-dir data/hug_bench
hf download kevinywu/1m-hugs --repo-type dataset --include "test/**" --local-dir data/hug_bench
scripts/fetch_scenes.sh data/scenes test/small_2 test/small_1 val/medium_3   # etc.

# validate, then replay
uv run python scripts/validate_pose_mapping.py --out out/pose_mapping_validation.json
uv run python scripts/validate_object_frame.py
uv run python scripts/run_study.py --effort 0.4 --out out/study_capsule.json
uv run python scripts/render_montage.py --study out/study_capsule.json
uv run python scripts/render_scene.py --scenes small_2

Newton branch eric/manosim-mjcf · importer fix + regression tests · manosim · Human Universal Grasping · arXiv:2606.17054