Real2Sim Parameter Estimation

Executive Summary

Newton should expose real2sim as a toolbox around parameter maps, experiment losses, solver notifications, batched rollout evaluators, and uncertainty-aware optimizers. Differentiable simulation is one important backend, but practical robot calibration also needs finite differences, simulation-based inference, Bayesian optimization, and classical inverse-dynamics identification because contacts, solver caches, actuator firmware, and parameter identifiability all break the clean gradient story.

Problem Shape

Parameter estimation asks for physical settings \( \theta \) that make simulated observations \( \hat{x}_{1:T}(\theta) \) agree with real measurements \( x^r_{1:T} \). A practical Newton objective should look like:

\[ \theta^* = \arg\min_{\theta \in \Theta} \sum_{e \in \mathcal{E}} \sum_{t=1}^{T_e} \left\| W_e\left(h_e(s_t(\theta)) - y^r_{e,t}\right)\right\|^2 + R(\theta) \]

Here \(h_e\) can be a joint encoder readout, object pose, force/torque signal, tactile depth map, image feature, or task return. The parameter set \(\Theta\) should enforce physical bounds: positive mass and inertia, dissipative friction, valid contact coefficients, stable solver settings, and explicit update flags for solver caches.

For Bayesian methods, the same pieces become a posterior:

\[ p(\theta \mid D) \propto p(D \mid \theta) p(\theta), \quad p(D \mid \theta) \approx \exp\left(-\frac{1}{2} \sum_t \|r_t(\theta)\|_{\Sigma^{-1}}^2\right). \]

Literature Groups

1. Robot inertial and friction identification

The classical arm literature is the right starting point for Franka, UR10, KUKA iiwa, Kinova, and similar fixed-base arms because it makes the best use of high-rate encoders and torque/current logs. BIRDy is the broad map: it compares inverse-dynamics least-squares variants, instrumental variables, maximum likelihood, closed-loop input/output-error methods, Kalman approaches, neural estimators, and physically consistent semidefinite programs on serial manipulators. Its practical lesson is not that one estimator wins everywhere; it is that excitation design, filtering of velocity/acceleration, torque quality, and cross-validation dominate the result.

Wensing, Kim, and Slotine add the missing physical constraint layer: identified masses, centers of mass, and inertia tensors must correspond to a realizable mass distribution. Their LMI formulation is important for a simulator because a numerically good least-squares fit can still create impossible inertias and unstable rollouts. The later geometric observability work explains another simulator failure mode: many link inertial parameters are only identifiable up to transfers along the articulated chain. For Newton, this means the toolbox should report identifiable combinations and posterior spread, not quietly pretend that every URDF field was independently observed.

Robot-specific studies make this concrete. KUKA LBR iiwa identification with feasibility constraints estimates base and feasible physical parameters including friction from optimized excitation trajectories, using constraints to keep the model usable in simulation. Franka Panda identification with penalty-based feasible optimization is directly relevant to this report's prototype: it shows that a collaborative arm with rich internal sensing still needs careful friction modeling and feasibility projection. Inertial identification including friction and motor dynamics is also relevant because real arms often expose motor currents and joint torques through a controller stack; the difference between link friction, motor inertia, actuator lag, and hidden compensation can be larger than the residuals we are trying to fit.

Newton fit Implement a robot-arm initializer that estimates feasible link inertials, payload mass/CoM, joint friction, and armature from logs before running expensive rollout matching. The output should be priors and bounds for Newton, not final truth. Newton should preserve the distinction between base parameters that are observable from inverse dynamics and simulator fields that are merely one feasible realization.

2. Bayesian and simulation-based inference

Simulation-based inference starts from a different premise: treat the simulator as the model being calibrated, and search over parameters using real rollouts. SimOpt adapts a distribution over simulator parameters from a small number of real executions interleaved with policy training; its core value is closing the loop with real behavior instead of hand-tuning randomization ranges. BayesSim makes the same idea explicitly Bayesian and black-box: it learns a posterior over simulator parameters from simulated rollouts and real observations using likelihood-free inference, which is attractive for Newton solvers where an analytic likelihood is rarely available.

BayRn shifts the target from trajectory agreement to task performance: Bayesian optimization searches the parameters of the randomization distribution so policies trained in simulation improve on the target system. Probabilistic Differentiable Simulation and CSVGD combine the Bayesian view with gradients and GPU-parallel particles; the paper is especially relevant because it includes a 7-DOF arm exciting an object with unknown mass configuration and shows why multimodal posteriors matter. Online BayesSim, DROPO, AdaptSim, and related task-driven variants push the same theme: sometimes the best simulator parameters are those that make the downstream policy robust, not those that minimize a narrow state residual.

Newton fit Newton should have a batched particle evaluator as a first-class API. A particle is just a candidate parameter map over body_mass, body_inertia, joint_friction, shape_material_mu, solver settings, and object asset fields. Once Newton can tile hundreds of worlds with different parameter maps, finite differences, CEM, CMA-ES, Bayesian optimization, SMC, and SVGD all become library-level choices rather than one-off scripts.

3. Differentiable simulation and contact gradients

Differentiable physics promises the shortest path from data to parameters, but contact makes the promise conditional. End-to-End Differentiable Physics showed that differentiating through an LCP physics layer can learn physical parameters and support control. Nimble brings differentiability to feature-complete articulated rigid bodies with hard contact. Dojo uses a variational integrator and implicit differentiation through a relaxed contact problem, targeting stable robotics gradients rather than only fast rollouts. DiffMJX is a useful recent case study because it tries to keep hard-contact realism while supplying soft gradients that remain useful for learning and control.

The cautionary thread is equally important. Gradient correctness in differentiable physics with contacts shows that simulators can be differentiable and still supply gradients that are wrong for position, velocity, control, or parameter updates under the chosen contact model. This surfaced in the Newton prototype too: a gradient audit found a mismatch for the recurrent rollout kernel, so the report prototype uses finite-difference Adam over Warp kernels. That is not a retreat from differentiability; it is the contract Newton should enforce before exposing a gradient backend to users.

Newton fit Treat autodiff as an optional accelerator behind a GradientProvider interface. Each provider must pass a finite-difference audit on representative regimes: free-space arm motion, payload excitation, intermittent contact, sliding contact, resting contact, and solver-parameter changes. Newton should store these audits next to examples so a new solver cannot accidentally advertise gradients that only work in trivial cases.

4. Objects manipulated by robot arms

Object real2sim is the most valuable extension beyond robot-arm calibration. The object inertial-parameter survey separates methods by sensing mode and use case: some identify mass and inertia from force/torque while the object is held, others exploit grasping and manipulation, and others use vision or exploratory actions. Fazeli, Kolbert, Tedrake, and Rodriguez analyze rigid bodies undergoing frictional contact and estimate both inertial parameters and contact forces, which is exactly where a clean inverse-dynamics formulation breaks down.

Recent systems move toward deployable asset pipelines. Learning Object Properties from Robot Proprioception uses differentiable robot-object interaction and only joint encoder signals to infer object inertia and softness; the important idea is that the robot itself is the sensor. Scalable Real2Sim goes further by using a pick-and-place setup with torque sensors and camera input to build simulation-ready assets: visual geometry, collision geometry, and physical properties. Splatting Physical Scenes for End-to-End Real-to-Sim points in the same direction for imperfect robot data: jointly refine scene geometry, robot pose, and physical parameters rather than treating perception as a fixed input.

Phys2Real adds a useful prior mechanism. It uses visual-language estimates and geometric reconstruction to propose uncertain physical parameters, then refines them online with interaction data. Fast object inertial parameter identification for collaborative robots is the complementary pragmatic path: use targeted interactions to estimate reusable object properties quickly. For Newton, the actionable part is not "trust the VLM"; it is "accept cheap priors with explicit uncertainty, then let robot interaction overwrite them."

Newton fit Create an object asset calibration path parallel to robot-arm ID. It should estimate body_mass, body_inertia, body_com, collision geometry scale/offset, contact friction, restitution, and possibly deformable material fields. It should accept logs from pick, lift, shake, push, slide, drop, and place experiments, with camera/object-pose observations as optional losses.

5. Tactile and residual model calibration

Tactile and residual methods are less central for the first rigid-arm toolbox, but they shape the right API. DiffTactile couples a FEM elastomer model, contact dynamics, and optical response so tactile data can supervise physical-property refinement. DOT-Sim uses MPM and a residual optical renderer to calibrate optical tactile sensors from a small number of demonstrations and then transfer tactile policies. Few-shot neural differentiable simulators show another path: keep a physics core but add a learned contact residual that is grounded by a small amount of real contact data.

Dynamics as Prompts and in-context system-identification work points to a different interface: instead of optimizing every parameter online, use history-conditioned models to propose updates quickly. This is useful when a factory robot changes tools, a payload changes, or a contact surface wears down. The danger is interpretability drift: a learned residual can improve a rollout while hiding that the physical friction or actuator model is wrong.

Newton fit Keep the first toolbox physics-first, but make observations modular. A Newton real2sim experiment should be able to attach joint residuals, force/torque residuals, object-pose residuals, tactile-image residuals, and learned feature residuals without changing the parameter optimizer. Learned residuals should be reported separately from physical parameters.

Method contrast

Synergies for Newton

Newton Toolbox Design

Parameter families

Updated proposal: build the toolbox in layers

Minimal API sketch

from dataclasses import dataclass

@dataclass
class ParameterSpec:
    name: str                 # "joint_friction"
    indices: list[int]
    transform: str            # "exp", "softplus", "identity"
    lower: float
    upper: float
    notify: SolverNotifyFlags
    prior_mean: float | None = None
    prior_std: float | None = None

def evaluate(theta_raw, experiment, model, solver):
    theta = parameter_map.decode(theta_raw)
    parameter_map.assign(model, theta)
    solver.notify_model_changed(parameter_map.notify_flags)
    simulated = rollout(model, solver, experiment.controls)
    return experiment.loss(simulated, experiment.real_observations)

Prototype: Franka Parameter Fit

The Newton branch eric-heiden/param-estimation contains a runnable prototype at newton/examples/robot/example_robot_param_estimation.py. It loads the Franka FR3/Panda arm through Newton, reads armature/friction metadata, generates a noisy synthetic-real joint trajectory, and estimates three positive real2sim parameters using finite-difference Adam over Warp rollout kernels.

# in /home/horde/repos/newton-worktrees/param-estimation
uv run python -m newton.examples robot_param_estimation \
  --iterations 500 --steps 260 --noise-std 0.0015 \
  --learning-rate 0.03 \
  --output param_estimation_result_noisy.json

Raw data: data/franka_param_estimation_result.json. The finite-difference path is deliberate: it matches the toolbox recommendation that Newton first provide robust solver-agnostic calibration, then enable autodiff per solver only after gradient checks pass.

Practical Recommendations

1. Build the parameter registry before adding more optimizers. The literature only transfers cleanly if Newton has typed specs for fields, bounds, transforms, priors, units, and solver notifications.
2. Use classical arm ID as the first initializer. For Franka, UR10, KUKA, and Kinova arms, feasible inverse-dynamics estimates should seed payload, friction, armature, and inertial priors before rollout matching.
3. Make batched black-box rollouts the default path. Finite differences, CEM, Bayesian optimization, SMC, and posterior particles are immediately useful and work across Newton solvers.
4. Expose uncertainty as a product feature. Posterior particles, spectra, and parameter-correlation plots should be shown whenever payload mass, armature, friction, CoM, and contact parameters trade off.
5. Add differentiable backends only behind gradient contracts. Every differentiable parameter family needs a finite-difference audit on representative trajectories, especially through contacts and recurrent integration loops.
6. Treat object assets as first-class calibration targets. Robot-arm ID, object-property estimation, contact calibration, and geometry correction should share one experiment/parameter API.
7. Keep learned residuals and priors separate from physical fields. VLM priors, in-context updates, tactile features, and neural residuals are useful, but the report should show when they changed a prior versus when Newton changed a physical parameter.

Source Trail

• BIRDy and inertial parameter identification survey
• Physically consistent inertial parameter identification with LMIs
• Geometric observability for inertial parameter identification
• KUKA LBR iiwa identification with physical feasibility constraints
• Franka Panda feasible dynamic parameter identification
• Inertial parameter identification including friction and motor dynamics
• SimOpt: adapting simulation randomization from real rollouts
• BayesSim: likelihood-free Bayesian simulator parameter inference
• BayRn: Bayesian optimization for domain randomization
• Probabilistic Differentiable Simulation and CSVGD
• Dynamics randomization for robotic arm sim-to-real transfer
• Online BayesSim for parameter inference and policy improvement
• DROPO: offline domain randomization for sim-to-real transfer
• AdaptSim: task-driven simulation adaptation
• End-to-end differentiable physics for learning and control
• Nimble differentiable articulated rigid bodies with contact
• Dojo differentiable physics engine for robotics
• DiffMJX hard contacts with soft gradients
• Gradient correctness in differentiable physics with contacts
• Object inertial parameters in grasping and manipulation survey
• Parameter and contact force estimation under frictional contact
• Learning object properties from robot proprioception
• Fast object inertial parameter identification for collaborative robots
• Scalable Real2Sim physics-aware asset generation
• Splatting physical scenes for end-to-end real-to-sim
• Phys2Real with VLM priors and online adaptation
• DiffTactile differentiable tactile simulator
• DOT-Sim tactile real-to-sim calibration
• Few-shot neural differentiable simulator for rigid contact
• Dynamics as Prompts for in-context system identification

Last updated: 2026-06-05 UTC.