Live inspection and tuning of Newton simulations

The three conditions receive the same application code, references, solver, fitting libraries, task criteria, and working-directory conventions. Persistent conditions share the same SimulationSession and scenario helpers. IPython bypasses Newton’s MCP transport and Python executor; this comparison therefore measures execution-interface effects with simulation integration already supplied, rather than the cost of adding Newton-specific support to generic IPython.

The frozen registration specifies 61 independent GPT-6 Astra contexts at xhigh reasoning: 54 primary runs and seven exploratory corrected-IPython runs. The primary set crosses three conditions with nine existing cases and nine real-data replicates. Existing cases comprise two variants each of Panda posture tracking, Allegro posture tracking, and recorded HUG hand-motion replay, plus three synthetic Panda calibration instances. Real-data replicates use identical initial parameters and recordings; they measure agent variation on one robot dataset.

Existing cases allow 600 seconds and 12 completed candidates; real identification allows 1,200 seconds and 60. Offline fitting is permitted equally and does not count as a physical rollout. Contexts run serially in fresh workspaces, in a frozen seeded order. Source was committed, and prompts, data, thresholds, and order were recorded and hashed before the first scored context; development attempts are separate.

Four preliminary development contexts are excluded and preserved separately: one Panda IPython integration run and three real-data attempts. The real-data attempts overlapped while shared source and asset handling changed, so their timings cannot support a controlled comparison. Their original outcomes remain unchanged. A separate fresh check of the three final development configurations after source freeze passed training and held-out gates; it does not retroactively make those contexts eligible.

Startup-inclusive time includes application or kernel construction and agent execution, including bridge startup and kernel connection. Common immutable preprocessing is measured separately and excluded equally. Verification runs after agent exit and child-process cleanup. Newton Python source, task, input, geometry, and manifest hashes are checked before and after verification. Physics measurements remain distinct from eligible study success. All failures and timeouts are retained; elapsed time and input-plus-output tokens are reported for every run, with paired efficiency comparisons restricted to matched successful cases. Cached input is a subset of total input; uncached input plus output is reported separately.

The experimental public module newton.mcp adds no required dependency. A standard stdio MCP bridge forwards requests over authenticated loopback TCP to the application. Transport threads enqueue work; the thread owning simulation and rendering executes it through pump() or run(), including while paused. This internal connection is not a Streamable HTTP endpoint. The external boundary follows the MCP transport specification.

The compact profile exposes describe, execute, observe, and rebuild, with the newton_ prefix. Python can call the full structured interface through session.dispatch(). Queries use attribute-frequency metadata and world/body/shape/joint filters. Validated numeric edits update arrays in place and notify native solver caches; coupled inspection uses public child bindings. Reset restores initial state, control, time, and caches while retaining tuned model properties. Rebuild replaces the model/solver pair. Checkpoints contain public physical state, not arbitrary application history or every hidden solver variable.

Execution uses a persistent registered Python module. Imports, variables, functions, classes, and supported Warp kernels survive calls and physical resets. Live model, state, control, and solver bindings refresh around managed state changes. Ordinary aliases and captured default arguments retain normal Python reference semantics. Rebuild clears the workspace. This supports ordinary Python, including dataclasses and recent-cell source inspection, but not IPython magics, top-level await, or rich notebook display.

The previous executor discarded Python definitions between calls and could not recover cell source for Warp compilation. The preserved before/after interactions and failing/passing regressions establish that cross-cell imports, functions, dataclasses, and Warp kernels now work. This is correctness and usability evidence; the current agent comparison evaluates the revised executor and does not isolate the timing effect of namespace persistence.

# First call: preserve an import and function.
import numpy as np

def joint_positions():
    return np.asarray(state.joint_q.numpy()).copy()

# Second call: the function resolves the current state binding.
session.dispatch("step", {"count": 1})
joint_positions()[:7]

A final expression is displayed; explicit result = takes precedence for that call, and _ retains the last non-None value. Large arrays and opaque objects receive bounded summaries without calling arbitrary representation methods. Output is limited to 16,384 captured characters and 65,536 JSON characters; source retention is bounded. A result-encoding failure after successful execution preserves validity and warns against repeating the mutation.

Compilation errors leave the scene unchanged. Runtime exceptions preserve workspace variables but pause and invalidate the session: there is no automatic rollback or guess that an error was harmless. Explicit recovery="inspect" allows diagnosis; recovery="acknowledge" asserts that the caller has verified or restored coherence and leaves playback paused. Rebuild provides a fresh scene. Successful arbitrary Python likewise does not prove model/solver coherence. Trusted execution retains process privileges and cannot safely preempt running Python.

The primary IPython condition uses ipython-mcp revision c2fa8d6, without server edits. Its kernel supplies IPython semantics, while the pinned MCP server forwards text; image inspection requires file access. It does not invoke Newton’s executor or its exception-recovery guard.

A separate SDK characterization reproduced a protocol defect. The server stops waiting for a shell reply after 30 seconds without correlating subsequent replies to their execution request. A late exception from the first cell was returned as the second cell’s result even though the second cell executed successfully. The error also arrived with isError: false; accounting recognizes its textual failure marker.

The disclosed corrected control matches parent message IDs, discards stale replies, and permits a 300-second wait. A deliberately short-deadline regression verifies correct subsequent reply delivery. These are two changes, so their individual effects are not isolated. Seven separately labeled sensitivity contexts cover one case per existing task and three real-data replicates; they do not replace upstream failures in the primary comparison.

The physical measurements come from the LIP4RobotInverseDynamics dataset, distinct from the earlier Newton-generated calibration references. The publisher’s paper, section V-B, describes ROS position, velocity, and torque measurements filtered at 4 Hz, with acceleration obtained by acausal differentiation. The experiment uses published q, dq, ddq, tau_interp, and original timestamps without further filtering. Torque is filtered, interpolated measured generalized joint torque; the release does not establish reconstructed raw motor commands.

Training uses the three lowest published training seeds, 2, 3, and 4, totaling 8,771 joint-observation vectors. Verification uses all 16 published test recordings. The archive contains 13 training seeds despite the paper describing ten; this discrepancy is retained in the data provenance. Selection follows seed order, not fit quality. Actual irregular timestamps, approximately 17 ms apart, govern interpolation.

The Menagerie Panda geometry is sanitized to retain meshes, kinematics, topology, and joint ranges while removing authored inertias, actuator settings, losses, and initial configurations. Mesh density is zero and contacts are disabled. Each moving link starts at 1 kg, zero body-frame center of mass (COM), and diagonal COM inertia 0.01 kg·m². Joint losses, bias, and armature start at zero. These homogeneous placeholders contain no fitted or manufacturer dynamics.

The 98 model coefficients comprise seven sets of mass, three COM coordinates, and six symmetric inertia entries, plus seven each of viscous friction, Coulomb friction, torque bias, and reflected joint inertia. Mass is bounded to 0.05–10 kg and each COM coordinate to ±0.4 m; positive-inertia, triangle, and broad spatial-size constraints enforce physical link properties. Losses, bias, and armature also have fixed bounds. Full JSON tensors store mirrored entries redundantly. The training regressor has numerical rank 69 at relative cutoff 0.00001, leaving parameter combinations unresolved. The task therefore evaluates predictive effective dynamics on the recorded motions, without claiming unique recovery of individual link properties.

Every condition receives the same immutable inverse-dynamics regressor built through public Newton APIs from geometry and measured positions, velocities, and accelerations. Its 70 link columns use mass, first moments, and inertia about each link-frame origin; the parallel-axis theorem maps submitted COM tensors into that basis. Twenty-eight joint-parameter columns complete the matrix. Selecting 150 fixed time samples per training recording, with seven joint equations per sample, gives a 3,150 × 98 matrix. Newton basis calculations use float32, stored as float64. Manufacturer dynamics arrays, publisher estimators, fitted answers, and test responses are excluded. Agents write their own fitting programs with the same NumPy, SciPy, and CVXPY access.

Forward evaluation resets measured position and velocity at 12 fixed windows per recording, then applies measured torque for 50 steps of 2 ms. Torque is interpolated at step midpoints and reference states at step ends. Fitted bias is subtracted; friction and armature use native solver properties. There is no tracking controller. Training comprises 36 windows and 1,800 steps; test verification comprises 192 windows and 9,600 steps.

Forward root-mean-square errors (RMSEs) include every window step; torque errors use the fixed sampled rows. Torque normalization uses each joint’s sampled standard deviation within its recording, floored at 0.5 N·m. Physical validity and all expected finite samples are required; a later reset cannot erase a failed sample. The split, sampling, and gates were frozen before test responses were deserialized or scored. Study success requires a passing logged training candidate and fresh training plus complete test verification of the identical submitted JSON after agent exit.

Observations default to SensorTiledCamera, including when a viewer is attached, giving a consistent modality without an OpenGL context. Agents specify camera pose, resolution, world, and channels such as color, albedo, depth, normals, or shape IDs. Responses include an image and camera metadata; numeric artifacts and pixel picking support further inspection. Explicit backend="viewer" captures an attached ViewerGL on its owner thread, including wireframe mode. Unsupported settings fail explicitly. Recording writes bounded PNG sequences with simulation timestamps and a manifest.

Linux integration ran 97 tests with one expected attached-viewer skip. Actual official MCP SDK tests exercised Newton’s persistent cells, and actual upstream IPython MCP/kernel tests exercised persistent variables and rollout parity. Scientific regressions cover full COM and off-diagonal inertia mapping, native edits versus fresh models, reset, incomplete histories, and nonfinite failure parity.

On the frozen source, Windows Server 2025 validation ran 64 tests: 62 passed and two skipped, for unavailable CUDA and disabled attached ViewerGL capture. All 19 persistent-executor tests passed, including SDK calls, cross-cell Warp kernels, and dataclasses. The environment and artifact checksums accompany the exact CPU test log. This validates interface behavior, not Windows agent performance.

The camera selector below preserves actual historical sensor and viewer captures. Explicit camera pose, channel, dimensions, and scene revision make observations interpretable. Contact overlays depict collision geometry, not measured solver forces. These captures establish interface capabilities independently of the current timed study.

Historical observation channelSensor colorSensor albedoSensor color · second cameraSensor with contactsSensor forward depthSensor normalsSensor shape IDsViewerGL colorViewerGL wireframeDefault with ViewerGL attached

All 61 registered contexts completed: 58 passed and three missed a physical-quality gate. Every context passed the integrity and workflow audit. In the 54 primary runs, each method passed all nine existing cases and eight of nine real-data replicates. There were no observed context timeouts, reply-correlation failures, or physics-execution failures. The separate corrected-IPython sensitivity passed seven of seven cases.

On existing cases, Newton’s geometric mean paired time ratio against restart is 0.797: 20.3% less time, with a descriptive 95% bootstrap interval of 0.659–0.928. Input-plus-output tokens fall 31.4%, while uncached input plus output falls 18.3%. The three synthetic calibration cases show the largest task-specific reduction against restart: 36.6% in paired time and 49.4% in input-plus-output tokens. Against upstream IPython, however, Newton uses 10.6% more time and 7.8% more input-plus-output tokens across existing cases; the corresponding intervals include equality.

The real-data task does not reproduce Newton’s paired advantage. Across seven jointly successful replicates, Newton uses 4.5% more time and 15.0% more input-plus-output tokens than restart, and 22.5% more time and 48.0% more tokens than upstream IPython. The time intervals are 0.923–1.209 and 1.057–1.433, respectively. Uncached input plus output is also higher by 27.1% and 21.9%. These are small-sample comparisons on one recorded identification problem.

Table 3 retains all costs, including failures. Table 4 gives geometric means of within-case cost ratios for jointly eligible successes; each pair receives equal weight. Thus Newton’s lower total real-data time than restart can coexist with a paired ratio above one: failures differ between methods, and totals weight longer runs more heavily. Cached input changes token accounting substantially, so total input-plus-output is not a direct estimate of billed cost. Intervals resample whole pairs 20,000 times with seed 20260920. Exact sign and permutation tests, exclusions, and unrounded values are retained in the comparison JSON; these unadjusted descriptive intervals do not establish a population-wide ranking.

Physical checks include matching logged training and fresh verifiers where applicable; eligible success additionally requires valid budgets, process completion, and integrity. Paired ratios include only jointly successful eligible cases. The intervals resample registered pairs and are unadjusted descriptive comparisons; small samples and conditioning on success limit inference. Total costs are comparable only within the same task set. Corrected-IPython runs followed the primary block and cover fewer cases; their paired results are exploratory and may reflect run-order drift or agent variation. Bold, shaded values in individual tables identify the lowest eligible same-case costs; ties at displayed precision share emphasis. Sensitivity rows have no within-cohort competitors.

The three unsuccessful submissions narrowly exceed the normalized torque limit of 0.5 on joint 5 of one held-out recording: restart repeat 3 reaches 0.505271685, IPython repeat 3 reaches 0.500533412, and Newton repeat 5 reaches 0.502287762. All pass training, every other held-out gate, and all 9,600 finite forward steps. They remain failures under the unchanged protocol; they are neither runtime defects nor evidence of a large practical quality difference.

Search behavior differs materially. Real-data agents evaluate 286 Newton candidates, 125 restart candidates, and 204 IPython candidates. Twenty-six of 27 primary real-data contexts first pass the training checks at candidate 2; one Newton context does so at candidate 3. Agents may continue refining, and intermediate candidates have no held-out scores. These counts do not establish that later work was unnecessary or that an earlier stopping policy would retain final test success. Newton agents voluntarily rebuild five times across four real-data contexts while retaining one application process per trial.

The command audit retains two pre-execution agent syntax errors, one offline fitting-helper indexing error, ordinary source-lookup failures, and numerical warnings. These were distinct from protocol or physics-execution defects. No timed agent called observation or recording tools or viewed an image. The rendering evidence in Section 5 therefore establishes capability, not a measured visual-tuning advantage. Available timing records do not resolve transport time from model-service latency and reasoning; recorded response verbosity alone cannot explain the observed time differences.

Corrected IPython’s seven-case time ratio against upstream is 0.881, with interval 0.690–1.055; its input-plus-output ratio is 0.819. This later sensitivity block changes timeout and reply correlation together and uses fresh agents with different search paths. Its descriptive reductions cannot be attributed causally to the patch. The timeout characterization establishes a reproducible correctness fix; the primary timed runs did not encounter that defect.

For these tasks, full Python access is effective through either persistent interface. Newton supplies application-thread ownership, bounded scene queries, managed edits, and consistent camera observations; IPython remains a competitive execution interface when simulation integration is already provided. The evidence supports retaining Python as the main editing interface and using structured tools for lifecycle and observation. It does not support claiming that Newton MCP generally outperforms IPython.

Figure 6 compares costs within the same registered case. Each point divides Newton’s cost by the comparator’s cost; values below 1 favor Newton. The vertical axes use logarithmic scales, with equality at 1; horizontal labels identify cases. Only pairs passing physical checks and the study audit appear. Omitted pairs remain in the individual results and downloadable figure data.

Figure 7 reports held-out numerical quality for all 27 primary real-data submissions. For each metric, it takes the maximum across the pooled result and every recording, then divides by the frozen upper limit. The vertical scale is logarithmic; values at or below 1 satisfy that numerical gate. Circles mark eligible successful trials and crosses mark other submissions; gray crosses show the uniform initial model. Full success also requires finite, complete trajectories, valid parameters, training checks, and study eligibility. Missing or nonfinite measurements are disclosed in the figure data.

Figure 8 uses the prespecified trace selection: replicate 0, the sixth prediction window of training recording 2 and test recording 21, and all seven joints. Each curve is simulated joint position minus the interpolated physical measurement, so the measured reference is zero. Every window starts from measured position and velocity and then predicts 100 ms under the recorded generalized torque. Joint-specific vertical limits match between training and test panels. These traces illustrate a fixed window; the quality gates evaluate all registered windows and recordings.

The study uses one agent model, a small fixed task set, CPU simulation, and warm on-disk kernel caches. Elapsed times include model-service latency and agent reasoning in addition to simulation work. Nine real-data replicates are independent agent attempts, not independent robots. The corrected-IPython control changes reply correlation and timeout together. Shared simulation helpers remove integration work from the measured comparison. Scored trajectories advance on demand; the timing comparison does not measure interruption of a continuously running GUI, remote imaging latency, or integration of generic IPython with application thread ownership. Data separation uses explicit task restrictions and a logged-command audit, not an operating-system sandbox; the audit cannot prove the absence of unlogged access.

Physical identification is offline and limited to this publisher’s recordings. Filtering, acausal derivatives, unmodeled hardware, and simplified friction affect inferred effective parameters. A 100 ms prediction window does not establish long-horizon stability, contact accuracy, or robot deployment safety. Synthetic calibration and recorded human hand replay remain separate evidence categories.

Runtime authentication is not a Python sandbox. Checkpoints cannot generally promise bitwise replay; the scored real-data application rejects nonzero-frame restoration because measurement history is absent. Topology changes and new runtime arrays can require CUDA graph recapture. Windows CUDA, attached ViewerGL, and the Windows agent harness remain unverified.

The frozen Newton source is fe4fda0199ada. The evaluation guide documents data preparation, all execution conditions, and fresh verification. The frozen environment records Linux, Python 3.12.13, Warp 1.17.0, MuJoCo 3.12.0, MCP SDK 1.26.0, SciPy 1.17.1, CVXPY 1.9.3, and the four-CPU quota.

The integration guide shows how to embed a session around a model, solver, state buffers, viewer, and application step/reset callbacks. The bridge requires this instrumentation; it cannot attach to an arbitrary uninstrumented Python process. Keep pumping the owner-thread session while paused, enable trusted Python explicitly, and close the session in finally. Launch the compact bridge in the same environment:

uv run -m newton.mcp --connect session.json --profile code

The registration records exact order, source and input hashes, and per-context budgets; its SHA256 is 2a9af47f10b2dcc752e060db846e27a48aad203bee07ecc8290a676305108f22. The comparison design, physical-data protocol, and preprocessing and licensing record define interpretation independently of trial outcomes.

The audited comparison JSON, individual CSV, and group CSV retain costs and outcomes for every registered context. The completion ledger records summary hashes as each trial finishes; a separate cross-check compares them and the recorded costs against the audited originals. The logged-command audit records reviewed access and interface use. Public trial copies redact credentials and private connection descriptors; the export manifest distinguishes original from exported digests. Redacted copies are not byte-identical substitutes for original logs.

The measured references, immutable features, and sanitized geometry support fresh verification of each trial’s submitted parameters. The data manifest records digests, membership, and licensing. Separate fixed-window trace excerpts and their manifest reproduce Figures 6–8 together with the comparison JSON and plotting source. The excerpts contain only the registered illustration windows, not complete verifier trajectories. The reproduction notes explain the commands and evidence boundaries.

An independent recount reconciles all 61 raw trial records, 27 cost groups, and 117 metric-statistics blocks without importing the report builder. A public-export reproduction checks both archive memberships and digests, reproduces exact figure data, and freshly verifies the Newton real-data replicate-0 submission using only exported references and geometry. Its 1,800 training and 9,600 held-out samples pass, with numerical metrics matching the original verification. This is a reproduction of one submitted fit; all original trial verifiers remain available separately.

Measured data and derived references retain MERL’s 2024 attribution and CC-BY-SA-4.0 license; sanitized Menagerie assets retain Apache 2.0. Publisher estimator code is not copied into the Newton implementation. Historical downloads remain at their original paths; current reproducibility artifacts use data/v2/.