Improving the Newton MCP harness for simulation setup and tuning

Newton's live MCP hosts an unmodified Newton example script in a running Python session. A coding agent drives it with two tools: newton_execute runs Python cells against the live simulation (with helpers such as rollout(), checkpoint(), and contact reports), and newton_rebuild reloads the script after the agent edits it.

Each trial gives a coding agent a task in a sandboxed workspace with a fixed time budget. In a pair, the same agent (Claude Opus 5.5 in Claude Code, or GPT-6 Astra in Codex) solves the same task once with the MCP and once by editing and rerunning scripts. Both start from the same compiled kernels and the same prompt apart from the workflow paragraph, and every submission is verified in a fresh process by a hidden verifier. After each iteration the transcripts are mined for friction, the harness is changed, and the next iteration measures the new version. From iteration 12 on, trials run one at a time.

The tasks grew from tuning existing scenes to working from real-robot data and, most recently, starting from a nearly clean plate.

Iterations 0–8 tuned the harness on simulation tasks and real-robot calibration; 9–11 improved what agents see of the simulation; 12–14 added physical replays of real robot episodes under a sandboxed harness that runs one trial at a time; 15–16 start agents from a nearly clean plate. Most iterations ran one pair per task and agent, so a single iteration is noisy; Table 3 pools all iterations by task.

1. The MCP saves time, and the saving comes from the live session itself. Over 121 timed pairs agents with the MCP needed 0.87× the time and passed about as often (111 vs 115 of 122). Transcript analyses trace the saving to not paying startup, scene loading, and compilation for every experiment, and to inspecting the running simulation instead of re-running it. Specialized MCP helpers contributed little.
2. The task decides how much it saves. The MCP helped most where each experiment is cheap but has overhead: G1 tracking (0.58×), cube-toss calibration (0.76×), and the hard G1 clips (0.77×). It did not help where thinking or heavy simulation dominates: the grasp fix (1.00×), the scene from photos (1.04×), SDF grinding (1.06×), and the screwdriver replay (1.19×).
3. Tokens and cost depend on the agent. GPT-6 Astra turned the faster loop into fewer tokens (0.78× input tokens). Opus used the time for more experiments (1.18× input tokens, cost 1.02×).
4. Agents reach a solution sooner with the MCP but do not stop. From iteration 16 the harness verifies workspace snapshots, which measures when each run first passed. In iteration 17 the MCP runs passed first in 6 of 7 pairs (0.68× the time; pooled with iteration 16, 0.85× over 16 pairs), yet total time was 1.07×: agents kept polishing and stress-testing after they passed, more so with the live loop. Three of the three tray failures in iterations 16–17 were MCP runs that kept tuning a constraint they could not see until they crossed it.
5. Harness details matter as much as features. How tools are advertised and loaded, and how long their results are, moved input tokens from 1.30× (h0) to 0.74× (h3). Agents without a clock stopped after a fifth to a quarter of their budget until both prompts stated the start time (h7). Idle background workers that the prompt never mentioned cost about 2 minutes over the four MCP trials of iteration 15's first half until they started on demand (h16).
6. Agents want numbers and batches, not pictures. In iterations 0–14, newton_execute made up 88% of MCP calls and rollout() was the most used helper. Given a numeric score, agents rarely looked at renders, and image comparison, filmstrips, and an RTX or Blender backend went almost unused. Instead agents wrote their own copies of the verifier and their own multi-world batch harnesses in most recent trials, which is what the next MCP version targets. The MCP's batch helpers added in h17 went unused, but Newton's own BatchRollout, named in both prompts' docs line, became the planning engine of most Opus G1 controllers in both conditions.
7. Frontier agents can build scenes and controllers from scratch. Both agents rebuilt the fruit-bowl scene from photos and logs and wrote model-predictive controllers (GPU sampling MPC with 512 parallel robot copies, or iLQR) that track humanoid motions, including unseen clips, within 90 minutes. Without the MCP they passed the scene task in all 10 runs, with it in 7.
8. Verifiers need as much scrutiny as the harness. A G1 verifier changed the playback rate after construction, and the scene verifier measured the tray against a too-small outline, scored the low-friction control by a point that rises when a pear tips over in place, and scanned helper files the submission never imports. Each fix was applied to every trial of the task afterwards; re-verification changed 6 scene-task verdicts from fail to pass.
9. A live simulator is a good bug finder. Live editing and recapture exercise code paths that one-shot scripts do not; the study found 13 Newton bugs (Section 4).

Result

Five harness iterations and 156 valid trials. Ratios are geometric means over the six tasks of per-task medians.

• Expensive models benefit, mostly from the MCP. On one commit, interleaved per task, Opus and Astra with the MCP alone reached their first passing snapshot in 0.61× and 0.69× the time without it. The lite harness on top (card list and four rules) cut that further, to 0.53× and 0.61×, with total time 0.67× and 0.68× and Astra's cost 0.57×. Pooled over two replicates, they passed 12 of 12 per model in both arms.
• A mid-tier model with the harness is the cost-efficient choice. GPT-6 Sol with the lite harness and the MCP reached first passes in 0.71× the expensive models' time at 0.37× their cost to that pass. It passed 11 of 12, missing one hard-G1 run, while plain Sol passed 12 of 12 at higher cost.
• The cheapest models do not close the gap with a harness. GPT-6 Luna solves five of six tasks at 3–5% of the cost but 1.2–1.3× the time, and never the hard G1 clips. Its best harness, the lite one, is only slightly faster (0.92×) and cheaper (0.72×) than no harness. Claude Haiku 4.5 passes at most 2 of 6 in any arm: it reports checks it did not run, and instructions did not change that.
• Less structure worked better. The seven-step procedure with mandatory checklist, metric module and check script slowed every model that followed it. The cards themselves were read early and often by every model except Haiku. The cards that name a task's actual cause were decisive almost every time they were read: FIT-03, BLD-05, CON-02, CTL-02 and CON-01, 10–11 times each. Agents read cards as files; card() was called twice in 78 trials, and 13 of 15 automatic attachments had no visible effect.

Design

Can a cheaper model with a harness reach a verified solution faster and more cheaply than an expensive model working alone? Text2Sim reports that an older, cheaper model with a structured harness and distilled “Debug Cards” (symptom → likely causes → checks → repairs) beat a newer frontier model. This phase tests the same idea for Newton, with the live MCP as part of the harness.

Tasks. There are six 30-minute tasks: G1 tracking, the hard G1 clips (budget cut from 60 to 30 minutes, which 17 of 18 earlier passes met), grasp drift, cube toss, ABC arm identification, and SDF grinding. SDF grinding is held out: no card was mined from its transcripts. The other five are in-distribution for the cards, which is a caveat for their results. Costs use list prices per million tokens (input / cached input / output):

• Opus 5.5: $4 / $0.20 / $20
• Haiku 4.5: $1 / $0.10 / $5
• GPT-6 Astra: $10 / $1 / $50
• GPT-6 Sol: $2 / $0.20 / $10
• GPT-6 Luna: $0.10 / $0.01 / $0.50

The Debug Cards and how useful each was

Verdict. Keep the cards as a short, title-listed reference and drop the procedure. 8 of the 36 cards, each naming a task's actual cause, account for all 65 decisive uses among the 372 classified from transcripts. 15 others helped as routes and checks, and 13 were mostly unused or never opened. The cards pay off for Sol and the frontier models, not for Luna or Haiku. Agents read them as files from the title list; the MCP card search and attachments were almost unused. The Debug Cards page has the verdict, all 36 cards rendered, and the evidence for every use.

Harness versions

Results per model

What the agents used

Iteration log

Baseline B (2026-10-10). Opus and Astra without the MCP or cards passed all 12 trials (6 tasks × 2 models). The median trial took 8.1 minutes, the first passing snapshot came at a median of 4 minutes, and the median cost was $1.82. This is the bar for the cheap models.

Harness v6.1 (cards-v1): 6 of 12, at a fraction of the cost. Haiku and Luna with the MCP, the 35 cards and the procedure did as well as B on G1 tracking and grasp drift:

• first passes at 2–12 minutes;
• median cost $0.68 and $0.21 per trial, against B's $2.21 and $1.71;
• Luna passed grasp drift for $0.05.

• Haiku ignored the cards in 3 of 6 trials and wrote measurement code it never tested. On G1 tracking it believed a passing controller still failed. On SDF grinding it reported “material removal confirmed” while nothing reached the collision geometry, which is the problem card BLD-05 describes. It finished two failing trials with more than half the time left.
• Luna followed the procedure and read 1–12 cards per trial, but it used a fresh process for every experiment. It built new mechanisms (a balance controller, a different solver) before sweeping the existing parameters widely. On cube toss it tuned on all provided tosses down to the fourth digit and missed the held-out tosses by 0.16%. Opus solved the same task in 4 minutes by raising contact damping and validating on half the tosses.

Cheap models alone (C): Luna is already close to B. Without the MCP, the cards or the procedure:

• Luna passed 5 of 6 tasks for a median of $0.08 per trial, with first passes at a median of 4 minutes, as fast as B. On cube toss it found the decisive contact damping and friction in 4 minutes for $0.04, much as Opus did.
• Haiku passed 2 of 6 (G1 tracking and grasp drift).
• Neither model passed the hard G1 clips, which both expensive models solved.

Harness v6.2 (cards-v2): same pass rate as plain Luna, but slower. The changes from v6.1:

• The prompt lists the cards by title and asks for the matching cards before the first experiment.
• Metric code goes into one module, tested on a case with a known answer.
• When verification uses unseen cases, the agent tunes on part of the data and checks on the rest.
• Experiments run in-process; parameters are swept widely before any new mechanism.
• The agent does not finish while a criterion fails.
• CON-02 covers overdamped impacts, and a new card, FIT-05, covers held-out failures.

• about 2 minutes on the mandated checklist, metric module and check script;
• rebuilding with BatchRollout an evaluation the task already provided, including a time-step bug;
• a detour into torsional friction prompted by a solver warning and the card it attached.

v6.3: lite harness against the MCP alone. The lite harness has the card list, the live card helpers and four rules, with no mandatory files. Luna passed 5 of 6 with it, and with the MCP alone. Summed over the five tasks Luna can solve, the minutes to the first passing snapshot were:

• plain: 29
• lite harness: 34
• MCP alone: 51
• cards-v2: 54
• B: 29, at 15–30 times the cost

What the agents used (counts corrected). The trial logger counted only literal card paths. Agents often read several cards with a shell glob or brace pattern (cat cards/CTL-02*, cards/CON-{01,03}.md), so a quarter of all reads went unlogged, including most of Opus's. The counts in this section come from the transcripts, and the logger is fixed. Earlier versions of this page reported too few card reads, and wrongly said Opus never opened a card.

Plateau for Luna and Haiku. By the protocol's stopping rule (two harness iterations without a gain over the plain arm), the harness has plateaued for the two cheapest models. Luna is as good without it, and Haiku does not rise above 2 of 6 with or without it.

v6.4: GPT-6 Sol and the expensive models with the lite harness. All 24 trials passed.

• Sol (a fifth of Astra's price) passed all six tasks both plain and with the lite harness and the MCP, including the hard G1 clips that no cheaper arm had passed. With the harness, its first passes summed to 26 minutes against 30 without, and its cost up to the first pass to $2.85 against $4.66. On cube toss it read FIT-04 and FIT-05 and passed in 4 minutes, against 10 without.
• Opus and Astra with the lite harness and the MCP (E) passed 12 of 12. Against B, the first passing snapshot came sooner in 7 pairs, later in 1, and at the same 2-minute snapshot in 4. Summed over the tasks, first passes took 0.49× B's time for Opus and 0.74× for Astra, and total time 0.66× and 0.72×. Astra's cost fell to 0.63×. Both read cards in every trial: Opus 1–8 targeted cards, Astra 5–13 in bulk. The card holding the task's fix was read and acted on in all 24 E trials. (An earlier version of this page said Opus never opened a card; the logger had missed its glob reads.)

v6.5: one commit, interleaved, and replicates. B, B with the MCP alone, E, and Sol plain and with the lite harness ran side by side per task (48 trials, all valid), and Luna got a second replicate of plain and lite (12 trials). Every expensive trial passed again.

• The MCP alone sped up the first pass by 0.61× (Opus) and 0.69× (Astra), consistent with iterations 0–17.
• The cards and rules added 0.86× and 0.88× on top of the MCP.
• Sol with the harness failed one hard-G1 run ($6.16 spent on a balance controller), while plain Sol passed it.
• Luna's second replicate reversed its first-replicate ranking slightly in favour of the lite harness. That shows how noisy single trials are, and why the conclusions above use the pooled medians.

Live use of the simulator surfaced Newton bugs that one-shot scripts rarely hit. Re-recording a CUDA graph around a warm SolverMuJoCo corrupted memory; the fix is upstream as newton#4377.

Thirteen more issues came out of the trials and the task building (#4445–#4457). Among them: MJCF bodies attached without a joint collided with their own parent and jammed a gripper; MJCF floors became invisible next to visual meshes; the camera sensor skipped convex hulls and the back of cloth; heightfields rendered with holes; ViewerRTX failed with the current ovrtx release. Ten pull requests with regression tests (#4458–#4467) fix ten of them; the cloth and texture-transform fixes for SensorCamera are merged, and the ViewerRTX fix was superseded by upstream's OVRTX update.

Current MCP (h16). Two tools. newton_execute runs Python cells against the live simulation with helpers for rollouts, checkpoints, resets, contact and solver-parameter reports, health checks, observations, and background workers; a failed cell rolls the simulation back. newton_rebuild reloads the edited script, optionally with overrides of its constants, and keeps the previous scene if the rebuild fails. Newton utilities named in both conditions' prompts: a headless runner, solver and health reports, per-world edits by label (WorldView), a Newton↔MuJoCo state bridge, camera projection, and label lookup.

Next. The cheaper-model study in section 4. If cheaper models with the harness and the MCP match the expensive ones, the useful parts of the cards go upstream as Newton documentation, and the card hooks go into the MCP.

• Two models and one or two runs per cell in most iterations: single pairs vary as much as the MCP effect on some tasks. Iteration 16 runs three replicates per cell.
• Pass rates are near the ceiling on most tasks, so the comparison mostly measures time, tokens, and cost.
• Each iteration changes the harness, so iterations are not replicates of each other; the pooled ratio mixes harness versions and tasks.
• Verifier fixes during an iteration are applied to every trial of the task afterwards (re-verification), but a verifier can still be wrong in ways nobody noticed.
• One machine with one shared GPU slice; heavy simulation in both conditions can contend for it.

Newton branch eric-heiden/newton-live-mcp: the MCP host and helpers are in newton/_src/mcp; the loop harness, tasks, and verifiers in tools/mcp_evaluation/v4 (run_v4.py, run_seq.py).

Trial data: loop.json (every trial's metrics and the paired ratios) and one transcript archive per iteration, for example loop-transcripts-i16.zip (prompts, transcripts, submitted scripts, verifications, host logs).

The full development log keeps the per-iteration notes, figures (observations, renderers, the ABC station), the real-robot calibration references, the MCP usage statistics, and the table of every trial.

Cheaper-model study (v6). The data and code are in this directory:

• per-trial results, including card usage: data/v6/trials.json;
• the 36 cards with provenance and verification notes: data/v6/cards.json;
• the trace classification of every card use: data/v6/card_usefulness.json;
• the three harness texts: data/v6/harnesses.json;
• the pages are built by export_v6.py, build_cards.py and build_summary.py.