USD scene preservation for Newton RTX

USD (Universal Scene Description) combines referenced assets, overrides, materials, lights and physics into a composed scene. Newton converts that scene into a simulation model and portable visual shapes. Reconstructing an RTX scene from those shapes can lose material networks, subdivision settings or environment content. Issue #4051 requests visual and lighting parity with Omniverse Kit; extending Newton’s material parser addresses individual losses but also gives the physics engine more rendering semantics to maintain.

The proposal addresses two separate problems: preserve authored appearance for rendering, and eliminate duplicated physics interpretation. ovstage enables the first; adopting it does not automatically solve the second.

The intended division follows OVRTX’s external-stage architecture. OpenUSD still handles USD composition through population; Newton should interpret the physics needed for simulation, while the rendering backend interprets supported visual schemas.

Ordinary Newton application: call ModelBuilder.add_usd(...), finalize the model, then run a solver and viewer. Newton’s OpenUSD importer reads physics and extracts portable visuals. The existing generated-scene ViewerRTX path builds a new render scene from those visuals.

Application using ovnewton: populate an ovstage, import it with ovnewton.add_ovstage(...), finalize a Newton model, and attach a state binding. The application runs the solver and publishes its results to the same stage that OVRTX renders. These are alternative import routes; an application need not run both physics importers.

How an application uses ovnewton today

builder = newton.ModelBuilder()
result = ovnewton.add_ovstage(builder, stage, ordinal=ordinal)
model = builder.finalize(
    skip_validation_joints=result.has_orphan_joints,
)
binding = ovnewton.attach_ovstage(
    stage, model=model, ordinal=ordinal,
)

# After the solver advances state for a frame:
ordinal += 1
binding.update_to_ovstage(state, ordinal=ordinal)
stage.advance_write_floor(ordinal).wait()
renderer.step(
    render_products={product}, delta_time=dt, ordinal=ordinal,
)

Current API excerpt from the inspected ovnewton example (repository access required). Assumes a populated, published stage, renderer, render product, solver state and frame duration dt. Register solver-specific attributes before import. add_ovstage currently requires an otherwise empty builder; attach_ovstage(model=...) binds an existing model, while omitting model imports and finalizes one for convenience.

“Parsing ovstage” means querying runtime attributes and interpreting their physics meaning, rather than opening USD files again in Python. The inspected implementation separates this work into three layers:

For example, both paths decide how authored mass interacts with density-derived mass and missing inertia, how units and joint angles are converted, and how colliders and joints become Newton objects. A policy change can therefore require two fixes and two sets of coverage. The duplication is physics semantics across two implementations, even if each reads its own source efficiently. Newton importer, ovnewton interpretation, model construction.

Per-frame synchronization is separate from import. ovnewton publishes body poses, velocities and supported joint state, and can read supported stage state and drive targets back into Newton. It does not rebuild the physics model each frame. The current binding uses body/joint labels as prim paths and must be recreated after incompatible topology changes. Binding contract.

Preserve the composed render scene. Newton imports physics and publishes motion through a body-to-prim mapping, which associates simulated bodies with scene objects. Keep scene ownership and live USD prims outside Model; retain portable visuals for other viewers.

A source prim path on each visual shape is useful provenance but insufficient as the scene contract: it can omit lights, environment and material-binding context. Storing live UsdPrim handles also ties their validity to the source stage’s lifetime. Simulation bodies and render prims need not correspond one-to-one, especially after body collapse or instancing. An application-owned stage plus explicit bindings preserves that distinction.

ViewerRTX can remain independent of ovnewton. Put shared pose transport in Newton’s optional ovstage integration; ovnewton delegates pose writes to it. OVRTX renders; ViewerRTX presents the result.

The prototype implements borrowed-scene rendering and explicit CPU/CUDA pose bindings. Its ordinary physics import can still use Newton’s existing OpenUSD importer while population loads the render scene. This proves the rendering boundary, not unified import or a single startup traversal.

One physics interpretation, two source readers. Newton owns units, mass/inertia, collision geometry and joint construction. The OpenUSD reader and ovnewton’s ovstage reader supply source facts to that implementation. Moving ovnewton’s parser into Newton while retaining the old parser would preserve the duplication.

The reader contract must carry values, units, hierarchy, stable source identities and whether a value was authored, inherited/defaulted or blocked. A resolved number alone can lose information needed to choose a Newton default. Readers also need a common representation of instances and their transforms. ovstage does not need to acquire Newton-specific model-building policy.

Dependency direction: ovnewton depends on Newton; Newton’s optional RTX integration depends on ovstage/OVRTX. Newton does not depend on ovnewton. ovnewton retains its useful state/control synchronization and convenience API, delegating common pose writes to Newton’s transport. Sharing pose transport alone does not unify the physics importers.

Mapping contract: keep a binding object beside the model with prim paths, stable body identities and rest transforms. Resolve final body indices after construction, and remap bindings when models are cloned, bodies are collapsed or arrays are reordered. The prototype requires final indices explicitly; automatic remapping is proposed work. The application owns stage lifetime and publication.

Steps 2–3 proceed together. Inspected ovnewton pins Newton >=1.5,<1.6 and ovstage 0.2.0.378985; it cannot simply be added to Newton 1.7 development. That alignment is separate from the viewer-only runtime pair below. These are findings at the linked source revision, not claims about every later release.

Migrate the viewer before unifying the importers. Newton’s rtx extra allows ovrtx>=0.3; its generated-scene path still calls deprecated renderer scene APIs. A version bump alone leaves that migration unfinished. Current code, OVRTX architecture.

ovrtx.register_schema_paths()
renderer = ovrtx.Renderer(config=ovrtx.RendererConfig(sync_mode=True))
stage = ovstage.Stage("scene")

renderer.attach_ovstage(stage)
ovstage.population.open_usd(
    stage,
    scene_usd,
    ordinal=1,
    domains=ovstage.PopulationDomain.ALL,
)
stage.advance_write_floor(1).wait()

binding = OvstageBodyBinding(
    stage,
    model,
    ordinal=1,
    prim_paths=paths,
    body_indices=indices,
    body_local_transforms=offsets,
)
viewer = ViewerRTX(
    stage=stage,
    renderer=renderer,
    render_product=product,
)

# Finish rendering before the next write.
binding.write(state, ordinal=frame)
stage.advance_write_floor(frame).wait()
viewer.render(ordinal=frame)  # frame starts at 2

Excerpts assume an authored camera/RenderProduct and final-model mappings. Register additional physics schemas before population. See imports and cleanup and 0.5 API changes. The prototype window is a fixed-camera preview; full interaction still needs porting.

Both sides use OVRTX 0.5 with matched camera, lighting and authored settings. These are Isaac Lab asset-derived scenes, including a generated heightfield; complete reinforcement-learning (RL) tasks and parity with Omniverse Kit rendering remain untested. Select a scene and compare the overlay.

ANYmal on an Isaac Lab heightfield

Overlay comparisonOVRTX 0.5 · 640 × 480 pixels

Reconstructed Newton visuals

Preserved source in ovstage

Preserving the source addresses appearance discarded during import in #4051. Exact Kit matching still depends on backend features, exposure, color management and temporal settings.

13 research runs · RTX 4090 / Windows · 640 × 480 · ≥30 s warmup + 60 frames. Synthetic pose replay, not solver execution; mostly one process repetition. GPU figures are allocation snapshots. Prototype: 64 robots at 41.4 ms/frame, including 22.5 ms encoding/writes/publication, without importing PXR or ovnewton in the replay’s Python process. Population still uses OpenUSD internally.

Release the viewer upgrade after its functional checks. Qualify Isaac Lab separately: repeated timing with physics, peak CPU/GPU memory, cloning, multiple cameras and matched Kit captures. These establish the workload limits.