Solver Observables Architecture

1. What changes

The existing extended-attribute mechanism asks the model to allocate solver-produced data. A request such as model.request_state_attributes("body_parent_f") changes subsequent state allocations even though the selected solver determines whether and how that field can be computed.

With solver observables, the request goes to the producer: solver.observables(flags). Simulation state retains the quantities used for integration; contacts retain collision geometry and indexing. Optional diagnostic arrays have their own lifetime and a solver-specific extension point.

2. Allocation and runtime flow

Setup: request and allocate once

The application unions sensor requirements with any fields it needs directly. For contact-indexed fields, a completed collision pipeline first publishes its rigid and soft contact capacities on the model. These are storage limits, not the current number of active contacts.

Runtime: fill the existing arrays

Each step receives the input state, control, contacts, and optional observable container. It advances the simulation and fills the requested diagnostics. A consumer then reads the updated state and the diagnostics from that same step.

Reading the arrows: arrows indicate data or a request entering the next operation. Setup and runtime are separate figures so allocation is not mistaken for per-step work. Figure 4 shows how sensors consume the populated arrays.

3. Data contracts

Flags compose as a set of plain Enum members.

Standard and custom flags can be unioned without assigning disjoint bit ranges. Each member's string value names its array field. Integer enums, string-mixin enums, raw strings, and integers are rejected: equal values from independent extensions could otherwise collide in a set.

Every requested array declares its row frequency.

The container's ATTRIBUTE_FREQUENCIES maps field names to Model.AttributeFrequency members or custom string frequencies. Lookup follows inheritance. Missing metadata is rejected before allocation. Frequency means row indexing, not update rate. SUPPORTED_OBSERVABLE_FLAGS is the sole capability set; contact dependencies are derived from frequencies.

Allocation establishes ownership.

Each container belongs to the solver instance that allocated it. The solver checks its concrete type and ownership before using it. supported_observable_flags reports supported requests.

None means unrequested, not pending.

Every requested array exists when observables() returns. A valid zero-capacity request produces an empty array; it does not leave the field as None.

Selection controls requests without reallocating storage.

observables.select(flags) returns a same-type subset sharing selected arrays and gradients. Omitted fields are None only in the subset; the source is unchanged. Its read-only flags and is_requested(flag) identify requested fields, not fresh values. The existing flag in observables syntax is equivalent.

Contact arrays use resolved capacity.

contact_f has rigid_contact_max + soft_contact_max entries: rigid slots first, then soft-rigid slots. The soft segment begins at the rigid capacity, not the live rigid count. Each segment's live count, clamped to its capacity, identifies active slots; force availability remains solver-dependent. Before pipeline initialization, the model capacities are None.

Contact allocation freezes capacity; the first step binds storage.

Changing the model's capacities after contact-observable allocation is rejected. The first step requesting a contact-indexed field verifies the contacts' device and both capacities, then binds that exact Contacts instance for the source and all contact-indexed subsets. Subsequent steps and sensors must use it. Body-only subsets need no contact binding.

Allocate before graph capture.

The observable arrays are reusable in ordinary and conditional CUDA graphs. There is no deferred observable allocation. Other solver scratch storage may still require its usual warmup.

Contact indexing domains

Three domains preserve the current packed layout while allowing precise declarations for new fields. They describe solver-observable storage; they do not add builder custom-attribute allocation or new numerical force producers.

Recommendation: use the specific rigid or soft domain for new arrays. Retain the combined domain for existing packed fields. Using only two domains would require splitting or changing contact_f; using only the combined domain would hide meaningful differences in indexing.

4. Application usage

The snippets assume a finalized model, input and output states, a control buffer, and a time step dt in seconds. Allocate observables before entering the simulation loop.

Body-level diagnostics

from newton.solvers import SolverMuJoCo, SolverObservableFlags

solver = SolverMuJoCo(model)
observables = solver.observables(
    {
        SolverObservableFlags.BODY_QDD,
        SolverObservableFlags.BODY_PARENT_F,
    }
)

for _ in range(num_steps):
    solver.step(state_in, state_out, control, contacts, dt, observables=observables)

    acceleration = observables.body_qdd
    parent_wrench = observables.body_parent_f
    state_in, state_out = state_out, state_in

Contact-indexed diagnostics with Newton collision detection

import newton
from newton.solvers import SolverObservableFlags

pipeline = newton.CollisionPipeline(model)
solver = newton.solvers.SolverXPBD(model)
observables = solver.observables({SolverObservableFlags.CONTACT_F})
contacts = pipeline.contacts()

pipeline.collide(state_in, contacts)
solver.step(state_in, state_out, control, contacts, dt, observables=observables)

# Same ordering and capacity as `contacts`.
contact_force = observables.contact_f

Only contact-indexed requests need a completed pipeline. No Contacts argument is required by solver.observables(), and no contact-count readback is needed to size its arrays.

Native collision backends

solver = newton.solvers.SolverMuJoCo(model)
pipeline = newton.CollisionPipeline(
    model, rigid_contact_max=solver.get_max_contact_count(), soft_contact_max=0
)
observables = solver.observables({SolverObservableFlags.CONTACT_F})
contacts = pipeline.contacts()

# Native MuJoCo fills contact geometry and forces; no pipeline.collide() here.
solver.step(state_in, state_out, control, contacts, dt, observables=observables)

With native MuJoCo collision detection, the solver writes contact geometry and forces into the export buffer. Do not also call pipeline.collide() for this path. Construct native Kamino before the pipeline to seed its rigid-contact budget. Both backends reject an insufficient export capacity at observable allocation.

5. Custom solver extension

There are two distinct extension mechanisms: subclass the container and solver, but define custom flags in an independent enum. A populated Python enum is not subclassed to add members.

1. Define a flag and the derived container

from enum import Enum
import warp as wp

from newton import Model
from newton.solvers import SolverObservables


class ThermalObservableFlags(Enum):
    BODY_TEMPERATURE = "body_temperature"


class ThermalSolverObservables(SolverObservables):
    ATTRIBUTE_FREQUENCIES = {
        "body_temperature": Model.AttributeFrequency.BODY,
    }

    def __init__(self, flags=()):
        super().__init__(flags)
        self.body_temperature: wp.array[wp.float32] | None = None

2. Advertise support and extend allocation

from newton.solvers import SolverBase, SolverObservableFlags


class ThermalSolver(SolverBase):
    OBSERVABLES_TYPE = ThermalSolverObservables
    SUPPORTED_OBSERVABLE_FLAGS = frozenset(
        {
            SolverObservableFlags.BODY_QDD,
            ThermalObservableFlags.BODY_TEMPERATURE,
        }
    )

    def _allocate_observables(self, observables, *, requires_grad):
        super()._allocate_observables(observables, requires_grad=requires_grad)

        if observables.is_requested(ThermalObservableFlags.BODY_TEMPERATURE):
            observables.body_temperature = wp.zeros(
                self.model.body_count,
                dtype=wp.float32,
                device=self.model.device,
                requires_grad=requires_grad,
            )

super()._allocate_observables() allocates requested standard fields. The override allocates custom fields using dimensions known to the solver. In this illustrative extension, temperature has one value per body; the temperature model and its physical units must be specified by the concrete solver.

3. Fill requested arrays inside the step

The following method belongs to ThermalSolver. The integration, acceleration calculation, and compute_body_temperature kernel are solver-specific implementation points, not complete numerical implementations in this example.

    def step(
        self,
        state_in,
        state_out,
        control,
        contacts,
        dt,
        *,
        observables=None,
    ):
        self._validate_observables(observables, contacts)

        # ...advance simulation and populate standard observables...

        if observables is not None and observables.is_requested(ThermalObservableFlags.BODY_TEMPERATURE):
            wp.launch(
                compute_body_temperature,
                dim=self.model.body_count,
                inputs=[state_out.body_qd],
                outputs=[observables.body_temperature],
                device=self.model.device,
            )

4. Compose requests from both enums

solver = ThermalSolver(model)
requested = {
    SolverObservableFlags.BODY_QDD,
    ThermalObservableFlags.BODY_TEMPERATURE,
}

observables = solver.observables(requested)
solver.step(state_in, state_out, control, contacts, dt, observables=observables)

temperature = observables.body_temperature

Use is_requested() in allocation and stepping code, including for standard fields. It remains true for requested zero-length arrays. Simple derived containers inherit select() unchanged: custom fields are selected by the same enum values that name their arrays. Containers with nested observable containers should override select(), call super(), and select their children in the returned object. SolverCoupled.Observables does this for entry-local containers. Other custom metadata is shallow-copied.

Contact-indexed custom fields

Declare a contact frequency on the container, just as temperature declares BODY. A solver advertises the flag only in SUPPORTED_OBSERVABLE_FLAGS. The base solver derives pipeline initialization, capacity freezing, and storage binding from the frequency, even when the standard force array is not requested.

class ContactObservableFlags(Enum):
    CONTACT_PRESSURE = "contact_pressure"

class ContactObservables(SolverObservables):
    ATTRIBUTE_FREQUENCIES = {
        "contact_pressure": Model.AttributeFrequency.CONTACT_RIGID,
    }

    def __init__(self, flags=()):
        super().__init__(flags)
        self.contact_pressure = None

The allocator override creates contact_pressure with model.rigid_contact_max rows. Units, dtype, and computation remain the custom solver's responsibility. Standard frequency declarations are inherited automatically. No separate contact flag registry or field-name heuristic is needed.

Existing solver-specific entries

MuJoCo defines SolverMuJoCo.ObservableFlags.QFRC_ACTUATOR. Requesting it returns a SolverMuJoCo.Observables container whose qfrc_actuator array contains generalized actuator forces, with frequency JOINT_DOF. It can be combined with standard flags in the same set.

6. Sensor consumption

SensorIMU.solver_observable_flags requests BODY_QDD; SensorContact.solver_observable_flags requests CONTACT_F. The application unions these requirements and makes one allocation. The sensors do not request fields on the model.

The keyword solver_observables= makes that input role explicit at consumer boundaries. A viewer can read the same container; it does not need a second allocation.

Rigid forces only in the current contact sensor

SensorContact launches its aggregation over rigid_contact_max and reads rigid_contact_count, rigid shape endpoints, points, and normals. It does not read the soft segment. Allocating a packed contact_f therefore does not enable deformable-contact measurements. This follows the existing sensor aggregation implementation.

Soft-rigid sensing would need both a solver that exports those forces and an aggregation path for soft endpoints, including particles and potentially edges or faces. Soft self-contact uses separate storage. These are numerical and selection semantics to design explicitly, not behavior that a frequency declaration can supply.

A combined sensor loop

imu = newton.sensors.SensorIMU(model, sites="imu_*")
contact_sensor = newton.sensors.SensorContact(
    model, sensing_shapes="foot_*"
)

# This example uses Newton collision detection with MuJoCo Warp.
solver = newton.solvers.SolverMuJoCo(model, use_mujoco_contacts=False)
pipeline = newton.CollisionPipeline(
    model, rigid_contact_max=solver.get_max_contact_count(), soft_contact_max=0
)
contacts = pipeline.contacts()
flags = imu.solver_observable_flags | contact_sensor.solver_observable_flags
observables = solver.observables(flags)

for _ in range(num_steps):
    state_in.clear_forces()
    pipeline.collide(state_in, contacts)
    solver.step(
        state_in, state_out, control, contacts, dt, observables=observables
    )
    imu.update(state_out, solver_observables=observables)
    contact_sensor.update(state_out, contacts, solver_observables=observables)
    viewer.log_contacts(contacts, state_out, solver_observables=observables)
    state_in, state_out = state_out, state_in

The selected solver must support the union of all requested flags. MuJoCo's CPU backend does not expose CONTACT_F through this API. MuJoCo Warp must keep sensors enabled for RNE-derived body acceleration and parent-wrench diagnostics.

7. Selection and row frequencies

An articulation view groups rows by selected articulations. Its existing static layouts include bodies, joints, joint degrees of freedom, joint coordinates, and shapes. Previously, generic attribute lookup always asked the model for a field's frequency, so a solver-specific array could not be selected unless its name was also registered on the model.

The view now asks a SolverObservables source for get_attribute_frequency(name) and checks that the container belongs to the same model. This reuses the existing strided and indexed selection layouts. It does not duplicate the solver field schema on the model.

from newton.selection import ArticulationView

view = ArticulationView(model, "robot_*")
temperature = view.get_attribute("body_temperature", observables)

This example continues the thermal solver extension above. A custom string frequency can also be selected when the model declares its articulation ownership and a compatible layout. A frequency name alone does not establish row ownership. Arrays must be allocated before selection and remain stable while the view caches their layout.

Why raw contacts need a different selection policy

Contact rows can change after every collision pass, have variable counts, and connect two different articulations. There is no unique, stable rectangular articulation layout. The current view rejects all three contact frequencies with an explicit error.

Recommended next step: define endpoint-based filtering with an explicit choice of either endpoint, both endpoints, or signed force on a selected body. For rectangular articulation data, reduce contact forces to a BODY-frequency observable first. Automatic filtering and reduction are not implemented here. A future schema could reuse Model.AttributeSpec when row widths, reference domains, or compaction policies are needed; the present declaration contains only the shared frequency contract.

8. Substep scheduling

Allocation specifies which diagnostics can be produced; a selection specifies which are requested on one solver call. For example, an IMU can sample acceleration every substep while a contact sensor samples forces only on the last substep. This update rate is independent of AttributeFrequency, which describes row indexing.

After configuring the solver, collision pipeline, and sensors, allocate and select before the loop or graph capture. Here frame_dt is the frame duration in seconds, num_substeps is a positive integer, and substep_dt is each solver step's duration.

flags = imu.solver_observable_flags | contact_sensor.solver_observable_flags
observables = solver.observables(flags)
every_substep = observables.select(imu.solver_observable_flags)
substep_dt = frame_dt / num_substeps

for substep in range(num_substeps):
    is_last = substep == num_substeps - 1
    requested = observables if is_last else every_substep
    # Update collisions here when using external collision detection.
    solver.step(
        state_in, state_out, control, contacts, substep_dt,
        observables=requested,
    )
    state_in, state_out = state_out, state_in
    imu.update(state_in, solver_observables=every_substep)

contact_sensor.update(state_in, contacts, solver_observables=observables)

If all diagnostics are needed only at the end, pass observables=None on earlier substeps. An empty selection, observables.select(set()), also requests none. The full container still owns all allocated buffers.

Selection contract

• select(flags) creates a lightweight Python container of the same concrete type. It neither allocates nor copies observable arrays, including for custom fields.
• Selected fields reference the exact source arrays and gradients. Omitted fields are None in the subset. Do not temporarily clear and restore fields on the source.
• is_requested(flag) checks the subset's request. It does not test array length, completed computation, or freshness.
• Flags absent from the source are rejected. Selecting a selection can only narrow it; create another combination from the original container.
• Contact-indexed subsets share one storage binding, even when created before the first step. Coupled solvers propagate selections into their child containers.

Sampling and execution limits

Skipped arrays retain their previous values. A mixed-rate container is not a synchronized snapshot: update each sensor only after its required fields have been produced, using the matching state and contact geometry. Collision detection may overwrite or reorder contact rows, so old contact forces must not be consumed against new geometry. Last-substep forces are not frame averages; averaging or integration requires samples from the relevant substeps.

Create selections before graph capture and reuse them. A fixed Python substep schedule is captured with the graph; changing Python selections afterward does not change an existing graph. Device-driven choices require the desired selections to be captured in graph-control-flow branches.

Selection controls observable writes and optional diagnostic work. It does not remove calculations needed by the dynamics, other requested fields, or legacy extended attributes. MuJoCo currently retains its post-constraint RNE computation once enabled, so skipping those observables does not necessarily skip that internal stage on later steps.

9. Migration from extended attributes

The existing paths are deprecated in Newton 1.7 and remain available during migration. New code should request diagnostics on the solver and pass the resulting container explicitly.

Before: model-driven allocation

model.request_state_attributes("body_parent_f")
model.request_contact_attributes("force")

state_in, state_out = model.state(), model.state()
contacts = model.contacts()

solver.step(state_in, state_out, control, contacts, dt)
solver.update_contacts(contacts, state_out)

parent_wrench = state_out.body_parent_f
contact_force = contacts.force

After: solver-driven allocation

# CollisionPipeline has already published compatible model capacities.
observables = solver.observables(
    {
        SolverObservableFlags.BODY_PARENT_F,
        SolverObservableFlags.CONTACT_F,
    }
)

solver.step(state_in, state_out, control, contacts, dt, observables=observables)

parent_wrench = observables.body_parent_f
contact_force = observables.contact_f

During migration, sensors can opt into their automatic request behavior with request_state_attributes=True or request_contact_attributes=True. These options are deprecated in Newton 1.7 and each emits one sensor-specific warning at the caller's location. Both default to False, which neither warns nor requests legacy fields. New sensor code should use the flag sets and explicit input shown above.

10. Benefits and limitations

• Ownership follows computation. Capability checks and allocation are attached to the solver that produces the data.
• Allocation follows demand. Unrequested diagnostic arrays stay absent. This can avoid storage and computation, but the report makes no measured speedup claim.
• Extensions remain local. A custom solver can add flags and arrays without editing the global state schema or reserving flag bits.
• Consumers compose requirements. Applications can union sensor requirements and share one container across sensors, viewers, and logging code.
• Row semantics are declared once. Custom fields participate in dependency checks and supported articulation layouts through the same frequency metadata.
• Sampling does not change storage. Reusable selections schedule different fields across substeps while sensors retain access to the same buffers.

The tradeoff is explicit lifecycle management: applications must keep the container with its owning solver and keep contact-indexed results with the bound storage. A replacement pipeline with different capacity requires rebuilding the relevant allocation setup. The API does not permit a caller to silently resize contact buffers in the middle of a captured loop.

Availability and physical interpretation remain solver-dependent. Kamino reports discrete step-average acceleration, including impact impulse divided by the step duration. XPBD's reaction wrenches are approximate applied constraint reactions. The coupled solver wrapper exposes a body observable only when all body-owning entries support it; contact observables require an explicit index-remapping contract and are not exposed by that wrapper.

Custom attributes on models, states, controls, and contacts remain supported for authored or persistent simulation data. This proposal only migrates built-in solver-produced diagnostics. Additional observable families remain possible, but do not become supported merely because their fields can be declared.