Newton design proposal · draft for review

Cameras in the Model

Declare cameras in ModelBuilder, import them from USD and MJCF, and render them with SensorCamera without hand-maintained pose, ray and world bookkeeping.

Status Draft, prototyped Tracking issue newton#4899 Date 2026-10-22 Updated 2026-10-23 Base main @ c81f4f332 Builds on newton#3276, newton#3627 and an earlier prototype PR

Summary

• Cameras become a Model entity, like bodies, shapes and tendons. ModelBuilder.add_camera(body, *, xform, ...) declares one; add_usd and add_mjcf create them from UsdGeom.Camera prims and <camera> elements. add_builder and replicate copy them into every world.
• The Model stores what the asset describes; the sensor stores what the render needs. The camera's frame, body, world, intrinsics and clip range go in model.camera_*. Resolution, ray bundles and output buffers stay in SensorCamera. This is the same split USD makes between UsdGeom.Camera and UsdRender.Product, and it answers the objection that rays depend on resolution and therefore cannot be model data.
• SensorCamera gains a cameras= selector that uses the same label matching as SensorIMU(sites=...). A bound sensor works out each view's camera, world and pose from the model and the state, so update(state, color_image=...) needs no transforms, rays or world_indices. The current explicit API is unchanged.
• Camera models are pluggable. A camera model is a ray function from pixel to ray. Built-in pinhole rays are either cached (RayMode.PRECOMPUTED, fast) or evaluated in the kernel from per-camera intrinsic arrays (RayMode.ANALYTIC, for per-environment intrinsics randomization), like mujoco_warp's use_precomputed_rays. Users bring their own model as data (CameraLens.CUSTOM plus set_camera_rays) or as code (an experimental @wp.func ray function whose parameters can be CAMERA custom attributes).
• A prototype exercises the whole design (section 14). Model cameras, USD and MJCF import, all four ray sources, a site-mounted lidar and a thermal camera work end to end, and the existing camera, builder and replication tests still pass. The main addition it motivated is per-hit output (face index and barycentrics), which lets users build their own imaging modalities on top of the renderer.
• Lidar needs no new sensor class. A SensorCamera bound to sites, with a custom ray pattern and depth output, is a range scanner. The proposal adds what is missing for that today: site mounting, per-view min and max range, and graph-safe pattern updates.
• No generic add_sensor(), and no cameras-as-shapes or cameras-as-sites. Sites stay the way to declare frame-only sensors (IMU, frame transform, and lidar through SensorCamera(sites=...)). A sensor kind gets its own Model entity only when assets carry description data that has to survive import and replication. Today that is only cameras.

1. Problem

SensorCamera is now Newton's camera sensor (SensorTiledCamera was deprecated in #3913). It is a good renderer, but Newton has no concept of a camera. Every caller passes in three things on every update() call (sensor_camera.py:900):

camera.update(
    state,
    camera_transforms,      # (view_count,) world-space poses, computed by the caller every frame
    camera_rays,            # (H, W, S, 2) one lens + resolution, shared by every view
    world_indices=...,      # (view_count,) which world each view renders
    color_image=color,
)

Everything that describes the camera itself (where it is mounted, which world it belongs to, its lens, its clip range) therefore lives in user code. The consequences show up in Newton's own example and, more seriously, in Isaac Lab:

• Pose bookkeeping. A body-mounted camera needs a hand-written kernel that composes body_q with a mount offset (example_sensor_camera.py:94 fill_body_camera_transforms). Isaac Lab runs a whole parallel site system for this (NewtonSiteFrameView: its own site_body/site_local arrays, body-ancestor resolution, per-world expansion and pose kernel), even though Newton's own add_site is what IMU, frame transformer and ray caster use.
• View-to-world mapping. world_indices and the view order are implicit conventions that callers have to rebuild. This was the main confusion in the #3276 review, and Isaac Lab has had several ordering bugs in this area (IsaacLab #8545, #7688).
• No import. add_usd and add_mjcf drop camera prims and <camera> elements. Isaac Lab reads intrinsics back from the USD stage at init and builds its own K matrix and ray kernel, so the Newton Warp path still needs a stage.
• No per-environment intrinsics. A single shared ray bundle means intrinsics randomization across environments is impossible. Isaac Lab raises "Newton Warp requires identical camera intrinsics across environments: its native ray field is shared across worlds."
• Clip range is global. The far clip is RenderConfig.max_distance, which is baked into the kernel with wp.static (render.py:182). Isaac Lab overwrites the shared default config before every camera's render, and the near clip is never applied.
• No subset rendering. Isaac Lab renders every world on every update, even when only some environments are due or are being reset.

2. Requirements

These requirements come from reading Isaac Lab main (8c9b995) and its issue tracker, from the prior Newton PRs, and from how mujoco_warp handles cameras. "Source" names the evidence.

3. Design principles

P1. The Model describes and the sensor renders.

Anything an asset author writes down (mount frame, parent body, world, focal length, aperture, clip range) goes in the Model. Anything that depends on how you choose to render (resolution, sample count, ray buffers, output images, lighting) goes in SensorCamera. USD makes the same cut: UsdGeom.Camera has no resolution, and resolution lives on UsdRender.Product, which points at a camera. This settles a recurring design question: rays never go in the Model, because they depend on resolution, but cameras do.

P2. A camera is a per-world instance.

Each camera row has a world index, like bodies and shapes, so heterogeneous worlds work by construction. A camera added in the global scope (world -1) is a single declaration that renders every world. This is the "observer camera" case that the example and #3627 both had to special-case.

P3. Reuse existing Newton vocabulary.

add_camera(body, *, xform, label, custom_attributes) follows add_site. SensorCamera(model, cameras=...) follows SensorIMU(model, sites=...) and label matching. Model arrays use the camera_* prefix and the existing AttributeSpec registry, so replicate, add_builder, custom attributes and model views all handle them.

P4. The explicit path stays.

SensorCamera(model) with no cameras= behaves exactly as it does today. A bound sensor can still be given explicit camera_transforms or camera_rays as per-call overrides, which keeps custom lenses and externally driven poses simple.

4. Architecture

What changes compared to today

5. Builder and Model API

5.1 ModelBuilder.add_camera

The signature mirrors add_site (a positional body, then keyword-only xform, label and custom_attributes). The intrinsics parameters reuse the names of the existing SensorCamera.compute_camera_rays_pinhole helper, so the same words mean the same thing everywhere.

def add_camera(
    self,
    body: int,
    *,
    xform: Transform | None = None,
    lens: CameraLens = CameraLens.PINHOLE,
    fov: float | None = None,
    focal_length: float | None = None,
    horizontal_aperture: float | None = None,
    vertical_aperture: float | None = None,
    horizontal_aperture_offset: float = 0.0,
    vertical_aperture_offset: float = 0.0,
    clipping_range: tuple[float, float] = (0.01, 1.0e6),
    resolution: tuple[int, int] | None = None,
    label: str | None = None,
    custom_attributes: dict[str, Any] | None = None,
) -> int:
    """Add a pinhole camera rigidly attached to a body or to the world.

    The camera looks along its local -Z axis with +Y up and +X right
    (USD, OpenGL and MuJoCo convention).

    Args:
        body: Parent body index, or -1 for a camera fixed in the world
            frame of the current world context.
        xform: Camera frame relative to the parent body [m, rad]. For
            ``body == -1`` it is the world-frame pose. Defaults to identity.
        lens: Lens model. ``PINHOLE`` uses the intrinsics below.
            ``CUSTOM`` means the rays are supplied to the sensor (custom
            rays or a ray function) and the intrinsics are ignored.
        fov: Vertical field of view [rad]. Mutually exclusive with
            ``focal_length``. The horizontal extent then follows the image
            aspect ratio (square pixels), which matches MJCF ``fovy``.
        focal_length: Lens focal length [m]. Only ratios with the aperture
            affect the projection.
        horizontal_aperture: Sensor width [m]. If ``None`` or 0, it is
            derived from ``vertical_aperture`` and the image aspect ratio.
        vertical_aperture: Sensor height [m]. If ``None`` or 0, it is
            derived from ``horizontal_aperture`` and the image aspect ratio.
        horizontal_aperture_offset: Principal-point offset [m].
        vertical_aperture_offset: Principal-point offset [m].
        clipping_range: (near, far) distances [m]. Hits outside the range
            are reported as misses.
        resolution: Nominal (width, height) [px] authored by the asset.
            Only a default for sensors; it does not allocate anything.
        label: Unique label, e.g. a USD prim path. Defaults to
            ``camera_{index}``.
        custom_attributes: Values for CAMERA-frequency custom attributes.

    Returns:
        The camera index.
    """

If no intrinsics are given, the camera gets fov = radians(45), the MuJoCo default. newton.CameraLens starts with PINHOLE and CUSTOM. Built-in distortion models (OpenCV, fisheye, F-theta, Kannala-Brandt) are added in phase 2; until then a distorted lens is a CUSTOM camera with rays from the existing helpers (section 7.2).

5.2 New Model fields

A new Model.AttributeFrequency.CAMERA lets importers and users attach per-camera data such as semantic tags, ISP settings, randomization ranges or the parameters of a user camera model (7.3), using the same mechanism as every other entity. There is deliberately no per-camera ray array, sensor object or enabled flag in the Model: those are render state (P1), and enabling is a sensor concern (view_mask).

6. SensorCamera API

SensorCamera is the established camera sensor, so this extends it rather than adding a new class. The constructor gains a frame source (model cameras or sites) and a choice of ray source. Everything new is keyword-only and defaults to today's behavior.

class SensorCamera:
    class RayMode(IntEnum):
        PRECOMPUTED = 0   # cache built-in lens rays, one bundle per distinct lens (default)
        ANALYTIC = 1      # evaluate built-in lens rays in the kernel from model intrinsics

    @wp.struct
    class RayQuery:       # input to a user ray function (section 7.3), experimental
        view: int
        camera: int       # model camera index, or site index for site-bound sensors
        world: int
        x: int
        y: int
        sample: int
        width: int
        height: int
        sample_count: int

    def __init__(
        self,
        model: Model,
        *,
        cameras: str | list[str] | re.Pattern[str] | list[int] | None = None,
        sites: str | list[str] | re.Pattern[str] | list[int] | None = None,
        width: int | None = None,
        height: int | None = None,
        sample_count: int = 1,
        ray_mode: RayMode = RayMode.PRECOMPUTED,
        ray_function: wp.Function | None = None,   # experimental
        ray_params: wp.array2d[wp.float32] | None = None,  # experimental
        clipping_range: tuple[float, float] | None = None,
        default_clear_data: ClearData | None = None,
        default_render_config: RenderConfig | None = None,
        load_textures: bool = True,
    ): ...

    # Available when constructed with ``cameras`` or ``sites``:
    view_count: int                  # number of rendered views
    width: int
    height: int
    view_camera: wp.array[wp.int32]  # model camera (or site) index per view
    view_world: wp.array[wp.int32]   # world index per view

    HIT_TRIANGLE_MESH, HIT_PARTICLES, HIT_NONE  # shape_index_image sentinels (section 7.5)

    def compute_view_transforms(self, state, out=None) -> wp.array[wp.transformf]: ...
    def set_camera_rays(self, rays, *, cameras=None) -> None: ...  # custom rays (section 7.2)
    def sync_camera_rays(self) -> None: ...   # PRECOMPUTED: rebuild after intrinsics change

    def update(
        self,
        state: State,
        camera_transforms: wp.array[wp.transformf] | None = None,  # now optional
        camera_rays: wp.array4d[wp.vec3f] | None = None,           # now optional
        *,
        color_image=None, depth_image=None, forward_depth_image=None,
        shape_index_image=None, normal_image=None, albedo_image=None,
        hdr_color_image=None,
        view_mask: wp.array[wp.bool] | None = None,                 # new
        face_index_image: wp.array3d[wp.int32] | None = None,       # new, per-hit data
        barycentric_image: wp.array3d[wp.vec2f] | None = None,      # new, per-hit data
        world_indices: wp.array[wp.int32] | None = None,
        clear_data=None, render_config=None, sync_deformables=True,
    ) -> None: ...

6.1 Semantics

• Frame source. cameras selects model cameras by label matching on model.camera_label. Their frames and optics come from the Model. sites instead selects sites, which provide only a frame. A site-bound sensor therefore needs its rays from set_camera_rays, a ray_function or the per-call camera_rays, and its clip range from clipping_range (default (0, inf)). This is how ray sensors such as lidar are built (section 12). Pass at most one of the two; with neither, the sensor is unbound and behaves as today. The selection is fixed at construction, like SensorIMU(sites=...), which is the static half of R6.
• Views. Each selected camera or site with world w ≥ 0 produces one view that renders world w. Each selected global one produces one view per world. Views are ordered by index, then world. For a pattern such as "*/wrist_cam" on a replicated scene, view i is environment i, so the output is (num_envs, H, W) with no reshaping. view_camera and view_world make the mapping explicit (R1).
• Resolution. width and height default to camera_resolution when all selected cameras agree, and otherwise are required. One sensor has one resolution, because outputs are dense (view_count, H, W) arrays. Different resolutions use different sensors (R7).
• Rays. Each view gets its rays from exactly one source: a built-in lens (PRECOMPUTED or ANALYTIC), a custom bundle, or a ray function. Section 7 covers all four.
• Poses. Each update() launches a small kernel, view_pose = body_q[body] * frame_transform (using the camera or site mount, or just the transform for fixed frames), into an internal buffer with a stable address, then renders. Both launches are graph capturable. An explicit camera_transforms argument overrides the computed poses for that call.
• Masking. view_mask[v] == False skips view v and leaves its outputs untouched (the existing WorldRenderFlag.DISABLE_PRESERVE path). This is the dynamic half of R6: update periods, partial resets, retired cameras.
• Clip range. A bound sensor reads the near and far distances per view at run time instead of the wp.static max_distance, which removes the kernel variant per far value and adds near clipping (R8). The values come from camera_clipping_range for cameras, or from clipping_range for sites; passing clipping_range also overrides the camera values.
• Shared scene state. All SensorCamera instances for the same model and load_textures setting share one internal RenderContext (BVH bindings, textures, lights). Several bound sensors (one per resolution, camera group or lidar) then cost what the single shared Isaac Lab sensor costs today (R7). See open question 5 on lights.
• Per-hit outputs. face_index_image and barycentric_image report which triangle, mesh face or particle each pixel hit and where. Together with shape_index_image they let user kernels compute new imaging channels (section 7.5). allocate_image_face_index() and allocate_image_barycentric() allocate them.
• Unbound sensors. With neither cameras nor sites, camera_transforms and camera_rays are required, as they are today, and the view_* attributes raise.

6.2 Minimal example

import warp as wp
import newton
from newton.sensors import SensorCamera

robot = newton.ModelBuilder()
robot.add_usd("franka.usda")  # UsdGeom.Camera prims in the asset are imported as well
hand = robot.body_label.index("/franka/panda_hand")
robot.add_camera(hand, xform=wp.transform((0.0, 0.0, 0.05), wp.quat_identity()),
                 fov=1.0, clipping_range=(0.02, 3.0), label="wrist_cam")

builder = newton.ModelBuilder()
builder.add_ground_plane()
builder.add_camera(-1, xform=overview_pose, fov=0.8, label="overview")  # global: every world
builder.replicate(robot, copy_count=1024, label_prefixes=[f"env_{i}" for i in range(1024)])
model = builder.finalize()

wrist = SensorCamera(model, cameras="env_*/wrist_cam", width=128, height=128)
overview = SensorCamera(model, cameras="overview", width=640, height=480)
wrist_rgb = wrist.allocate_image_color()   # (1024, 128, 128); view_count/size default from the sensor
overview_rgb = overview.allocate_image_color()

for _ in range(steps):
    solver.step(state_0, state_1, control, contacts, dt)
    model.bvh_refit_shapes(state_1)
    wrist.update(state_1, color_image=wrist_rgb)
    overview.update(state_1, color_image=overview_rgb)

This removes the two transform kernels, the two hand-built ray bundles and the world_indices array that example_sensor_camera.py needs today.

7. Camera models and rays

A camera model is a mapping from (pixel, sample) to a ray in the sensor frame: an origin and a direction. Everything in this section is about who defines that mapping and whether the sensor caches its output. Rays never live in the Model, because they depend on resolution. Lens parameters may live in the Model, either as the built-in intrinsic arrays or as CAMERA-frequency custom attributes.

Ray contract, shared by every source below:

• The origin and direction are expressed in the frame of the camera or site. Origins can vary per ray, which supports non-central lenses and multi-emitter scanners.
• Directions must be unit length. The kernel reports depth_image as the hit parameter along the ray without normalizing it, so it is a metric range only for unit directions. The built-in helpers already normalize.
• A zero direction marks an unused slot. The kernel treats it as a miss without tracing it, which is what the fisheye helpers already rely on outside the lens's field of view.
• The -Z forward convention matters only for built-in lenses and for forward_depth_image. Custom rays can point anywhere, including behind the frame.

7.1 Built-in lenses: precomputed vs analytic

Both modes read the same Model arrays; they differ only in whether the sensor caches the derived rays. PRECOMPUTED groups the selected cameras by identical intrinsics and stores one bundle per group. After writing model.camera_*, sync_camera_rays() rebuilds the bundles in place at a stable address, and raises if the grouping of equal lenses changes. ANALYTIC stores nothing and evaluates the pinhole projection per pixel from camera_*[view_camera[v]], so per-environment intrinsics cost no memory (a per-env bundle would be 4096 × 128² × 2 × 12 B ≈ 1.6 GB).

Kernel change. The render kernel currently reads camera_rays[py, px, s, k] from a single bundle (render.py:589, render.py:690). It gains a wp.static switch between a bundle indexed by view_ray_bundle[view] and an inline ray generator. newton#3276 (camera_indices[:, 1]) and mujoco_warp (ray_offset) both used a per-view bundle index, so that half is already proven.

# Per-env focal-length randomization, captured in the same CUDA graph as the render.
wrist = SensorCamera(model, cameras="env_*/wrist_cam", width=128, height=128,
                     ray_mode=SensorCamera.RayMode.ANALYTIC)

@wp.kernel
def randomize_focal(seed: int, nominal: float, focal: wp.array[float]):
    c = wp.tid()
    focal[c] = nominal * (1.0 + 0.1 * (wp.randf(wp.rand_init(seed, c)) - 0.5))

wp.launch(randomize_focal, dim=model.camera_count, inputs=[seed, 0.024, model.camera_focal_length])
wrist.update(state, color_image=rgb)   # each env renders with its own focal length

7.2 Custom rays as data

A camera declared with lens=CameraLens.CUSTOM has no analytic model, and its intrinsic fields are ignored. Its rays come from the sensor:

sensor.set_camera_rays(rays, cameras=None)
#   rays:    (H, W, S, 2) vec3f, shared by the matched cameras, or
#            (N, H, W, S, 2) vec3f, one bundle per matched camera (e.g. per-env calibrations)
#   cameras: label pattern selecting a subset of this sensor's cameras; None = all of them

• The sensor copies the rays into its own bundle storage. The first call allocates; later calls with the same shapes are device copies at a stable address, so a scan pattern can be updated every step inside a CUDA graph.
• H, W and S must match the sensor's height, width and sample_count.
• update() raises if a selected CUSTOM camera has no rays. ANALYTIC mode raises for CUSTOM cameras rather than guessing a model.
• PINHOLE and CUSTOM cameras can share a sensor and a launch, because each view already carries a bundle index.
• Site-bound sensors use the same method, since sites never have built-in optics.
• The per-call camera_rays= override in update() stays as an escape hatch and remains the only option for unbound sensors.

proto.add_camera(hand, xform=mount, lens=newton.CameraLens.CUSTOM,
                 resolution=(640, 480), label="wrist_cam")
...
wrist = SensorCamera(model, cameras="env_*/wrist_cam", width=640, height=480)
rays = my_calibrated_lens_rays(640, 480, device=model.device)  # (480, 640, 1, 2), any kernel
wrist.set_camera_rays(rays)
wrist.update(state, color_image=rgb)

7.3 Custom camera models as code experimental

Data alone cannot express a custom model whose parameters change per environment without one bundle per environment, or a scan pattern that changes every step without re-uploading rays. For those cases a sensor accepts a user @wp.func that the render kernel inlines in place of the built-in pinhole. The render kernel is already specialized per configuration with wp.static, so the function is captured the same way and compiled once per distinct function.

• Signature: (params, query: SensorCamera.RayQuery) -> (origin, direction), following the ray contract above.
• Parameters: params is a wp.array2d[float32] table passed as ray_params, typically one row per model camera (indexed by query.camera) or per world. It lives on the device, so values and step counters can be written inside a captured graph. The prototype settled on this fixed type because it keeps the render kernel's signature stable; arbitrary user structs remain an open question.
• Where the parameters come from: declare them as CAMERA-frequency custom attributes. They are then authored in USD (newton:<namespace>:<name>) or MJCF, parsed by the importers, replicated per world, and can be randomized, with no Newton change. A one-line kernel gathers them into the parameter table.
• Scope: one ray function per sensor, applied to all of its views (CUSTOM cameras or sites). To cache a function's output instead, evaluate it once into a bundle with your own kernel and use 7.2.
• Built-in lenses: the phase-2 lens types (OpenCV, fisheye, F-theta, Kannala-Brandt) can be implemented as built-in ray functions on the same mechanism.

# A barrel-distorted lens whose field of view and distortion are per-camera custom attributes.
for name in ("lens_fov", "lens_k1"):
    builder.add_custom_attribute(
        ModelBuilder.CustomAttribute(
            name=name, namespace="thermal", frequency=Model.AttributeFrequency.CAMERA, dtype=wp.float32
        )
    )  # USD: newton:thermal:lens_fov / newton:thermal:lens_k1 on the camera prim
builder.add_camera(-1, xform=pose, lens=newton.CameraLens.CUSTOM, resolution=(192, 144), label="thermal_cam",
                   custom_attributes={"thermal:lens_fov": 0.73, "thermal:lens_k1": 0.12})

@wp.func
def barrel_lens(params: wp.array2d[wp.float32], q: SensorCamera.RayQuery):
    half = wp.tan(0.5 * params[q.camera, 0])
    k1 = params[q.camera, 1]
    x = ((float(q.x) + 0.5) / float(q.width) - 0.5) * 2.0 * half * float(q.width) / float(q.height)
    y = (0.5 - (float(q.y) + 0.5) / float(q.height)) * 2.0 * half
    scale = 1.0 + k1 * (x * x + y * y) / (half * half)
    return wp.vec3f(0.0), wp.normalize(wp.vec3f(x * scale, y * scale, -1.0))

@wp.kernel
def gather(fov: wp.array[float], k1: wp.array[float], out: wp.array2d[float]):
    c = wp.tid()
    out[c, 0] = fov[c]
    out[c, 1] = k1[c]

params = wp.zeros((model.camera_count, 2), dtype=wp.float32)
wp.launch(gather, model.camera_count, inputs=[model.thermal.lens_fov, model.thermal.lens_k1], outputs=[params])
thermal = SensorCamera(model, cameras="thermal_cam", ray_function=barrel_lens, ray_params=params)
# Randomize model.thermal.lens_k1 per camera and gather again; the next update() uses the new values.

7.4 Comparison with mujoco_warp

mujoco_warp's camera model is what MJCF can describe: cam_projection (perspective or orthographic), cam_fovy, cam_sensorsize and cam_intrinsic, with no lens type and no distortion. With use_precomputed_rays=True, create_render_context fills flat rc.ray/rc.ray_offset arrays from the unbatched intrinsics, one block per active camera, shared by all worlds. With False, the kernel calls the same compute_ray with per-world cam_fovy/cam_intrinsic, which is the documented path for intrinsics randomization. Both paths are the same analytic model, so "precomputed" is a cache, not an extension point. Custom lenses are only possible by overwriting rc.ray in place, with an implicit layout and per-ray origins honored only if the model also has an orthographic camera. Sources 1 and 2 above match mujoco_warp; sources 3 and 4 are where this design goes further.

7.5 Custom imaging from per-hit outputs

Some sensors need a different quantity per pixel rather than different rays: temperature for a thermal camera, a material or semantic ID, an emissivity-weighted radiance. Compiling user shading into the renderer would couple every such sensor to renderer internals. Instead the sensor reports what each ray hit, and a user kernel computes the quantity:

• shape_index_image: the shape index, or SensorCamera.HIT_TRIANGLE_MESH for deformable triangle surfaces, HIT_PARTICLES for particles, and HIT_NONE for misses.
• face_index_image: the triangle index into Model.tri_indices for deformable hits, the face index for mesh shapes, or the particle index.
• barycentric_image: the barycentric coordinates (u, v) of the hit, using Warp's convention p = u·p0 + v·p1 + (1 − u − v)·p2.

With these three images, any per-vertex, per-particle or per-shape attribute (including custom attributes) can be interpolated at the hit point. Requesting barycentrics makes the kernel use its full hit path, because the depth-only path skips them. The prototype's thermal camera (section 14) is built this way:

@wp.kernel
def compose_thermal(shape_index: wp.array3d[wp.uint32], face: wp.array3d[wp.int32], bary: wp.array3d[wp.vec2f],
                    tri_indices: wp.array2d[wp.int32], vertex_temp: wp.array[float], shape_temp: wp.array[float],
                    background: float, out_temp: wp.array2d[float]):
    y, x = wp.tid()
    s = shape_index[0, y, x]
    t = background
    if s == wp.uint32(wp.static(SensorCamera.HIT_TRIANGLE_MESH)):
        f = face[0, y, x]
        b = bary[0, y, x]
        t = (b[0] * vertex_temp[tri_indices[f, 0]] + b[1] * vertex_temp[tri_indices[f, 1]]
             + (1.0 - b[0] - b[1]) * vertex_temp[tri_indices[f, 2]])
    elif s < wp.uint32(0xFFFFFFF0):
        t = shape_temp[wp.int32(s)]
    out_temp[y, x] = t

thermal.update(state, shape_index_image=si, face_index_image=fi, barycentric_image=bc)
wp.launch(compose_thermal, (thermal.height, thermal.width),
          inputs=[si, fi, bc, model.tri_indices, model.thermal.temperature, model.thermal.surface_temperature, 12.0],
          outputs=[temperature_image])

8. USD and MJCF import

Both importers gain a flag named after their existing site flags: add_usd(..., load_cameras=True) (next to load_sites) and add_mjcf(..., parse_cameras=True) (next to parse_sites). Body resolution uses the same "nearest rigid-body ancestor, fold intermediate Xforms into the local transform" rule as sites and visual shapes, which fixes the IsaacLab #8211 class of failures by construction.

8.1 USD (UsdGeom.Camera)

Schema disposition (required by CODING_GUIDELINES "USD schema and importer support"): supported through standard UsdGeom.Camera and UsdRender.Product, with no new schema in phase 1. Lens distortion is deferred to phase 2 and needs an owner and an issue (it can track newton#3578). Tests must check that a USD-authored camera and an equivalent add_camera call produce identical Model rows, including defaults, units, frames and invalid inputs.

8.2 MJCF (<camera>)

9. Worlds, replication and views

Cameras follow the rules shapes already follow in add_builder/replicate: the body index is remapped, the world becomes the destination world, world-fixed cameras (body == -1) inside the source builder get the per-copy xform/spacing applied, and label_prefixes are prepended. Global cameras are only ever declared at the global scope of the outer builder.

Heterogeneous worlds need no special handling. If world 7 has no wrist camera, the pattern simply matches one fewer camera, and view_world tells the caller which environments are present. This is the case that "one SensorCamera per camera replicated like meshes" (#3627) could not express.

10. Isaac Lab integration

Isaac Lab already registers IMU, frame-transformer and ray-caster sites on the prototype builder before replication (NewtonManager.cl_register_site → builder.add_site). Cameras would follow the same path, either from the env_0 USD prim it already spawns or directly from CameraCfg:

# Prototype builder, before replication (R1)
proto.add_usd(stage, root_path="/World/envs/env_0", load_cameras=True)
# ...or straight from the config, like cl_register_site does for sites:
proto.add_camera(
    body=hand_body,
    xform=offset_in_opengl_convention,          # CameraCfg.OffsetCfg converted once
    focal_length=lens_m(cfg.spawn.focal_length),         # cfg lens units -> [m]
    horizontal_aperture=lens_m(cfg.spawn.horizontal_aperture),
    clipping_range=cfg.spawn.clipping_range,
    resolution=(cfg.width, cfg.height),
    label="Robot/ee_link/palm_link/Camera",
)
builder.replicate(proto, copy_count=num_envs,
                  label_prefixes=[f"/World/envs/env_{i}" for i in range(num_envs)])
model = builder.finalize()

# One SensorCamera per CameraCfg (R7); all share the model's RenderContext.
cam = SensorCamera(model, cameras="/World/envs/env_*/Robot/ee_link/palm_link/Camera",
                   width=cfg.width, height=cfg.height,
                   ray_mode=SensorCamera.RayMode.ANALYTIC)   # if intrinsics are randomized (R5)
rgb = cam.allocate_image_color()        # (num_envs, H, W), aliased into the torch buffer
depth = cam.allocate_image_depth_forward()

# Inside the physics CUDA graph
cam.update(state, color_image=rgb, forward_depth_image=depth, view_mask=envs_due)  # R6

11. Alternatives considered

The useful way to compare the options is to ask where each piece of camera data lives. Model = replicated scene description, Sensor = render state, User = recomputed by the caller.

Why not "one SensorCamera per camera, replicated like meshes"? Newton has no shared-asset-to-instance mapping for shapes either: meshes are shared, but shape instances are per world. Cameras need the same per-instance treatment for heterogeneous worlds and per-env intrinsics. The underlying goal of that design, one array of camera records instead of loose index arrays, is met by the view table, which the sensor owns, not the user.

12. Lidar and other sensors

12.1 Lidar is a site-bound SensorCamera with custom rays

A scanning range sensor is a set of rays from a mounted frame plus the hit distance along each ray. SensorCamera already traces arbitrary per-ray origins and directions and writes depth_image, so with sites as a frame source (6.1) and sensor-owned custom rays (7.2, 7.3), a lidar needs no new sensor class. That also matches the direction from the review of newton#3347 (the closed SensorLidar request newton#3346): consolidate ray sensing around the camera sensor and newton.intersect_ray, and keep models of specific scanners downstream. Newton provides the mechanism, and a scanner model is a ray pattern or ray function.

# Mount: an ordinary site; no new Model concept.
robot.add_site(base, xform=wp.transform((0.0, 0.0, 0.35), wp.quat_identity()), label="lidar")
...
channels, steps = 32, 1024
lidar = SensorCamera(model, sites="env_*/lidar", width=steps, height=channels,
                     clipping_range=(0.3, 100.0))          # min / max range [m]
lidar.set_camera_rays(spinning_pattern(channels, steps))   # (32, 1024, 1, 2), unit dirs in the site frame
ranges = lidar.allocate_image_depth()                       # (num_envs, 32, 1024) [m]
no_return = SensorCamera.ClearData(clear_depth=-1.0)        # miss value

lidar.update(state, depth_image=ranges, clear_data=no_return)
# Optional: shape_index_image for per-point semantics, normal_image for incidence angle.

Irregular patterns (rosette or solid-state scanners, sparse fans) use a (1, N) "image", or a grid with zero directions in unused slots. Non-repeating patterns either call set_camera_rays each step (a graph-safe copy) or use a ray function that reads a step counter from its parameters.

12.2 How close to trivial is it?

Most of it is already there. These are the gaps, and what closes them:

12.3 Declaring other sensors

Lidar fits the rule that this proposal uses for every sensor kind:

An entity that follows the pattern provides add_<kind>(body, *, xform, ..., label, custom_attributes), <kind>_body/_transform/_world/_label arrays, an AttributeFrequency and a <kind>s= selector on its sensor. Sites stay; nothing is deprecated. If a standard lidar schema appears in USD later, lidar can move to an entity through the same steps cameras take here.

13. Compatibility

• SensorCamera(model) and update(state, camera_transforms, camera_rays, ...) are unchanged. Making the two positional arguments optional is not a breaking change.
• allocate_image_* gain optional view_count/width/height defaults for bound sensors. Existing positional calls are unchanged.
• Assets that contain cameras get a nonzero camera_count after import. Nothing renders unless a sensor selects them. Pass load_cameras=False/parse_cameras=False to opt out, for example for stages with viewport cameras inside root_path.
• SensorTiledCamera stays deprecated and gets no camera-entity support.
• Mark RayMode.ANALYTIC and AttributeFrequency.CAMERA custom attributes as experimental for one release, and ray_function, ray_params and RayQuery as experimental until the signature has settled, following the experimental-API procedure.

14. Prototype

To check that the design holds together, phase 1 and parts of phase 2 were prototyped against Newton main @ c81f4f332. The prototype is exploratory and unpublished; it is meant to answer design questions, not to be merged as is. All images below were rendered by the prototype's SensorCamera.

14.1 What was built

The existing camera, tiled-camera, USD-camera, builder replication, custom-attribute, model, coupled-solver and tendon-builder tests (562 tests, 109 skipped) and the MJCF and USD importer suites pass with the prototype.

14.2 Examples

Thermal camera on a squeezed soft block (R5, R13, R15). A tetrahedral block simulated with VBD rests on the ground and is squeezed in a loop by keyframed parallel jaws. Temperatures are a PARTICLE-frequency custom attribute; the jaws' and ground's surface temperatures are a SHAPE-frequency attribute. Each frame, every tetrahedron releases heat q = L · ‖ΔE‖F · V, where ΔE is the change of its Green strain E = ½(FᵀF − I) since the last frame, V its rest volume and L a hysteresis loss per unit strain change [J/m³]. The heat is split over the four vertices, and each vertex's temperature rises by its share divided by its lumped heat capacity (density × specific heat × the vertex's share of tetrahedron volume). Heat then diffuses along tetrahedron edges and surface vertices relax toward the air, jaw and ground temperatures. Loss, diffusivity and cooling rates are exaggerated so that a single squeeze is visible. The thermal camera is a CUSTOM camera declared in the builder; its barrel-distorted lens is the ray function from section 7.3, and the thermal image is composed with the kernel from section 7.5.

Lens zoo (R4, R11, R13). One sensor renders a PINHOLE camera and two CUSTOM cameras on a rotating rig in a single launch, with four samples per pixel and SSAA. The custom cameras receive a Kannala-Brandt fisheye bundle and an OpenCV-distorted bundle through one set_camera_rays call.

Site-mounted lidar (R14). Two worlds with different obstacle layouts each contain a robot driving in a circle, with a lidar site inside a small housing. SensorCamera(sites=...) uses a 16 × 360 spinning pattern updated in place every frame; a near clip of 0.12 m keeps the beams from hitting the housing, and the ranges are converted to a world-space point cloud.

Replicated cameras and USD import (R1–R3, R5–R8). The MJCF humanoid's two <camera> elements and a head camera added before replicate give per-world body-mounted cameras. A global overview camera renders alternate halves of the worlds via view_mask, and the side cameras use ANALYTIC rays with per-world randomized focal lengths. A separate example imports a UsdGeom.Camera nested under a non-body Xform of a rigid body, with its clip range and a UsdRender.Product resolution, and selects the cameras by prim path.

14.3 Findings

• Stacked bundles instead of a fifth array dimension. Warp arrays have at most four dimensions, so per-view bundles are stacked along rows as (bundle_count · H, W, S, 2) and selected with view_ray_bundle. Deduplicating identical pinhole lenses kept the replicated scenes at one bundle per sensor.
• Near clipping is cheap. The trace starts near along each ray, and distances are reported from the unshifted origin. Passing far per view at run time also removes the kernel variant per max_distance value for bound sensors.
• Custom imaging needs per-hit data, not only custom rays. Shape index, face index and barycentrics are enough for the thermal camera and generalize to other channels without a shading hook. The depth-only trace path drops barycentrics, so requesting them selects the full path.
• A fixed parameter table works for ray functions. wp.array2d[float32] keeps the kernel signature stable, and gathering CAMERA custom attributes into it is one small kernel.
• Render-context sharing changes lighting semantics. The prototype shares one context only among bound sensors, so their lights are shared while unbound sensors keep their own. Open question 5 remains.
• Import order matters. USD cameras must be added after fixed-joint collapsing so body indices are final, and collapse_fixed_joints itself must remap cameras like shapes.
• MJCF tracking modes are visible. The humanoid's trackcom cameras rotate with the torso when imported as fixed mounts. MuJoCo parity needs tracking modes.
• Heat models must be robust to solver noise. A first version that heated from the squared strain rate was dominated by velocity jitter and the release snap. Per-frame strain-change hysteresis was stable. Normalizing by each vertex's volume share removed a lattice pattern from the five-tetrahedra grid split.

15. Phasing

16. Open questions

1. Lens length units. SI meters (guideline-consistent, MJCF-native) or USD-style "consistent units, only ratios matter" (what the existing compute_camera_rays_pinhole documents)? This proposal says meters, with conversion in the USD importer.
2. Default ray mode. PRECOMPUTED for speed, or ANALYTIC so randomization "just works"? The analytic pinhole is a handful of flops per pixel next to a BVH traversal, so a benchmark (#3627's asv camera bench) should decide.
3. Global camera semantics. Is "a world -1 camera renders every world" the right default, or should it require an explicit worlds= on the sensor?
4. Runtime poses in State? Writing model.camera_transform follows the shape_transform precedent. Externally animated cameras (teleop, cinematics) could argue for a State field instead.
5. Lighting when RenderContext is shared. Lights are scene state. Until phase 3, should create_default_light on one sensor affect all sensors on the model (honest), or should each sensor keep a light override?
6. Keep camera_resolution? It is only a hint (MJCF resolution, UsdRender.Product). Dropping it makes width/height always required.
7. Lens representation for phase 2. An enum plus fixed-width coefficient rows (device side, randomizable, mujoco_warp-like), or frozen lens dataclasses (as in the earlier prototype)?
8. Exposing view poses. Is SensorCamera.compute_view_transforms enough for viewers and Isaac Lab's FrameView, or do we want a model-level eval_camera_transforms(model, state, out)?
9. Ray function parameters. The prototype uses (params: wp.array2d[float32], RayQuery) -> (origin, direction). Is a fixed float table enough, or should arbitrary user structs be supported? Should the function also get the view pose, for rolling-shutter-style models?
10. Per-hit outputs vs a shading hook. Shape index, face index and barycentrics covered the thermal camera. Are they enough for other modalities, or should the renderer also accept a user shading function, for example for lighting-dependent channels?
11. Sites as a frame source. SensorCamera(sites=...) keeps lidar out of the Model. The alternative is declaring lidars as CUSTOM cameras, which gives one selection path but puts meaningless intrinsics on them.
12. What ray sensors hit. The render scene includes visual-only shapes, particles and Gaussians, and it has no per-view shape or body exclusion. Does range sensing need a collision-geometry mode or an exclusion list, or is that intersect_ray's domain?

17. References

• newton#3276 "Add SensorBatchedCamera" (closed): camera_indices = [world, ray_bundle]; review threads on index layout.
• newton#3627 "New CameraSensor Implementation" (closed): camera-as-site with set_site_camera, USD/MJCF parsing, asv camera benchmark.
• Earlier prototype PR "Add cameras as first-class objects" (closed): add_camera, deduped projections, viewer frusta.
• mujoco_warp render_util.create_render_context(cam_active=..., use_precomputed_rays=...); batched cam_fovy/cam_intrinsic.
• Isaac Lab main 8c9b995: newton_warp_renderer.py, newton_site_frame_view.py, newton_manager.py (cl_register_site), camera_cfg.py; issues and PRs #6928, #7036, #7688, #7916, #8000, #8211, #8545, #8664, #8665, #8682, #8765.
• Newton: sensor_camera.py, builder.py (add_site, replicate), model.py (AttributeFrequency, AttributeSpec), sites.rst, sensors.rst; newton#3346 and newton#3347 (closed SensorLidar request and PR, with the review direction cited in section 12), newton#3578 (OpenCV distortion).