Multiple cameras and parallel environments#

Two orthogonal patterns the Nyx plugin composes for free: multiple cameras in one scene and multiple Genesis environments built in parallel. This example puts both to work at once — two NyxCameraSensors and n_envs=4 — and tiles the resulting (2, 4, H, W, 3) image stack into a single PNG so the symmetry is visible at a glance.

Rendered output of

Both features are independent knobs on the same APIs — multi-camera is a second scene.add_sensor call, multi-env is a n_envs=N argument to scene.build. Either alone is already useful (multi-view supervision, batched RL rollouts, synthetic data generation). The two sections below cover what each one changes; the rest of the page covers the bits that only show up when you combine them.

What changes when you add a second camera#

Concretely, nothing in your code except a second scene.add_sensor(NyxCameraOptions(...)) call. Each sensor gets its own resolution, FOV, pose, attachment, and pick_pixel() entry-point. Under the hood the plugin keeps one native renderer alive for the whole scene and contributes every camera (plus its lights, env maps and light fields) to that shared instance.

That sharing is worth a moment because it turns three potential footguns into non-issues:

Lights and light fields are scene-wide. lights and light_fields from every Nyx sensor are merged at build time, so every env sees the union of them through both cameras. Declare them on whichever camera reads best in code; duplicating them on every camera duplicates the underlying asset. Env maps are the exception — see Different lighting per environment below.
Render mode is scene-wide too. Every Nyx sensor must agree on render_mode — the shared renderer can only serve one mode at a time. Mixing ERenderMode values like FastPathTracer and Debug raises at build.
pick_pixel indexes registration order. The first sensor added is camera_index=0, the second 1, and so on, all reachable from pick_pixel() (camera_index, x, y) on any sensor. See Object picking for the ray API itself.

What changes when you add parallel environments#

scene.build(n_envs=N) widens every sensor’s read shape from (H, W, 3) to (N, H, W, 3). The cost is honest: the plugin renders one env at a time internally, so wall-clock time grows linearly with N. The benefit is that the simulator does too — one scene.step() advances every env, and one cam.read() returns the whole batch.

Per-env state lives on the entities, not the camera. After build(), any state setter accepts an (N, ...)-shaped argument and writes a different value into each env:

ball.set_pos([
    [ 0.0,  1.2, 0.3],
    [ 1.2,  0.0, 0.3],
    [ 0.0, -1.2, 0.3],
    [-1.2,  0.0, 0.3],
])

That single call is what makes the four columns of the wide overview differ — the camera pose is constant, but the simulator has placed the ball somewhere else in every env. Anything you can set per env in Genesis (rigid pose, joint positions, particle state, …) shows up the same way in the next render.

Different camera positions per environment#

Per-env state is on entities, so to place a camera differently in every env you bind it to one. Add an invisible rigid entity to act as an attachment target, attach the camera to it via entity_idx / offset_T, and set_pos / set_quat it per env — the camera moves with it, exactly the way it would move with a Franka wrist link in Attaching a camera. offset_T is the camera’s pose in the rig’s local frame, so the same offset stays consistent relative to the rig no matter where the rig ends up; per-env position slides the camera around the world, per-env orientation rotates its viewing angle. In this example the rig is set to point outward from the centre of the plane in each env, so the follow row shows the same ball framed from four different radial angles, each against a different slice of the HDRI sky.

Different lighting per environment#

env_maps is the one piece of the camera config that the renderer indexes by env rather than merging scene-wide: env i is lit by env_maps[i]. Passing an N-long tuple to a single NyxCameraOptions is therefore enough to render every env under its own HDRI, with no extra calls or per-step bookkeeping. This example alternates two assets across four envs:

env_maps = (env_map_sky, env_map_studio, env_map_sky, env_map_studio)

Even-indexed envs render under an outdoor sky, odd-indexed envs under a studio softbox. The choice flows through both cameras automatically because the renderer instance is shared, so the top-row wide overview and the bottom-row follow camera both see env 0 lit by env_map_sky, env 1 by env_map_studio, and so on. A few constraints worth knowing:

The mapping is strictly positional. There is no per-env override API or “default” entry; the tuple must be at least n_envs long.
The tuple is collected at build time across every Nyx sensor in the scene. Declare it on one sensor (as here) to keep the source of truth in a single place; declaring it on multiple cameras concatenates the lists and the indexing drifts.
Each entry is a full EnvironmentMapAsset, so per-env intensity (multiplier), rotation, and tint are all available — useful for domain-randomized lighting on top of a single base HDRI.

Heterogeneous environments: not supported#

The (N, ...)-shaped state setters above all assume every env contains the same geometry — only the state differs. Genesis itself has a stronger form of variation called heterogeneous simulation, where you pass a tuple of morph variants to scene.add_entity and the rigid solver hands each env a different variant:

# Valid Genesis call. Will simulate correctly, but Nyx will NOT render it correctly.
scene.add_entity(morph=(
    gs.morphs.Box(size=(0.04, 0.04, 0.04), pos=(0.0, 0.0, 0.1)),
    gs.morphs.Sphere(radius=0.03,           pos=(0.1, 0.0, 0.1)),
))

The Nyx plugin does not currently support heterogeneous morphs. The scene description the plugin builds at scene.build() time captures a single set of meshes shared across all envs, so even if the rigid solver is dispatching a different variant per env on the physics side, every env will render with the same geometry the plugin saw first. The mismatch is silent — no exception is raised, the image just doesn’t match the simulation.

If you need different models per env, the options today are:

Spawn every variant in every env and toggle their state per env. Add all candidate morphs as separate entities, then in each env push the inactive ones below the plane (or to a far-away corner) via set_pos. Geometry stays homogeneous from the plugin’s perspective, so rendering is correct.
Build separate scenes. Run one Genesis scene per variant and read each one independently. You lose the batched-step speed-up but keep both physics and rendering correct.

State-based variation works fine: positions, joint angles, velocities, masses through domain randomization, and per-env lighting (see Different lighting per environment above). The limitation only applies when you need a different mesh in each env.

Reading the saved image#

The PNG is a 2 × 4 grid. Top row: the wide overview, four times — same camera pose, four different ball positions, with the lighting alternating outdoor sky / studio softbox / outdoor sky / studio softbox across columns. Bottom row: the follow camera, four times — both the camera pose and the ball move per env, with the rig rotated outward from the centre so each cell shows the ball from a different radial angle, lit by the same HDRI as the wide cell directly above it. Comparing top and bottom of the same column tells you per-env state and per-env env maps are shared across cameras; comparing left to right of the same row tells you the per-env variation actually took effect.

Source#

"""Multi-camera, multi-environment example for the Nyx renderer plugin.

Builds a scene with **two** ``NyxCameraSensor`` instances and ``n_envs=4``
parallel Genesis environments, then teleports the rendered ball to a
different spot on the plane in each env. The first camera is a static
high overhead view shared across every env; the second is a low follow
camera attached to an invisible rigid rig whose position **and** orientation
are set per env, so each follow cell approaches the ball from a different
direction. A single ``read()`` per camera returns a batched
``(N_ENVS, H, W, 3)`` tensor, which the script tiles into one PNG
(rows = cameras, columns = envs).

Usage:
    uv run python examples/07_multi_camera_multi_env.py
"""

from __future__ import annotations

import math
import os

import torch
from PIL import Image

import genesis as gs
import gs_nyx.nyx_py_renderer as npr
import gs_nyx.nyx_py_sdk as nps
from gs_nyx_plugin.nyx_camera_options import NyxCameraOptions


HERE         = os.path.dirname(__file__)
PBR_BALL     = os.path.join(HERE, "assets", "PBR_Ball.glb")
ENV_MAP_SKY  = os.path.join(HERE, "assets", "kloppenheim_07_puresky_4k.hdr")
ENV_MAP_STUD = os.path.join(HERE, "assets", "green_sanctuary_4k.hdr")
OUTPUT_PATH  = os.path.join(HERE, "out", "07_multi_camera_multi_env.png")

N_ENVS  = 4
CAM_RES = (640, 480)

# One ball position per env, arranged on a circle of radius R so the wide
# overview shows the ball at four clearly different spots. The Z component
# lifts the ball off the plane so it sits cleanly above the ground.
R = 1.2
BALL_POSITIONS = [
    [ 0.0,  R, 0.0],
    [  R, 0.0, 0.0],
    [ 0.0, -R, 0.0],
    [ -R, 0.0, 0.0],
]

# Per-env rig orientation (Z-axis rotation, WXYZ quaternion). Rotates the
# rig so that the follow camera's "behind" direction always points outward
# from the centre of the plane — every env sees the ball from a different
# radial angle, which is what makes the follow row visually distinct.
RIG_OUTWARD_QUATS = [
    [           0.0, 0.0, 0.0,            1.0],  # 180° around Z
    [math.sqrt(0.5), 0.0, 0.0, -math.sqrt(0.5)],  # -90° around Z
    [           1.0, 0.0, 0.0,            0.0],  #   0° around Z
    [math.sqrt(0.5), 0.0, 0.0,  math.sqrt(0.5)],  # +90° around Z
]


def main() -> None:
    gs.init()
    os.makedirs(os.path.dirname(OUTPUT_PATH), exist_ok=True)

    scene = gs.Scene(
        sim_options=gs.options.SimOptions(dt=0.01),
        show_viewer=False,
    )

    scene.add_entity(morph=gs.morphs.Plane(plane_size=(10.0, 10.0)))

    # Override the GLB's embedded materials with a polished sapphire-blue
    # chrome — low roughness + full metallic so each follow camera's
    # reflection of the HDRI sky reads as a distinct slice of the dome.
    # ``align=False`` keeps the link origin at the GLB's authored pivot
    # (the cup base) instead of re-framing it to the composite's COM, so
    # ``set_pos(z=0)`` lands the cup cleanly on the plane.
    ball = scene.add_entity(
        morph=gs.morphs.Mesh(file=PBR_BALL, pos=(0.0, 0.0, 0.0), align=False),
        surface=gs.surfaces.BSDF(metallic=1.0, roughness=0.15),
    )

    # Invisible rigid rig used as the follow camera's attachment point.
    # Both its position and its orientation are set per env, which is how
    # the follow camera ends up at a different world pose in every env.
    # Must be added before any FEM / MPM / PBD entity so its ``entity.idx``
    # matches the rigid solver's index — that's what
    # ``NyxCameraOptions.entity_idx`` uses.
    follow_rig = scene.add_entity(
        morph=gs.morphs.Sphere(
            pos=(0.0, 0.0, 0.0), radius=0.01,
            visualization=False, fixed=True,
        ),
    )

    # Lighting. ``env_maps`` is indexed positionally by env: env ``i`` is
    # lit by ``env_maps[i]``. We alternate two HDRIs across the four envs
    # so even envs render under a clear outdoor sky and odd envs render
    # under a studio softbox setup — same scene, two completely different
    # lighting moods, visible side by side in the tiled output. ``lights``
    # and ``light_fields`` (not used here) are merged scene-wide instead.
    def _make_env_map(path: str) -> nps.EnvironmentMapAsset:
        asset            = nps.EnvironmentMapAsset()
        asset.texture    = path
        asset.layout     = nps.EEnvMapLayout.LongLat
        asset.multiplier = 2.0
        return asset

    env_map_sky    = _make_env_map(ENV_MAP_SKY)
    env_map_studio = _make_env_map(ENV_MAP_STUD)
    per_env_maps   = (env_map_sky, env_map_studio, env_map_sky, env_map_studio)

    # Camera A — high overhead overview. Same world pose in every env; what
    # differs per column is the simulator state inside the frame.
    wide_cam = scene.add_sensor(NyxCameraOptions(
        res         = CAM_RES,
        pos         = (0.5, -2.5, 4.0),
        lookat      = (0.0,  0.0, 0.5),
        fov         = 42.0,
        spp         = 64,
        render_mode = npr.ERenderMode.FastPathTracer,
        env_maps    = per_env_maps,
    ))

    # Camera B — eye-level follow camera. ``offset_T`` is the camera pose
    # in the rig's local frame: 1.1 m back along the rig's -Y axis and
    # 0.45 m above it, aimed at the ball portion of the composite (0.25 m
    # above the cup base). Rotating the rig per env rotates the camera's
    # relative pose with it, so the follow cell for each env shows a
    # different radial angle on the same ball.
    offset_T = gs.utils.geom.pos_lookat_up_to_T(
        torch.tensor((0.0, -1.1, 0.45), dtype=torch.float32),
        torch.tensor((0.0,  0.0, 0.25), dtype=torch.float32),
        torch.tensor((0.0,  0.0, 1.0),  dtype=torch.float32),
    ).cpu().numpy()

    follow_cam = scene.add_sensor(NyxCameraOptions(
        res         = CAM_RES,
        fov         = 45.0,
        spp         = 64,
        render_mode = npr.ERenderMode.FastPathTracer,
        entity_idx  = follow_rig.idx,
        offset_T    = offset_T,
    ))

    # ``n_envs=N`` makes every sensor return a batched tensor:
    # ``cam.read().rgb`` has shape ``(N_ENVS, H, W, 3)``.
    scene.build(n_envs=N_ENVS)

    # Per-env state setters. All three calls share the outer dimension
    # ``N_ENVS``: ball position, rig position (rig follows the ball), and
    # rig orientation (rig points outward in each env).
    ball.set_pos(BALL_POSITIONS)
    follow_rig.set_pos(BALL_POSITIONS)
    follow_rig.set_quat(RIG_OUTWARD_QUATS)

    # One step drives every sensor; one ``read()`` returns the full batch.
    scene.step()
    wide_batch   = wide_cam.read().rgb.cpu().numpy()
    follow_batch = follow_cam.read().rgb.cpu().numpy()

    # Tile: rows = cameras (wide on top, follow below), columns = envs.
    H, W = CAM_RES[1], CAM_RES[0]
    grid = Image.new("RGB", (W * N_ENVS, H * 2))
    for env_idx in range(N_ENVS):
        grid.paste(Image.fromarray(wide_batch[env_idx]),   (env_idx * W, 0))
        grid.paste(Image.fromarray(follow_batch[env_idx]), (env_idx * W, H))
    grid.save(OUTPUT_PATH)
    print(f"Saved {OUTPUT_PATH}")


if __name__ == "__main__":
    main()

Run it:

uv run python examples/07_multi_camera_multi_env.py

The PNG is written to examples/out/07_multi_camera_multi_env.png. The Sphinx build copies it to _static/generated/examples/07_multi_camera_multi_env.png and embeds it at the top of this page, so the docs site always shows whatever the latest run produced.