Attaching a camera#

A worked tour of the Genesis ↔ Nyx integration surface. The script mounts a Nyx camera on a Franka panda’s wrist, records the wrist view as an MP4 while the robot performs a short pick, and never once touches plugin-internal code, every interaction goes through stock Genesis APIs.

The integration in one sentence#

Nyx ships its renderer as a standard Genesis sensor. NyxCameraOptions derives from genesis.options.sensors.camera.BaseCameraOptions, NyxCameraSensor derives from genesis.engine.sensors.camera.BaseCameraSensor. Everything else, attachment, lifecycle, recording, follows from that.

Where the plugin meets Genesis#

Five hand-offs happen in this example. They are the entire integration contract; nothing in user code is plugin-private.

1. Sensor registration — `scene.add_sensor`#

cam = scene.add_sensor(NyxCameraOptions(
    res            = (1280, 720),
    fov            = 60.0,
    near           = 0.02,
    far            = 50.0,
    entity_idx     = franka.idx,
    link_idx_local = franka.get_link("hand").idx_local,
    offset_T       = WRIST_OFFSET_T,
    spp            = 32,
    render_mode    = npr.ERenderMode.FastPathTracer,
    env_maps       = (env_map,),
))

scene.add_sensor is Genesis’ built-in sensor factory. It uses the type of the options object to pick a sensor class, NyxCameraOptions resolves to NyxCameraSensor via the typed generic in its base class. The sensor is then held by Genesis’ SensorManager and gets the same lifecycle (build, step, read) as a contact sensor, an IMU, or any other built-in.

2. Link attachment — Genesis-owned, not Nyx#

entity_idx, link_idx_local, and offset_T are inherited from the KinematicSensorOptionsMixin Genesis ships. When you set them, Genesis’ SensorManager calls move_to_attach() on the sensor every time the scene steps. move_to_attach() reads the link’s world transform from the rigid solver, composes it with offset_T, and feeds the result back into the Nyx renderer as the camera pose. The plugin only implements the small _apply_camera_transform callback that Genesis invokes with the final 4×4 matrix.

Net effect: the wrist camera tracks the gripper for free. There is no cam.update_pose() loop in the example, and there couldn’t be, the per-step camera transform is owned by the Genesis side.

3. SDK lifecycle — `gs.register_external_module`#

# excerpt from gs_nyx_plugin/nyx_camera_options.py
gs.register_external_module(nps.startup, nps.shutdown)

The plugin registers its SDK start / stop pair with Genesis at import time. gs.init() then walks its registry and calls every registered startup; gs.destroy() (or interpreter teardown) does the symmetric shutdown.

This is also why mocking genesis for docs builds works, the plugin doesn’t have a hard dependency on a running Nyx SDK to be importable, only to be used.

4. Reading triggers rendering — `cam.read()`#

scene.start_recording(
    data_func   = lambda: cam.read().rgb,
    rec_options = gs.recorders.VideoFile(filename=OUTPUT_PATH, fps=FPS),
)

cam.read() is the Genesis sensor read API. For NyxCameraSensor the read is lazy: it asks Genesis’ base machinery “am I stale for the current sim step?”, and only if the answer is yes calls _render_current_state(). That method does the actual Genesis ↔ Nyx glue:

scene.visualizer.update_visual_states() (Genesis) syncs visual transforms from the solvers.
move_to_attach() (Genesis, on each attached sensor) refreshes the camera pose from the link.
The Nyx renderer’s update_scene / render are then called with the up-to-date state.
The frame is wrapped in a NyxCameraData named-tuple. Because the scene was built without n_envs, cam.read().rgb is a (H, W, 3) uint8 CUDA tensor; with scene.build(n_envs=N) the same field comes out as (N, H, W, 3) and you’d index [0], that’s the single change between this script and 01_hello_nyx.py.

User code never calls a render function explicitly. One scene.step() followed by one cam.read() is the whole loop.

5. Recording — stock Genesis pipeline#

scene.start_recording and gs.recorders.VideoFile are both Genesis, not Nyx. The Genesis RecorderManager invokes data_func after each step, hands the returned (H, W, 3) uint8 array to the VideoFile writer, and the writer pipes frames through PyAV to an H.264 MP4. The Nyx camera is just a data source, swap it for a Genesis rasterizer camera and the same recorder code works unchanged.

Why the timing lines up#

dt = 1 / FPS ties simulation time to video time: one scene.step() produces exactly one frame of the recording, so the MP4 plays back at the same speed the robot moved. If you raise FPS, drop dt correspondingly; the recorder doesn’t care about real wall-clock time, only step counts.

Source#

"""Attached camera example for the Nyx renderer plugin.

Mounts a ``NyxCameraOptions`` on the Franka panda's wrist via Genesis'
built-in sensor-attachment fields (``entity_idx`` / ``link_idx_local`` /
``offset_T``), then streams its RGB output to an MP4 through
``scene.start_recording`` while the robot performs a short pick. The camera
follows the link automatically every step, no plugin-specific glue.

Usage:
    uv run python examples/02_attached_camera.py
"""

from __future__ import annotations

import os

import numpy as np

import genesis as gs
import gs_nyx.nyx_py_renderer as npr
from gs_nyx_plugin.nyx_camera_options import NyxCameraOptions


HERE        = os.path.dirname(__file__)
OUTPUT_PATH = os.path.join(HERE, "out", "02_attached_camera.mp4")

FPS = 30

# Wrist camera pose in the Franka "hand" link frame. The camera sits at
# (10 cm, 8 cm, 0) in the hand frame and is rotated to look at the gripper
# fingertip at (0, 0, 10 cm) — the rotation is a standard look-at with
# hand -Z as the up hint, which is world up during the grasp because the
# wrist's R_x(180°) pose flips Z between hand and world. That keeps the
# closing jaws centered as the gripper descends onto the cube.
WRIST_OFFSET_T = np.array(
    [
        [ 0.624695, -0.480604,  0.615457,  0.10],
        [-0.780869, -0.384483,  0.492366,  0.08],
        [ 0.000000, -0.788170, -0.615457,  0.00],
        [ 0.000000,  0.000000,  0.000000,  1.00],
    ],
    dtype=np.float64,
)


def main() -> None:
    gs.init(backend=gs.cpu)
    os.makedirs(os.path.dirname(OUTPUT_PATH), exist_ok=True)

    scene = gs.Scene(
        sim_options=gs.options.SimOptions(dt=1.0 / FPS, substeps=3),
        show_viewer=False,
    )

    scene.add_entity(gs.morphs.Plane())
    scene.add_entity(gs.morphs.Box(size=(0.04, 0.04, 0.04), pos=(0.65, 0.0, 0.02)), surface=gs.surfaces.Gold(roughness=0.1))
    franka = scene.add_entity(gs.morphs.MJCF(file="xml/franka_emika_panda/panda.xml"))

    # Camera intrinsics + attachment to the Franka wrist + lighting. The
    # ``lights`` list is plain Genesis sensor config; the plugin converts each
    # dict into an Nyx ``LightAsset`` at build time.
    cam = scene.add_sensor(NyxCameraOptions(
        res            = (1280, 720),
        fov            = 60.0,
        near           = 0.02,
        far            = 50.0,
        entity_idx     = franka.idx,
        link_idx_local = franka.get_link("hand").idx_local,
        offset_T       = WRIST_OFFSET_T,
        spp            = 32,
        render_mode    = npr.ERenderMode.FastPathTracer,
        lights         = [{
            "type":      "directional",
            "dir":       (-0.4, -0.4, -0.8),
            "color":     (1.0, 1.0, 1.0),
            "intensity": 5.0,
            "shadow":    True,
        }],
    ))

    # Stream RGB frames to an MP4 file. ``cam.read().rgb`` is shape
    # (H, W, 3) uint8 in single-env mode.
    scene.start_recording(
        data_func   = lambda: cam.read().rgb,
        rec_options = gs.recorders.VideoFile(filename=OUTPUT_PATH, fps=FPS),
    )

    scene.build()

    # Franka controller gains (from the Genesis IK tutorial)
    franka.set_dofs_kp(np.array([4500, 4500, 3500, 3500, 2000, 2000, 2000, 100, 100]))
    franka.set_dofs_kv(np.array([ 450,  450,  350,  350,  200,  200,  200,  10,  10]))
    franka.set_dofs_force_range(
        np.array([-87, -87, -87, -87, -12, -12, -12, -100, -100]),
        np.array([ 87,  87,  87,  87,  12,  12,  12,  100,  100]),
    )
    motors_dof  = np.arange(7)
    fingers_dof = np.arange(7, 9)
    hand        = franka.get_link("hand")

    def execute_path(target_pos, gripper=0.04, num_waypoints=60):
        qpos = franka.inverse_kinematics(
            link=hand, pos=np.array(target_pos), quat=np.array([0, 1, 0, 0]),
        )
        qpos[-2:] = gripper
        for waypoint in franka.plan_path(qpos_goal=qpos, num_waypoints=num_waypoints):
            franka.control_dofs_position(waypoint)
            scene.step()

    def hold(seconds: float):
        for _ in range(int(round(seconds * FPS))):
            scene.step()

    # --- Scripted pick task -------------------------------------------------
    execute_path((0.65, 0.0, 0.25))                                # pre-grasp above cube
    execute_path((0.65, 0.0, 0.13))                                # descend onto cube
    franka.control_dofs_force(np.array([-0.5, -0.5]), fingers_dof) # close gripper
    hold(0.5)

    qpos_lift = franka.inverse_kinematics(
        link=hand, pos=np.array([0.4, 0.2, 0.35]), quat=np.array([0, 1, 0, 0]),
    )
    franka.control_dofs_position(qpos_lift[:-2], motors_dof)       # lift while gripping
    hold(1.5)

    scene.stop_recording()
    print(f"Saved {OUTPUT_PATH}")


if __name__ == "__main__":
    main()

Run it:

uv run python examples/02_attached_camera.py

The MP4 is written to examples/out/02_attached_camera.mp4. The Sphinx build copies that file into _static/generated/examples/02_attached_camera.mp4 and embeds it at the top of this page, so the video on the docs site stays in lock-step with whatever the latest run produced.