Sensor lifecycle#

The NyxCameraSensor integrates the Nyx renderer with Genesis using a render-on-read model. Scene.step() advances physics but does not render. Rendering is performed on the first read() call that follows a step, and the resulting image is cached in a CUDA tensor and reused by every subsequent read() until the simulation time advances or the camera pose changes.

This page describes the sensor’s order of operations across its three phases: construction, build, and the per-step render-on-read cycle.

Lifecycle overview#

Phase	Triggered by	GPU work
Construction	`scene.add_sensor()`	None
Build	`scene.build()`	Renderer startup, buffer allocation
Render-on-read	`cam.read()` after `scene.step()`	Per-camera, per-env render pass
Cached read	`cam.read()` while not stale	None
Destruction	Scene destruction	Renderer shutdown, buffer release

Construction#

scene.add_sensor(NyxCameraOptions(...)) instantiates a NyxCameraSensor from the given NyxCameraOptions and registers it with the sensor manager. The sensor options are stored on the instance. No renderer is created and no GPU resources are allocated.

A single scene can contain multiple Nyx sensors. They share a single renderer instance and a single shared image cache.

Build#

scene.build() calls build() on every sensor in registration order. For each Nyx sensor, build() performs the following steps:

Attach to the shared metadata block used by every Nyx sensor in the scene.
Append a camera definition to the shared camera list (resolution, FOV, pose, clip planes, sample count, tone mapper, anti-aliasing).
Allocate a (B, H, W, 3) torch.uint8 CUDA tensor in the shared image cache, keyed by the sensor index.

When the last Nyx sensor finishes building, that sensor performs the one-time scene initialisation:

Collect lights, environment maps, and light fields from every Nyx sensor.
Export a JSON scene description to __nyx_cache__/<random>/.
Create a single renderer instance. The internal viewport is sized to the maximum resolution across all sensors.
Allocate CUDA interop buffers: rigid transforms, deformable vertex, UV, and index buffers, and camera pose tensors.
Pre-upload static deformable data (UVs and fixed face indices).

After build() returns, the renderer is initialised and the sensor is ready to render on the next read().

Note

All Nyx sensors must be added before scene.build(). Adding sensors after the build is not supported.

Note

All Nyx sensors must use the same render_mode. The plugin raises an exception if the values diverge.

Render-on-read#

scene.step() advances scene.t and does not render. The next cam.read() triggers the render.

read() is inherited from BaseCameraSensor and runs the following steps:

Step	Action
1. Staleness check	If `shared_metadata.last_render_timestep != scene.t`, every Nyx sensor’s `_stale` flag is set to `True` and `last_render_timestep` is updated to `scene.t`.
2. Cached path	If `_stale` is `False`, the cached image is returned without further work.
3. Pose update	If the camera is attached to a rigid link, `move_to_attach()` recomputes its world transform.
4. Render	`_render_current_state()` is called. This renders every Nyx camera for every environment in a single pass.
5. Mark fresh	The `_stale` flag is cleared.
6. Return	The requested environments are sliced from the cached `(B, H, W, 3)` tensor and wrapped in a `NyxCameraData` namedtuple with an `rgb` field.

Because _render_current_state() renders every Nyx camera in one call, only the first read() of the step performs the full render pass. Subsequent read() calls on other Nyx sensors in the same step return their pre-rendered image from the cache.

Render pass#

_render_current_state() runs a one-time setup block followed by a per-environment loop.

Per-step setup

Step	Action
1	`scene.visualizer.update_visual_states()` flushes Genesis visual state.
2	For every attached sensor, `move_to_attach()` computes its per-env world transform.
3	`renderer.update()` pumps the GUI event loop and updates renderer delta time.

Per environment (env_idx in 0..B-1)

Step	Action
1	For every Nyx sensor, write its position, look-at, and up vector into the renderer’s CUDA pose tensors. Attached cameras use their per-env transform; static cameras use the same pose for every env.
2	`update_scene(env_idx)` copies the env’s rigid `vgeoms` pose, deformable solver vertex positions (FEM, PBD, MPM-visual), and reconstructed MPM-recon meshes, then selects the env map.
3	`update_scene_buffers()` syncs the new buffer pointers to the renderer.
4	For every Nyx sensor, `render_frame(camera_index)` is called. The returned `(H, W, 3)` `uint8` tensor is copied into `image_cache[sensor_idx][env_idx]`.

When the env loop completes, every sensor’s cache holds the current step’s frames for every env. All sensors are marked fresh and last_render_timestep is set to scene.t.

Image cache#

The image cache is stored on NyxCameraSharedMetadata .image_cache as a dict[int, torch.Tensor] keyed by sensor index.

Property	Value
Shape	`(B, H, W, 3)`
Dtype	`torch.uint8`
Device	CUDA
Sized per	Sensor (different sensors may have different resolutions)
Lifetime	Tied to the scene; freed on scene destruction

cam.read().rgb returns a view into the cache. Convert to a CPU NumPy array with .cpu().numpy() if required.

The renderer’s internal viewport is sized to the maximum resolution across all Nyx sensors in the scene. Individual camera viewports are clipped to their configured resolution at render time.

Cache invalidation#

The cache is invalidated when:

scene.t changes. scene.step() advances simulation time, which marks every Nyx sensor stale on the next read().
An attached camera’s link moves. Handled inside _render_current_state() via move_to_attach(); the new transform is pushed before rendering.
update_camera_pose() is called. This detaches the camera from any rig and forces a re-render on the next read().

The cache is not invalidated when:

cam.read() is called multiple times within the same step.
Non-pose options such as spp or denoise are mutated after scene.build(). These options are read at build time only.

Multi-camera scenes#

The plugin enforces a single shared renderer per scene. This has three consequences:

All Nyx sensors must use the same render_mode.
All Nyx sensors must be registered before scene.build() is called.
Reading any one Nyx sensor renders all of them. For setups with identical camera layouts per env, this is the intended behaviour. For setups with an expensive secondary camera that is read infrequently, the secondary camera is still rendered on every step where any Nyx sensor is read.

Live window#

When open_window=True, the GUI event loop is serviced inside _render_current_state() via the renderer.update() call. This is the only point at which window events are pumped.

Warning

If cam.read() is not called every step, the live window appears frozen between reads. Call read() once per step for a smooth preview.