Object picking#

The Nyx plugin exposes a ray API on every Nyx camera: pick_pixel() (camera_index, x, y) casts a ray from the camera through one pixel of its framebuffer and returns the first Genesis entity it hits, along with the world-space hit point. It’s the entry point for picking, hover hit-tests, click-to-select tooling, and any “what’s under this pixel?” question you’d otherwise need a separate ID buffer for.

This example names four balls by their colour, casts a grid of rays across the rendered frame, and uses pick_pixel’s return value to (a) drop a colour-keyed dot on top of every hit and (b) print a per-ball hit count and mean world position to stdout.

Rendered output of

What it shows#

Four spheres named by their colour (gold, copper, red, cyan) on a Genesis Plane, lit by an HDR environment map.
A 48 × 36 grid of pick_pixel calls across the rendered frame — one ray per grid cell, 1 728 rays total.
Each hit drawn as a translucent coloured dot keyed by entity. Plane hits and sky misses are both filtered out — the overlay isolates only the four balls.
A stdout summary line per ball: <name> <count> hits mean pos=(x, y, z). The name comes from a Python dict keyed by the entity object the API returned; the position comes from the position field of the same pick_pixel result averaged across every hit on that ball.

The ray API in one call#

result = cam.pick_pixel(camera_index=0, x=px, y=py)

camera_index indexes into the cameras that share this sensor’s renderer. For a single NyxCameraSensor it’s always 0; multi-camera setups index in registration order.
x, y are pixel coordinates with the origin at the top-left of the framebuffer. They must lie inside the camera’s configured res.
The call returns either None (the ray hit nothing in the scene — the background / sky) or a NyxPickPixelResult named tuple with three fields:

Field	Type	Meaning
`entity`	`genesis.engine.entities.base_entity.Entity`	The same Python object the user passed to `scene.add_entity` — `id(result.entity)` is stable, so dict lookups by entity work. This is how the example recovers a colour name from a ray hit.
`link_name`	`str`	Sub-geometry hit. For URDF entities it’s the link name; for MJCF entities it’s `"<link>_<vgeom_idx>"`; empty string for `Mesh`, `Plane`, `Sphere`, etc. — primitives like the spheres in this example don’t have nameable sub-parts.
`position`	`tuple[float, float, float]`	World-space hit point `(x, y, z)` in Genesis Z-up coordinates — the plugin converts from Nyx’s internal Y-up before returning.

Getting a name back from the API#

Genesis primitives don’t carry a string identifier the renderer can hand you, so for non-URDF / non-MJCF entities link_name is empty. To label a hit with something the rendered image is named by, hold onto your own {entity: name} mapping at construction time and resolve through it after the pick:

scene.add_entity(morph=gs.morphs.Plane(...))                                          # in the scene, NOT in the name table
gold = scene.add_entity(morph=gs.morphs.Sphere(...), surface=gs.surfaces.Gold())
red  = scene.add_entity(morph=gs.morphs.Sphere(...), surface=gs.surfaces.Plastic(...))

names = {id(gold): "gold", id(red): "red", ...}

res = cam.pick_pixel(0, px, py)
if res is None:
    ...                                   # background (sky); ignore
elif id(res.entity) not in names:
    ...                                   # hit something we don't care about (the plane); ignore
else:
    name = names[id(res.entity)]          # "gold", "red", ...
    print(f"hit {name} at {res.position}")

The id() key works because pick_pixel returns the same Python object you passed to scene.add_entity — no copy, no proxy. Plain == works for the same reason, but id() keeps the lookup O(1) and side-effect-free. Leaving entities you don’t want to pick out of the lookup table is also how the example filters the plane: any ray that hits the ground falls into the not in names branch and is dropped, without ever needing a per-entity blacklist.

The picking loop#

The overlay is a transparent RGBA image the script draws into, then alpha-composites over the rendered RGB. Each accepted hit does two things — splat a dot, and accumulate the world hit position into a running sum keyed by name so the stdout summary at the end shows a mean rather than a single noisy sample:

for gy in range(GRID_H):
    py = int((gy + 0.5) * H / GRID_H)
    for gx in range(GRID_W):
        px  = int((gx + 0.5) * W / GRID_W)
        res = cam.pick_pixel(0, px, py)
        if res is None:
            continue                          # background (sky)
        name = name_by_eid.get(id(res.entity))
        if name is None:
            continue                          # entity outside the picking set (plane)

        draw.ellipse(..., fill=BALLS[name][2])
        wx, wy, wz = res.position
        h = hits[name]
        h[0] += 1; h[1] += wx; h[2] += wy; h[3] += wz

Notes worth keeping in mind#

Picking does not require a render. The renderer can pick-trace a ray as soon as the scene has been built — cam.read() (or even scene.step()) is not a prerequisite. The example happens to render first because it wants to draw the overlay over the rendered image, but the same pick_pixel calls would work right after scene.build(...).
Picking does not consume samples. Sample budgets (spp, render_mode, denoising) configure the render pipeline. pick_pixel always traces a single deterministic ray and is unaffected by those settings.
The picked entity object is the registered one. result.entity is entity_you_added_to_the_scene holds, so identity comparisons and id()-keyed dicts both work.
None is background, an unregistered entity is “ignore”. Sky rays return None; rays that hit something in the scene but not in your lookup table (the plane, in this example) come back as a fully valid NyxPickPixelResult you simply choose to skip. Both branches leave the original render untouched in the overlay.
One pick = one native call. The plugin doesn’t currently batch picks; pick_pixel() is a thin Python wrapper around a single native ray trace. For a few hundred picks per frame this is fine; for full-screen ID buffers you’ll want to render once at full resolution and only pick the pixels you actually need (the cursor location, a small lasso, etc.).

Reading the saved image and the stdout#

The coloured dots in the saved PNG are the entity-ID buffer: each one is one pick_pixel hit that resolved to a registered ball, coloured to match the rendered surface it landed on.
Where the dots disappear, the ray either missed every entity (sky) or hit the plane (filtered out by leaving it out of the picking table). Both cases let the original render show through unmodified.
The terminal output prints one line per ball, e.g. gold 123 hits mean pos=(-1.00, +0.50, +0.65). The colour name came back out of pick_pixel via the name_by_eid lookup; the mean position is res.position averaged across every hit on that ball, which is stable enough to compare against add_entity(pos=...) and confirm Z-up world coordinates.

Source#

"""Object picking example for the Nyx renderer plugin.

Demonstrates the **ray API** exposed through
``NyxCameraSensor.pick_pixel``: given a pixel coordinate, the renderer
traces a ray from the camera through that pixel and reports the first
scene entity it hits (or ``None`` for background).

The script names each ball by its colour, renders the scene, then casts
a coarse grid of rays. Every hit drops a translucent dot on the overlay
so the image reads like a low-resolution ID buffer. The per-ball hit
count and the mean world hit position recovered from ``pick_pixel`` are
printed to stdout — the API's return value is the only data source.

Usage:
    uv run python examples/06_object_picking.py
"""

from __future__ import annotations

import os

from PIL import Image, ImageDraw

import genesis as gs
import gs_nyx.nyx_py_renderer as npr
import gs_nyx.nyx_py_sdk as nps
from gs_nyx_plugin.nyx_camera_options import NyxCameraOptions


HERE        = os.path.dirname(__file__)
ENV_MAP     = os.path.join(HERE, "assets", "kloppenheim_07_puresky_4k.hdr")
OUTPUT_PATH = os.path.join(HERE, "out", "06_object_picking.png")

CAM_RES        = (960, 720)
GRID_W, GRID_H = 48, 36

# Four balls named by colour. ``name -> (surface, world_pos, dot_rgba)``.
# The plane is rendered for shadows but kept out of this dict, so any
# ray that hits the ground falls into the "not in BALLS" branch and is
# silently ignored.
BALLS = {
    "gold":   (gs.surfaces.Gold(),                                                      (-1.0,  0.5, 0.4), (255, 200,  50, 180)),
    "copper": (gs.surfaces.Copper(),                                                    ( 0.0,  0.0, 0.4), (255, 120,  60, 180)),
    "red":    (gs.surfaces.Plastic(color=(0.85, 0.10, 0.10)),                           ( 1.0,  0.5, 0.4), (255,  60,  60, 180)),
    "cyan":   (gs.surfaces.BSDF(color=(0.10, 0.60, 0.65), metallic=0.0, roughness=0.2), ( 0.5, -0.8, 0.4), ( 60, 220, 220, 180)),
}


def main() -> None:
    gs.init()
    os.makedirs(os.path.dirname(OUTPUT_PATH), exist_ok=True)

    scene = gs.Scene(sim_options=gs.options.SimOptions(dt=0.01), show_viewer=False)
    scene.add_entity(morph=gs.morphs.Plane(plane_size=(10.0, 10.0)))

    # Single lookup that ties an entity returned by ``pick_pixel`` back
    # to its colour name. Plane is not in here on purpose.
    name_by_eid: dict[int, str] = {}
    for name, (surface, pos, _) in BALLS.items():
        e = scene.add_entity(
            morph=gs.morphs.Sphere(pos=pos, radius=0.4, fixed=True, collision=False),
            surface=surface,
        )
        name_by_eid[id(e)] = name

    env_map            = nps.EnvironmentMapAsset()
    env_map.texture    = ENV_MAP
    env_map.layout     = nps.EEnvMapLayout.LongLat
    env_map.multiplier = 2.0

    cam = scene.add_sensor(NyxCameraOptions(
        res         = CAM_RES,
        pos         = (2.5, -3.5, 2.6),
        lookat      = (0.0, 0.0, 0.3),
        fov         = 30.0,
        spp         = 64,
        render_mode = npr.ERenderMode.FastPathTracer,
        env_maps    = (env_map,),
    ))

    scene.build(n_envs=1)
    scene.step()

    base    = Image.fromarray(cam.read().rgb[0].cpu().numpy()).convert("RGBA")
    W, H    = CAM_RES
    overlay = Image.new("RGBA", (W, H), (0, 0, 0, 0))
    draw    = ImageDraw.Draw(overlay)
    dot_r   = max(1, min(W // GRID_W, H // GRID_H) // 2 - 1)

    # Per-ball: [hit count, sum world x, sum world y, sum world z].
    # Stays empty for any ball the grid never lands on.
    hits: dict[str, list[float]] = {n: [0.0, 0.0, 0.0, 0.0] for n in BALLS}

    for gy in range(GRID_H):
        py = int((gy + 0.5) * H / GRID_H)
        for gx in range(GRID_W):
            px  = int((gx + 0.5) * W / GRID_W)
            res = cam.pick_pixel(0, px, py)
            if res is None:
                continue                  # background (sky)
            name = name_by_eid.get(id(res.entity))
            if name is None:
                continue                  # entity outside the picking set (plane)
            draw.ellipse((px - dot_r, py - dot_r, px + dot_r, py + dot_r), fill=BALLS[name][2])
            wx, wy, wz = res.position
            h = hits[name]
            h[0] += 1; h[1] += wx; h[2] += wy; h[3] += wz

    Image.alpha_composite(base, overlay).convert("RGB").save(OUTPUT_PATH)

    print(f"Saved {OUTPUT_PATH}")
    for name, (n, sx, sy, sz) in hits.items():
        if n:
            print(f"  {name:<6} {int(n):>4} hits   mean pos=({sx/n:+.2f}, {sy/n:+.2f}, {sz/n:+.2f})")
        else:
            print(f"  {name:<6}    0 hits")


if __name__ == "__main__":
    main()

Run it:

uv run python examples/06_object_picking.py

The PNG is written to examples/out/06_object_picking.png. The Sphinx build copies it to _static/generated/examples/06_object_picking.png and embeds it at the top of this page, so the docs site always shows whatever the latest run produced.