Advanced: Building the CUDA graph conditional fatbin#

The qd.graph.do_while feature uses a tiny CUDA kernel that calls cudaGraphSetConditional (a device-side function from NVIDIA’s libcudadevrt.a) to control CUDA graph conditional while nodes. These conditional nodes require SM 9.0+ (Hopper or later); on older GPUs, qd.graph.do_while falls back to a host-side loop automatically.

There are three distinct phases:

  1. Fatbin generation (rare, manual) — A developer runs scripts/build_condition_kernel_fatbin.py (and the sibling build_checkpoint_gate_fatbin.py / build_checkpoint_yield_check_fatbin.py), which compiles the kernel and device-links it with libcudadevrt.a to resolve cudaGraphSetConditional. Each script emits one fatbin per build toolkit (see Prerequisites), all committed to git in a single C header. Requires the CUDA toolkits listed below.

  2. Quadrants build (CI / developers) — The C header is #included as plain byte arrays. No CUDA toolkit needed.

  3. Runtime (end users) — The fatbins are loaded via cuModuleLoadData. No CUDA toolkit needed.

This page documents phase 1: regenerating the pre-built fatbins.

When to regenerate#

You only need to regenerate the fatbin if:

  • The condition kernel source (quadrants/runtime/cuda/graph_do_while_cond.cu) changes.

  • You need to add support for a new SM architecture.

Prerequisites#

Each SM architecture is compiled with the oldest suitable CUDA toolkit, and the per-toolkit results are bundled into one header. This is what keeps each cubin loadable on the widest range of drivers: a SASS cubin can only be loaded by a driver whose CUDA version is >= the toolkit that produced it, so building e.g. sm_120 (RTX 5090) with a newer toolkit than necessary would make it reject older drivers with CUDA_ERROR_INVALID_IMAGE. The per-arch toolkit requirement lives in the SM_TOOLKIT mapping in scripts/_fatbin_common.py, where each SM maps to a version spec — ==X.Y (exact) or >=X.Y (minimum):

  • ==12.8 (matched exactly) — every architecture except sm_110 (pre-Hopper sm_60 / sm_70 / sm_80, Hopper sm_90, Blackwell sm_100 / sm_120). 12.8 is the oldest toolkit that covers all shipping Blackwell parts, and is the wide-compatibility anchor, so it is matched exactly rather than “or newer”.

  • >=13.0 — sm_110 (Thor) only; earlier toolkits fail with Unsupported gpu architecture 'compute_110'. This is a minimum: the oldest available toolkit >= 13.0 is used (e.g. 13.1 is fine). There is no old-driver constraint pulling it lower, since Thor is new hardware.

All required toolkits must be installed; the scripts raise MissingToolkitError (listing what is missing and what was discovered) before compiling anything if any is absent. Toolkits are auto-discovered from /usr/local/cuda-*/bin/nvcc, nvcc on PATH, and CUDA_HOME / CUDA_PATH. To point at toolkits in non-standard locations, set QUADRANTS_NVCC_CANDIDATES to an os.pathsep-separated list of nvcc binaries and/or CUDA root directories, e.g.:

export QUADRANTS_NVCC_CANDIDATES=/opt/cuda-12.8:/opt/cuda-13.1/bin/nvcc

Regenerating#

Run the script from the repo root:

python scripts/build_condition_kernel_fatbin.py

This will, for each build toolkit:

  1. Compile quadrants/runtime/cuda/graph_do_while_cond.cu with relocatable device code for that toolkit’s target SM architectures.

  2. Device-link the result with libcudadevrt.a to resolve the cudaGraphSetConditional extern.

It then writes all the per-toolkit fatbins as C byte arrays (plus a small lookup table) to quadrants/runtime/cuda/graph_do_while_cond_fatbin.h.

Regenerate the checkpoint kernels the same way with scripts/build_checkpoint_gate_fatbin.py and scripts/build_checkpoint_yield_check_fatbin.py. The generator emits a compact byte-array layout; run pre-commit run -a (clang-format) on the regenerated headers so they match the committed formatting before committing them. Quadrants must be rebuilt to pick up the new fatbins.

Verifying the committed fatbins#

tests/python/test_fatbin_arch_coverage.py holds an explicit, hand-maintained EXPECTED_LAYOUT for each committed *_fatbin.h — the architectures and the CUDA toolkit expected in every blob (e.g. sm_90 / sm_100 / sm_120 built with CUDA 12.8 in blob 0 and sm_110 built with CUDA 13.0+ in blob 1 for the condition kernel). It deliberately does not re-derive this from scripts/_fatbin_common.py, so a regression in the SM_TOOLKIT mapping is caught rather than silently baked into a new header. For every embedded cubin it checks both the SASS arch and the build toolkit that cuobjdump reports, so it directly asserts the load-bearing fact behind #2942: the sm_120 (RTX 5090) cubin is built with the wide-compatibility CUDA 12.8, not a newer toolkit that would reintroduce CUDA_ERROR_INVALID_IMAGE on 570-series drivers. It also drift-guards each SM_VERSIONS list against the arch set actually baked into the checked-in header.

The test shells out to cuobjdump, so it is skipped unless cuobjdump is on PATH; the toolkit providing it must also be at least as new as the newest one used to build the fatbins (CUDA 13.0+, for sm_110), otherwise the affected header is skipped rather than failed. Run it after regenerating:

# Make a recent CUDA toolkit's cuobjdump visible on PATH, e.g.:
export PATH=/usr/local/cuda/bin:$PATH
python -m pytest tests/python/test_fatbin_arch_coverage.py -v

Adding a new SM architecture#

Add the new SM version number to the SM_VERSIONS list in the relevant scripts/build_*_fatbin.py, and add an entry for it to SM_TOOLKIT in scripts/_fatbin_common.py: use "==X.Y" to pin it to an exact toolkit (the oldest one that covers it, as for the ==12.8 archs) or ">=X.Y" for a minimum (as done for 110: ">=13.0"). Then make sure a matching toolkit is installed and regenerate. (Every SM in SM_VERSIONS must have an SM_TOOLKIT entry, or the scripts raise KeyError.)

Files#

File

Purpose

scripts/_fatbin_common.py

Shared multi-toolkit build + header generation; arch → toolkit mapping

quadrants/runtime/cuda/graph_do_while_cond.cu

CUDA C source for the condition kernel

scripts/build_condition_kernel_fatbin.py

Regeneration script (condition kernel)

quadrants/runtime/cuda/graph_do_while_cond_fatbin.h

Generated C header (checked into git)

scripts/build_checkpoint_gate_fatbin.py, scripts/build_checkpoint_yield_check_fatbin.py

Regeneration scripts (checkpoint kernels)

quadrants/runtime/cuda/checkpoint_gate_fatbin.h, quadrants/runtime/cuda/checkpoint_yield_check_fatbin.h

Generated C headers (checked into git)

tests/python/test_fatbin_arch_coverage.py

Regression test: committed headers match an explicit per-blob arch + toolkit layout (needs cuobjdump)

How it’s used at runtime#

GraphManager::ensure_condition_kernel_loaded() in quadrants/runtime/cuda/graph_manager.cpp loads the fatbins via GraphManager::load_first_matching_fatbin(), which tries each bundled per-toolkit fatbin with cuModuleLoadData and keeps the first that loads (a device only loads the fatbin containing SASS for its SM). If none contain SASS for the current GPU’s SM architecture, loading fails with a clear error pointing to the relevant regeneration script. The checkpoint kernels load the same way from graph_manager_checkpoint.cpp.