Kernel caching: whole-kernel vs per-offload#

Quadrants caches compiled kernels so that re-running an unchanged kernel does not recompile it. There are two granularities at which this caching happens. Both produce identical results; they differ only in how much has to be recompiled after you edit a kernel.

Whole-kernel caching. The whole @qd.kernel is compiled and cached as a single unit. Editing any part of the kernel invalidates that entry, so the entire kernel recompiles.
Per-offload caching (default on CUDA/AMD GPUs). The kernel is split into its top-level constructs (independent top-level loops or serial runs) and each is compiled in isolation, so the compiled result is cached per offloaded task. Editing one construct recompiles only that construct; the others are reused.

Which is better#

	Advantage
Per-offload	Fast incremental recompiles. Editing one task in a large kernel (for example a big `qd.graph.do_while` loop) reuses the other tasks’ compiled code instead of recompiling the whole kernel. This is the win for large kernels you iterate on.
Whole-kernel	Simplest and universal. It is always correct and is the only option for kernels the split cannot compile construct-by-construct equivalently.

Per-offload caching is on by default on CUDA and AMD GPUs with the offline cache enabled (itself the default). On other backends (CPU, Metal, Vulkan), or with the offline cache disabled, Quadrants uses whole-kernel caching. Beyond that, Quadrants falls back to the whole-kernel path automatically and transparently whenever the split would not be equivalent, or cannot be verified as safe for a given launch. You never choose between them and the results are identical either way; the only observable difference is compile time.

When it falls back to whole-kernel#

Some fallbacks are decided once, at compile time (the kernel never uses the split). Others are decided per launch (the kernel is split, but a particular call runs the whole-kernel variant instead).

Compile-time fallback (this kernel is never split; it always uses whole-kernel caching):

Condition	What it means
Autodiff (gradient) kernels	The backward pass links the constructs together, so they cannot be compiled one at a time.
One construct produces a value a later construct consumes	The later construct cannot be rebuilt on its own without redoing the earlier one.
A construct re-reads array data an earlier construct overwrote	Compiling it in isolation would observe the new value instead of the original. This includes clamped indexing (`boundary="clamp"`), where an out-of-range index can land on an element that was written.
Two array arguments where safety would depend on them not sharing memory, and one is a gradient	Whether two gradient buffers overlap cannot be verified even at launch, so Quadrants refuses to split rather than risk a wrong result.
A side effect, random draw, or volatile read would be repeated	Recomputing it inside another construct would change what the kernel does.
Kernels that iterate over an unstructured mesh	Mesh iteration is not expressed as independent top-level constructs.
A region explicitly scheduled to run concurrently	Concurrent constructs share one scratch buffer, so isolating them would corrupt it.
A diagnostic mode is enabled (line coverage, or a compiler debug dump, see optimization passes)	These need the whole-kernel form to keep their diagnostics meaningful.

Runtime fallback (a per-launch decision for a split kernel):

Condition	Reason
Two parameters the split assumed disjoint are bound to the same buffer	Recomputing a read across the aliased write could change the result, so the call uses the whole-kernel variant.
A parameter’s backing buffer cannot be read to confirm disjointness	For example a raw NumPy array or PyTorch tensor passed directly. The call conservatively uses the whole-kernel variant. Passing `qd.ndarray` / `qd.Tensor` objects avoids this.

Each compile-time fallback logs a one-line debug-level message naming the kernel and the condition, so raising Quadrants’ log level to debug tells you why a given kernel stayed whole-kernel. A runtime fallback instead emits a one-time [PER_OFFLOAD][FALLBACK] warning naming the kernel (at the standard warn level, so it is on by default and suppressible through the logging level).

Compile-time fallback examples#

Each kernel below trips one of the compile-time fallbacks above and stays whole-kernel. You can confirm it with per_offload_cache_observations (frontend_constructs_total is -1) or by reading the debug-level log line.

A value carried between constructs#

The first loop accumulates total; the second loop reads it. Because total exists only once the first loop has run, the split cannot compile the second loop on its own.

@qd.kernel
def running_total(x: qd.types.NDArray[qd.f32, 1], out: qd.types.NDArray[qd.f32, 1]):
    total = 0.0
    for i in range(x.shape[0]):    # construct 1: build up a running total
        total += x[i]
    for i in range(out.shape[0]):  # construct 2: reuse it
        out[i] = total

Supporting a split for cases like this is possible but involved; see Advanced: making these cases splittable.

Re-reading overwritten data#

first snapshots a[0], then the first loop overwrites a. Recomputing that snapshot inside the second loop would read the overwritten value instead of the original.

@qd.kernel
def stale_snapshot(a: qd.types.NDArray[qd.f32, 1], out: qd.types.NDArray[qd.f32, 1]):
    first = a[0]                   # snapshot a[0]
    for i in range(a.shape[0]):    # construct 1: overwrite a
        a[i] = 0.0
    for i in range(out.shape[0]):  # construct 2: reuse the snapshot
        out[i] = first

This shares the same fix as the previous case; see Advanced: making these cases splittable.

Possibly-aliasing gradient arguments#

a and b are different parameters, so the split would treat their buffers as disjoint and reuse that assumption while recomputing. But one access is a gradient buffer, and a caller can make a and b (or a value buffer and its own .grad buffer) share memory, which cannot be verified even at launch, so Quadrants refuses to split.

@qd.kernel
def grad_snapshot(a: qd.types.ndarray(dtype=qd.f32, ndim=1, needs_grad=True),
                  b: qd.types.ndarray(dtype=qd.f32, ndim=1, needs_grad=True),
                  out: qd.types.ndarray(dtype=qd.f32, ndim=1)):
    g = a.grad[0]                  # snapshot a's gradient
    for i in range(b.shape[0]):    # construct 1: write b's gradient (a different argument)
        b.grad[i] = 0.0
    for i in range(out.shape[0]):  # construct 2: reuse the snapshot
        out[i] = g

This one is fixable too, though for a narrower case; see Advanced: making these cases splittable.

Advanced: making these cases splittable#

This section is internal background for the curious; you never need it to use Quadrants.

To support a value carried between constructs, the split would need to let one construct hand the value to the next through a small buffer that persists between them, instead of rebuilding each construct from the kernel’s arguments alone. That buffer becomes a shared interface the cache has to track (its type and layout) so that editing one construct can’t silently break the other. It’s a real amount of machinery for an uncommon case, so today Quadrants takes the simpler, always-correct route and caches the whole kernel.

The same persisted-buffer idea covers re-reading overwritten data: capture the snapshot before the overwrite and store it for the second loop, instead of rebuilding it from the arguments (where it would re-read the overwritten a).

Possibly-aliasing gradient arguments is fixable too, but for a fairly narrow case (a non-autodiff kernel that manually reads two .grad buffers as plain arrays; true autodiff kernels are already excluded from the split). Today it refuses to split rather than assuming disjointness it can’t verify.

Inspecting the split#

See Inspecting the split for the per_offload_cache_observations attribute. frontend_constructs_total == -1 means the split did not run for that compile: either it took a fallback above, or the kernel was served from the offline cache or fastcache, so no compilation happened at all.