AI servingDive 04 of 06

How a GPU runs a kernel, from threadIdx to DRAM

Follow a CUDA thread into a warp and onto an SM. Make branches diverge, scatter a memory request across sectors, and fix a transpose one bank conflict at a time.

Levels
9
Reading
42 min
Depth
Idea to Silicon
Published

A100 · one warp-wide float load

One float over.
One sector more.

Lane l reads byte 4 × (l + 1). The same 128 useful bytes now reach into a fifth sector.

32 threads describe their own loads. The memory system sees the addresses together. Change the addresses in the coalescing lab.

Level 01/09Idea

A million small jobs

You have two arrays of a million floats. For every element, multiply x by a scalar a, add y, and put the answer back in y. This is SAXPY: single-precision a times x plus y. Element 812 does not depend on element 811. There is no search, no shared counter and no reason to wait for the previous answer.

A CPU can run this loop well. It can vectorize, prefetch and keep several memory operations in flight. But the loop also has exactly the shape a GPU likes: a great many independent pieces of work, each doing the same small calculation on different addresses. The useful question is how to expose that independence without spending more on coordination than you save on execution.

A CPU core spends substantial hardware on making one instruction stream advance quickly. It predicts branches, discovers independent instructions, executes them out of order and tries to keep frequently used data close. Those mechanisms are especially valuable when a program follows pointers or makes decisions whose answers determine the next load. The microprocessor dive follows that machinery down to pipelines and caches.

A GPU makes a different bargain. It asks the program to provide many independent threads up front. When one group waits for data, another group can use the execution machinery. Each thread advances less impressively on its own; the chip can still finish more total work. NVIDIA describes this distinction as throughput versus single-thread execution speed in the CUDA C++ Programming Guide.

WorkloadWhat the machine needsLikely advantage
Follow one linked listThe next pointer depends on the previous loadCPU latency machinery
Update a million resident floatsIndependent addresses, repeated instructionsGPU throughput
Update twelve host floats onceVery little work; transfer and submission overheadKeep the work on the CPU
Many independent listsEnough independent searches to cover waitsMeasure; irregularity alone does not decide
Choose by the shape of the work. Independence and data placement matter alongside the number of arithmetic operations.

Our SAXPY has a simple budget. Per element it reads two four-byte floats, writes one four-byte float, and performs a multiply and an add. That is 12 bytes of logical array traffic for two floating-point operations, or one sixth of an operation per byte. A fused multiply-add still counts as two operations in that convention. Once there are enough threads to keep memory busy, reducing arithmetic instructions has little room to help.

Data placement can overturn the decision. If both arrays start on the CPU, you must send them to the device and bring the result back. Doing that for a single tiny update often loses. If the arrays already live on the GPU, or feed a chain of kernels, you can amortize the crossing. A useful optimization may be to keep intermediate data resident rather than make this particular kernel cleverer.

The GPU also has finite capacity for outstanding work. Launching a million threads does not mean a million are executing instructions at once. Some blocks are waiting to be admitted. Some resident warps are waiting for memory. Some lanes are inactive. Those distinctions are the route from a fast-looking program to a fast one.

We will follow this array update through that route, then deliberately break its friendly address pattern with a matrix transpose. Tensor cores and tiled matrix multiplication have their own path in the LLM inference dive. Here the work runs through ordinary CUDA instructions and the memory system.

A GPU needs independent work to hide waiting. Give it a large array, useful work per transfer, and addresses that cooperate.

Level 02/09Interface

Give every thread an address

CUDA lets you write the body of the array update as if one thread owns one element. A A device function launched over a grid of thread blocks. Each thread runs the body with its own built-in indices. is a function marked __global__. The host launches it over a grid of blocks. Every block contains the chosen number of threads, and every thread can inspect its position.

For a one-dimensional block, threadIdx.x starts at zero. blockIdx.x names the block, and blockDim.x tells you its size. Multiply the block number by the block size, then add the thread's local index. That gives each thread a different global array index.

i = blockIdx.x × blockDim.x + threadIdx.x

Choose 256 threads per block for our example, then round the grid size up: (n + threads - 1) / threads. The last block can extend past the array, so every thread checks i < n before touching memory. For the small fixed dimensions here, signed integer arithmetic is sufficient; very large arrays need a wide index type and overflow-safe launch arithmetic.

Blocks can also have two or three dimensions. Those dimensions help you express an image or volume, but the hardware still forms warps from the block's linear thread order. The x index varies fastest, then y, then z. For a two-dimensional block, the linear local index is threadIdx.x + blockDim.x * threadIdx.y. Warps never combine the tail of one block with the start of another.

Try the mapper on a 43 × 7 array. A 16 × 4 block contains 64 threads, or two warps on A100 and H100. Notice that one warp covers two 16-element rows. Change to a one-dimensional block and the array becomes a 43-element vector. Thread IDs describe the work partition; they do not name a permanent physical execution lane.

Cell labels are global coordinates. The first 32 cells belong to warp 0; the rest, marked with a bottom rule, belong to warp 1. A cross means out of bounds; brackets mark the selected thread.

Selected thread
Valid threads

A complete trip to the device

The kernel is only part of the program. This complete example allocates separate device arrays, copies the inputs, launches the update, waits for completion and checks the returned values. It uses ordinary host vectors and synchronous copies to keep the ownership and order visible.

saxpy.cu · complete host and device example
#include <cuda_runtime.h>
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <vector>

void check(cudaError_t result, const char* file, int line) {
  if (result != cudaSuccess) {
    std::fprintf(stderr, "%s:%d: %s\n", file, line,
                 cudaGetErrorString(result));
    std::exit(EXIT_FAILURE);
  }
}
#define CUDA_CHECK(call) check((call), __FILE__, __LINE__)

__global__ void saxpy(int n, float a, const float* x, float* y) {
  const int i = blockIdx.x * blockDim.x + threadIdx.x;
  if (i < n) {
    y[i] = a * x[i] + y[i];
  }
}

int main() {
  constexpr int n = 1 << 20;
  constexpr int threads = 256;
  const size_t bytes = static_cast<size_t>(n) * sizeof(float);
  std::vector<float> x(n, 2.0f);
  std::vector<float> y(n, 1.0f);
  float* dx = nullptr;
  float* dy = nullptr;
  CUDA_CHECK(cudaMalloc(reinterpret_cast<void**>(&dx), bytes));
  CUDA_CHECK(cudaMalloc(reinterpret_cast<void**>(&dy), bytes));
  CUDA_CHECK(cudaMemcpy(dx, x.data(), bytes, cudaMemcpyHostToDevice));
  CUDA_CHECK(cudaMemcpy(dy, y.data(), bytes, cudaMemcpyHostToDevice));

  const int blocks = (n + threads - 1) / threads;
  saxpy<<<blocks, threads>>>(n, 2.0f, dx, dy);
  CUDA_CHECK(cudaGetLastError());
  CUDA_CHECK(cudaDeviceSynchronize());
  CUDA_CHECK(cudaMemcpy(y.data(), dy, bytes, cudaMemcpyDeviceToHost));

  bool valid = true;
  for (float value : y) {
    if (std::fabs(value - 5.0f) > 1e-6f || !std::isfinite(value)) {
      valid = false;
      break;
    }
  }
  std::printf("SAXPY: %s\n", valid ? "passed" : "failed");
  CUDA_CHECK(cudaFree(dx));
  CUDA_CHECK(cudaFree(dy));
  return valid ? EXIT_SUCCESS : EXIT_FAILURE;
}

The launch's angle brackets contain the grid size and block size. The function arguments follow normally. CUDA copies the argument values into the launch; a pointer argument still needs to point at storage the device may access. A host vector's address does not become a device allocation merely because you pass it to a kernel.

There are two error checks around the launch because there are two occasions for failure. cudaGetLastError() checks the launch submission, including an invalid configuration. Execution can fail later, so cudaDeviceSynchronize() checks completion. The blocking result copy would also establish completion for this sequence, but the explicit synchronization makes the boundary easy to see.

Threads in a block can cooperate through shared memory and barriers. Blocks are designed to be independently executable: the runtime can run them in any order, concurrently or one after another. A kernel that spins until an unscheduled block produces a value can deadlock. Separate producer and consumer kernels in the same stream when you need a simple grid-wide phase boundary.

Streams express order

A stream is an ordered sequence of device work. Operations submitted to the same stream respect that order. Different streams can overlap when dependencies and device resources allow it. An event can express a dependency between streams, and cudaStreamSynchronize() lets the host wait for just one stream.

Async copies become useful when you pipeline chunks of a larger problem. For host-to-device copy overlap, pinned host memory and suitable copy-engine support matter. Submitting two operations to different streams does not guarantee overlap; default-stream semantics can also add ordering. Start from a correct dependency graph, then measure whether independent stages actually run together. The stream documentation spells out those ordering rules.

Check yourselfA block is 16 × 4 threads. Which threads make its first warp on A100?

Level 03/09Runtime

From source to resident blocks

The source describes logical threads. Before any of them can run, CUDA must provide instructions for a particular GPU. nvcc separates host and device compilation. The host portion becomes ordinary CPU code with runtime calls for launches. The device portion goes through NVIDIA's device toolchain.

NVIDIA's virtual instruction set and execution model. PTX is a compiler target, not the final machine instruction encoding of a GPU. sits between CUDA C++ and the GPU's native instructions. The assembler, ptxas, turns it into architecture-specific machine code, commonly called SASS, in a device binary. A fat binary can carry native images for selected targets along with PTX for forward compatibility.

If a suitable native image is present, the driver can load it. If it needs to use compatible PTX instead, it can compile that PTX for the installed GPU and cache the result. That JIT compilation can add first-use cost. PTX compatibility is directional and constrained by driver support; architecture-specific targets have additional restrictions. NVIDIA documents the selection process in binary compatibility and PTX compatibility.

Build an A100 image plus compute_80 PTX, then inspect native code
nvcc -O3 -std=c++17 -lineinfo \
  -gencode arch=compute_80,code=sm_80 \
  -gencode arch=compute_80,code=compute_80 \
  -Xptxas=-v saxpy.cu -o saxpy
./saxpy
cuobjdump --dump-sass saxpy

The example command includes native code for compute capability 8.0, the A100 target, and a PTX image. It asks the assembler to print resource use. Look for registers, shared memory and spills there. The C++ variable count is a poor substitute: optimization can eliminate values, keep intermediate results alive, or need extra registers for addressing.

A launch is a request to run a grid after its dependencies. At a high level, the runtime and driver arrange the arguments, device code and execution configuration, and submit work to the device. The exact queueing and command-processing implementation varies; it is not a CUDA programming guarantee. Context setup and code loading can also make the first launch unlike later ones.

Host threadkernel<<<grid, block>>>()Submit to a streamContinue on the CPU

Device work

Block 0Block 1Block 2Block 3Block 4Block 5
SM ABlocks 0, 3 residentBlock 4 admitted
SM BBlocks 1, 2 residentBlock 5 admitted
  1. The host submits the grid and argument values to a stream. Submission is normally asynchronous with respect to the host.
  2. After earlier dependencies in that stream complete, blocks become eligible for device execution.
  3. An illustrative placement: each admitted block reserves resources on one SM. Its warps can be interleaved with other resident warps.
  4. Finished blocks release resources. More blocks can be admitted; CUDA promises no particular block ordering or placement.
1 / 4

On the device, an admitted block occupies one A streaming multiprocessor: a GPU execution unit with warp scheduling, registers, arithmetic pipelines and on-chip memory resources., or SM. Its threads are partitioned into warps. The block keeps its assigned SM for its execution, rather than having individual warps migrate around to chase free arithmetic units. Several blocks can reside on one SM if their resource needs fit.

Admission and instruction issue are separate decisions. Admission decides whether a block can reserve room. A warp scheduler chooses instructions from eligible resident warps. All the warps in a resident block have state allocated, but they do not all issue in the same cycle. A warp waiting at a barrier is resident too.

The block is useful as both a cooperation boundary and an allocation boundary. Shared memory belongs to that block for its lifetime, and its warps keep their register state while other warps run. Switching between those resident warps does not require the kind of operating-system context switch that saves an entire CPU thread to memory.

You should not build an algorithm around an imagined block order. Even the same kernel on the same device can see different resource availability when other work is present. A grid larger than resident capacity naturally runs in waves, but the waves are a consequence of available resources rather than a grouping you can name in CUDA.

Our million-element SAXPY launches 4,096 blocks of 256 threads. Most of those blocks wait their turn. Their number gives the GPU a supply of work; the number resident on each SM determines how many warps its schedulers can choose among right now. That leads to the next question: what keeps a block out?

The grid is the work supply. Resident blocks are the work currently allocated. Eligible warps are the work that can issue next.

Level 04/09System

Keep something ready to run

A warp has 32 threads on the A100 (compute capability 8.0) and H100 (9.0). The fraction of an SM's supported resident warp slots occupied by active, allocated warps. A resident warp may still be stalled. is the number of resident warps divided by the device's maximum. It measures allocated parallelism, not arithmetic-unit utilization or progress.

Here are the limits for those two specific targets, from the CUDA 12.8 technical specifications and the Ampere and Hopper tuning guides. KiB means 1,024 bytes; NVIDIA's tables label these capacities KB.

LimitA100 · 8.0H100 · 9.0
Threads per warp3232
Threads per block1,0241,024
Resident threads per SM2,0482,048
Resident warps per SM6464
Resident blocks per SM3232
32-bit registers per SM65,53665,536
Shared memory per SM, maximum164 KiB228 KiB
Shared memory per block, opt-in maximum163 KiB227 KiB
A100 and H100 share several residency limits but have different shared-memory capacities. An Ampere consumer GPU can have different limits from A100.

Each block needs room in several resource pools at once. A block of 256 threads occupies eight warps. Even with abundant registers and shared memory, the 64-warp limit permits at most eight such blocks per SM. A block of 32 threads uses one warp, but the 32-block cap then prevents filling all 64 warp slots. Smaller blocks do not automatically mean more occupancy.

Registers can bind first. Sixty-four registers per thread times 256 threads is 16,384 registers of logical demand per block. Four such blocks already account for the entire 65,536-register file. That makes four an upper bound before accounting for allocation rules. A compiler decision that increases register demand can push a kernel across a residency threshold even when its source change looks tiny.

A block must fit every resource pool. The first full pool prevents another block from becoming resident.

The exact answer requires allocation rounding and placement constraints. Registers are allocated in hardware-dependent units, not as a perfectly divisible scalar pool. Shared-memory requests are rounded too, and the chosen shared/L1 carveout, static storage, dynamic storage and runtime reservation all matter. On H100, CUDA reserves 1 KiB of shared memory per block; using more than 48 KiB requires dynamic allocation and explicit opt-in as described in the Hopper shared-memory section.

Try those allocation rules below. Enter the compiler's registers per thread and the sum of static and dynamic shared memory per block. The calculator assumes a valid static allocation and dynamic-memory opt-in up to the per-block limit. It models ordinary blocks without clusters or extra named barriers.

Blocks per SM allowed by each resource

Warp slots
Registers
Shared memory
Block cap
Active blocks / SM
Active warps / SM
Occupancy
Effective carveout

CUDA 12.8.90 rules: round registers × 32 up to 256 per warp; apply four 16,384-register subpartitions and the rounded per-block launch check. Round total shared bytes + 1 KiB reservation up to 128 bytes. The carveout preference rounds up to a supported size and expands to admit one block. Warps round up to 32 threads; caps are 64 warps and 32 blocks per SM.

The calculator follows cuda_occupancy.h from the CUDA Runtime 12.8.90 archive. For the actual compiled kernel, compare its resource budget with CUDA's occupancy query, which also sees compiled attributes beyond these inputs. This fragment goes inside the complete SAXPY program after threads is declared. The final argument is dynamic shared memory per block, zero for SAXPY.

Ask CUDA for compiled-kernel residency
// Inside main(), using saxpy and CUDA_CHECK from the full example.
int device = 0;
cudaDeviceProp properties{};
cudaFuncAttributes attributes{};
CUDA_CHECK(cudaGetDevice(&device));
CUDA_CHECK(cudaGetDeviceProperties(&properties, device));
CUDA_CHECK(cudaFuncGetAttributes(&attributes, saxpy));
int activeBlocks = 0;
CUDA_CHECK(cudaOccupancyMaxActiveBlocksPerMultiprocessor(
  &activeBlocks, saxpy, threads, 0));
const int warpsPerBlock = (threads + properties.warpSize - 1)
  / properties.warpSize;
const int maxWarps = properties.maxThreadsPerMultiProcessor
  / properties.warpSize;
std::printf("cc %d.%d, %d registers/thread, %zu static shared bytes\n",
            properties.major, properties.minor, attributes.numRegs,
            attributes.sharedSizeBytes);
std::printf("%d blocks/SM, theoretical occupancy %.1f%%\n",
            activeBlocks, 100.0 * activeBlocks * warpsPerBlock / maxWarps);

How many ready warps are enough?

Occupancy helps only when the extra warps cover a wait that would otherwise leave execution capacity idle. Little's law relates the work in flight to the completion rate and time in the system. In a simplified scheduler with one dependent instruction in flight per warp, a latency of L cycles and a desired issue rate r instructions per cycle require about Lr independent warp instruction chains.

Nchains ≳ L × r

Read left to right. Each slot issues one instruction from the labelled warp, or stalls (·). Toy scheduler: one dependent instruction per warp, one issue per cycle.

Issue utilization
Empty issue slots

Start with four warps and an eight-cycle delay. The model issues four instructions, then runs out of ready warps. Increase the warp count to eight and it can issue every cycle: by the time it returns to the first warp, that warp is ready again. Each displayed slot is a real scheduling decision computed from the warps' next-ready times.

Real kernels have more choices. A warp can issue independent instructions while an earlier result is pending; this is instruction-level parallelism. Memory systems can have many requests in flight, and issue restrictions differ by instruction pipeline. The toy's cycles are deliberately adjustable, rather than a claim about measured A100 memory latency.

Higher occupancy can also hurt. Forcing fewer registers may introduce spills, creating more memory operations. A smaller shared tile may lose reuse. Once enough requests are in flight to saturate bandwidth, adding warps cannot create more bandwidth. Profile both the stalls and the resource tradeoff before deciding that an occupancy percentage is a performance target.

Check yourselfAn SM reports high occupancy but low issue utilization. What can explain it?

Level 05/09Machine

One instruction, an active mask

A CUDA thread has its own registers, indices and control flow. A warp groups 32 of those threads for instruction execution on our two target architectures. NVIDIA calls the model single instruction, multiple threads, or SIMT. The important extra object is an The set of lanes participating in the instruction currently being executed. A lane can be inactive because it took another branch, exited, or lies beyond a partial warp.: which lanes participate in this instruction?

In the middle of our SAXPY array, all lanes pass the bounds check and execute the update. In a partial final warp, some lanes fail it. Those lanes must not load an invalid address, while the valid lanes still perform their work. A short guarded operation may compile to predicated instructions, where a per-thread predicate suppresses effects, rather than a branch around a separate instruction region.

Now imagine a longer branch, with one calculation for nonnegative inputs and another for negative inputs. If all lanes agree, the warp follows one path. If they disagree, the machine executes instructions for the participating lanes of each path while the other lanes are inactive. Work from the two paths occupies different execution opportunities. There is no requirement that the true path run first.

Filled cells are active; a dash marks an inactive lane. The model serializes two paths and excludes branch and join instructions.

Lane efficiency
Path cost

The simulator groups lanes by a threshold so you can choose the branch split directly. With sixteen lanes taking each four-instruction path, eight warp instructions carry 128 useful lane-instructions out of 256 possible lane-instructions: 50% lane efficiency. Put all lanes on one path and there are four issued instructions at 100%. The model counts instructions, not execution time.

Unequal path lengths make the distribution matter. If one lane takes a long exceptional path, most lanes may sit inactive through it. Conversely, a branch that avoids an expensive calculation for every lane can save work even when some divergence remains. Grouping similar inputs can improve agreement, but reordering the inputs has its own cost and can change memory locality.

Reconvergence is where execution can again include lanes that went down different paths. The diagram uses a single common continuation after two serialized regions. Actual reconvergence instructions and scheduling are compiler and architecture details; you cannot treat this teaching sequence as a SASS trace.

Volta changed a correctness assumption

Volta, compute capability 7.0, introduced independent thread scheduling. The GPU tracks execution state per thread and can diverge and reconverge below warp granularity. Threads still execute in SIMT groups; the feature does not give every lane a dedicated instruction-issue engine. It does mean that old code relying on implicit warp lockstep can be wrong. The Volta tuning guide documents that change.

When lanes exchange data through shared memory, use a synchronization primitive with the right participants. __syncwarp(mask) provides a warp barrier and memory ordering among the named participating threads. A block-wide exchange usually needs __syncthreads(). A barrier inside a condition that only some required participants reach is a correctness bug, not a tuning opportunity.

Move a value without a shared array

A shuffle reads another lane's register value within the warp. __shfl_down_sync(mask, value, offset) supplies a value from a higher-numbered lane when that source is in range and participates. The source and destination registers remain per-thread; the instruction arranges the exchange.

The following kernel creates one partial sum per warp. Invalid input positions contribute zero but still participate. After five shuffle-and-add steps, lane zero holds the sum. A later reduction must combine those partials to produce one array-wide result. Floating-point addition is not associative, so a tree reduction can differ in low bits from a serial CPU sum.

A full-warp reduction · launch only complete warps
#include <cuda_runtime.h>

__device__ float warpSum(float value) {
  constexpr unsigned mask = 0xffffffffu;
  for (int offset = 16; offset > 0; offset /= 2) {
    value += __shfl_down_sync(mask, value, offset);
  }
  return value;
}

// Launch with a 1-D block whose size is a multiple of 32.
// partial has room for ceil(n / blockDim.x) * (blockDim.x / 32) floats.
__global__ void warpPartials(const float* input, float* partial, int n) {
  const int i = blockIdx.x * blockDim.x + threadIdx.x;
  const float value = i < n ? input[i] : 0.0f;
  const float sum = warpSum(value);
  const int lane = threadIdx.x % 32;
  if (lane == 0) {
    const int warp = (blockIdx.x * blockDim.x + threadIdx.x) / 32;
    partial[warp] = sum;
  }
}

A full warp reduces 32 input values

offset = 16
offset = 8
offset = 4
offset = 2
offset = 1

Each step adds a value from a lane that is offset positions higher. Lane 0 holds the complete sum after offsets 16, 8, 4, 2 and 1.

The shuffle offsets halve on each step. Useful partials combine 2, 4, 8, 16 and finally 32 original values in the lanes that feed lane zero.

Do not discover a branch's intended participants by casually calling __activemask() inside it. That reports the lanes active at that particular instruction, which need not be all the lanes you meant to coordinate. A ballot at an appropriate converged point can form a membership mask, but a reduction over an arbitrary sparse mask also needs an algorithm that respects which source lanes exist. The full-warp example avoids that extra problem.

SIMT combines per-thread state with grouped execution. Reason separately about which lanes perform work and which lanes must synchronize.

Level 06/09Machine

Thirty-two addresses become a request

The source says x[i] independently in every thread. For a warp-wide load, the memory system receives a set of participating lane addresses. Combining the lane addresses of a memory instruction into the aligned memory segments needed to service them. is the process of servicing that address set with as few memory segments as its layout permits.

For A100 and H100 global-memory coalescing, count aligned 32-byte sectors. NVIDIA's CUDA 12.8 Best Practices Guide gives this rule for compute capability 6.0 and later. A full warp loading adjacent floats from a sector-aligned base touches four sectors. Permuting those addresses among the lanes still touches the same four sectors: ascending lane order is convenient, but the address set decides the count.

A sector contains eight floats. Starting one float later shifts the 128-byte footprint from bytes 0–127 to bytes 4–131. Now it overlaps sectors 0 through 4: five sectors. The useful payload is still 128 bytes, while the sector coverage is 160 bytes. That is 80% useful-byte coverage for this individual request.

Cache lines and sectors are different units. The Nsight Compute profiling guide defines an L1 or L2 cache line as four 32-byte sectors, 128 bytes in total. A line tag can cover sectors without requiring every sector of that line to be fetched for every request. Four sectors are not four complete 128-byte lines.

Each cell shows lane, byte address, sector and line. Addresses are relative to a 128-byte-aligned base; all 32 lanes load once.

Sectors per warp load
Useful sector coverage

The lab computes each lane's byte address, then collects the sector IDs touched by every byte of the load. It also groups sectors into 128-byte lines. Offset and stride are measured in elements. Doubling the element size changes both the payload and the addresses, so a full warp of contiguous doubles naturally covers eight sectors, twice as many as floats.

Try stride two with floats. The warp touches eight sectors but uses only half the bytes in each. At stride eight, each lane needs a different sector: 32 sectors to obtain 128 distinct bytes, or 12.5% useful coverage. At stride zero, every lane requests the same word. The lab counts those four useful bytes once, rather than reporting 128 useful bytes and an impossible efficiency above 100%.

The sector count predicts pressure from an address pattern. Cache reuse can change how much reaches the next level. Adjacent misaligned warps may reuse a boundary sector, and a repeated load can hit in L1. Stores also have their own behavior. Follow the request through the cache hierarchy before translating sector coverage into off-chip bandwidth.

Layout can be an algorithmic choice

Suppose you store particles as structures containing position, velocity and other fields. A warp that reads only x positions may jump by the whole structure size between lanes. A structure-of-arrays layout puts all x positions together. That often improves useful-byte coverage for a field-wise kernel, although a kernel that consumes every field may prefer a different arrangement.

A row-major image has the same issue in two dimensions. Neighbouring x coordinates are adjacent in memory; neighbouring y coordinates are separated by the row pitch. If a block is only sixteen threads wide, a warp spans two rows. Whether those two row fragments occupy four or more sectors depends on their alignment and pitch. A rectangular block shape is not itself a coalescing guarantee.

The path beyond a request

Global loads normally travel through the SM's L1/texture path, the chip's shared L2 and ultimately device DRAM, subject to cache policy and instruction type. A100 uses HBM2, with HBM2e in its 80 GB variant; H100 uses HBM variants depending on the product. We need no particular bandwidth figure to understand the address mapping. An L2 hit avoids the DRAM trip but still consumes resources in the hierarchy.

Registers are the closest per-thread storage. When register allocation cannot keep a value there, the compiler can spill it to local memory. Local means private to a thread, not a small on-chip scratchpad: its backing store is device memory and its accesses use the cache hierarchy. Arrays with dynamic indexing and register pressure are common reasons to inspect the generated local loads and stores.

The logical 12 bytes per SAXPY element assume each array value is moved once through the level you are accounting for. Cache hits, instruction choices and memory-system effects can change physical traffic. Still, the source-level budget is useful: optimize address coverage first, then use profiler counters to locate the actual traffic. The memory dive continues below the controller into DRAM banks, rows and cells.

Check yourselfA warp loads 32 adjacent floats starting at byte 4 of an aligned allocation. How many 32-byte sectors does it touch?

Level 07/09Machine

Thirty-two banks, one address puzzle

Sometimes good global addresses require threads to exchange values. Shared memory gives a block a programmer-managed on-chip array. You decide what to place there and when threads can use it. It is especially useful when it lets many threads reuse a load or rearrange data before a coalesced store.

For the ordinary 32-bit accesses used here, A100 and H100 shared memory has 32 banks, with successive four-byte words assigned to successive banks. The bank index is the word address modulo 32. This is the organization described for the relevant compute capabilities in the Programming Guide; the Best Practices Guide explains conflict handling.

bank = ⌊byte address / 4⌋ mod 32

A warp reading consecutive float words sends one word to each bank. Those banks can serve the words in parallel. A warp reading words 0, 32, 64 and so on sends distinct words to bank zero. The bank cannot serve all those distinct words together, so the access is split into conflict-free service rounds.

That is a bank conflict. Its degree is the number of distinct words needed from the busiest bank in this one-word-per-lane model. A degree of two needs two rounds, or one extra round beyond the conflict-free case. Wider accesses can require multiple transactions even without a conflict, so the lab stays with scalar 32-bit words.

Each lane reads one 32-bit word on A100/H100. Cells show lane, word and bank; distinct words in the same bank need separate service rounds.

Busiest bank
Extra service rounds

The broadcast preset is an important exception to a naive counting rule. Thirty-two lanes reading the same shared word do not create a 32-way conflict. The hardware can broadcast that word. The lab therefore groups distinct word addresses in each bank, rather than counting the number of lanes assigned to the bank.

A two-dimensional C array makes the layout problem easy to see. In float tile[32][32], the address of tile[row][col] advances by 32 words when row increases. Reading one column across a warp puts every lane in the same bank. Reading one row distributes the lanes over all banks.

Change the declaration to float tile[32][33]. The extra column changes the pitch to 33 words. For a fixed column, lane l now reaches word 33l plus the column offset. Modulo 32, that advances one bank per lane. The transpose still moves the same 32 × 32 useful values; the unused column changes their physical layout.

The extra storage is 32 words, or 128 bytes. It is small here, but it belongs in your resource accounting. An optimization that pads a much larger array can change shared-memory residency. The right question is whether the saved bank serialization outweighs any cost in resource allocation and instructions for the actual kernel.

Synchronization establishes the exchange

A shared-memory write by one thread does not by itself tell another thread when to read it. Place a block barrier between a cooperative load and a cross-warp read. Keep that barrier outside the bounds check: even threads that own no valid input element can still be needed as participants in the block.

At a partial tile edge, you can guard loads and stores separately while all threads reach the barrier. The important proof is that every shared location read by a valid output thread was written by a valid input thread. Initializing the entire tile to zero can conceal a missing proof and silently produce the wrong edge values. Derive the source coordinate for each output instead.

Shared memory and L1 also share physical capacity on these architectures. Choosing a larger shared-memory carveout can leave less room for hardware-managed caching. Avoid staging data merely because shared memory sounds faster. If each element is used once with already coalesced access, an extra load, store and barrier may add cost without reducing global traffic.

Global memory rewards compact address coverage. Shared memory rewards distinct bank assignments. The layouts that achieve those goals can differ.

Level 08/09Machine

Turn the addresses, not the answer

Now change the job. Transpose a row-major matrix: input element at row y, column x becomes output element at row x, column y. Use separate arrays so an early store cannot overwrite an input another thread still needs. This is a data movement problem with almost no arithmetic to hide mistakes in the address layout.

Consider a 1,024 × 1,024 matrix of floats. Use a block of 32 × 8 threads to process a 32 × 32 tile, with each thread handling four elements eight rows apart. For an interior warp, x varies across the row. The input load is adjacent: four sectors. The naive output store writes one value per output row, so neighbouring lanes are 1,024 floats apart.

Those stores occupy 32 sectors for 128 bytes of useful values. They also touch 32 cache lines. The address calculation shows why these stores waste sector bandwidth. Reducing the integer arithmetic will not fix that spread.

Naive · interior warp · 1,024-column input

Load sectors
4
Store sectors
32
Shared read rounds
No shared read

Naive store: byte address = lane × 1,024 × 4.

Shared tile · interior warp · 1,024-column input

Load sectors
4
Store sectors
4
Shared read rounds
32

Column read: word = lane × 32; bank = word modulo 32.

Padded tile · interior warp · 1,024-column input

Load sectors
4
Store sectors
4
Shared read rounds
1

Column read: word = lane × 33; bank = word modulo 32.

  1. Read rows, write columns. One source load uses 4 sectors. The destination store touches 32 sectors because neighbouring lanes write different rows.
  2. Exchange through shared memory. Both global accesses now use 4 sectors each, but reading a column of a 32 × 32 float tile needs 32 shared-bank service rounds.
  3. Add one column of padding. A 33-word pitch sends lane l to bank l. Global accesses stay coalesced; the shared column read needs one service round.
1 / 3

Change which thread owns the store

The first optimization loads the input tile into shared memory in row order. After a barrier, threads read the shared tile with its indices swapped. The output coordinates swap the block indices too, so the stores now run along an output row. A value loaded by one lane can be stored by another lane. The required matrix permutation stays the same.

That turns a global layout problem into a shared layout problem. Both the global load and store use four sectors per full interior warp instruction. But the shared read is a column read from a 32-word-pitch array, exactly the 32-way conflict in the bank lab. We fixed the off-chip address pattern and exposed a new local bottleneck.

Padding the shared tile removes that conflict. Use transposeShared<0> for the unpadded version and transposeShared<1> for the padded one. Both execute the same index transformation and the same barrier. The pitch is the consequential difference.

Transpose kernels · fixed 32 × 8 block · rectangular edges handled
#include <cuda_runtime.h>
#include <cstddef>

__global__ void transposeNaive(const float* input, float* output,
                               int width, int height) {
  const int x = blockIdx.x * 32 + threadIdx.x;
  const int y = blockIdx.y * 32 + threadIdx.y;
  for (int j = 0; j < 32; j += 8) {
    if (x < width && y + j < height) {
      output[static_cast<size_t>(x) * height + y + j] =
        input[static_cast<size_t>(y + j) * width + x];
    }
  }
}

template<int Padding>
__global__ void transposeShared(const float* input, float* output,
                                int width, int height) {
  __shared__ float tile[32][32 + Padding];
  int x = blockIdx.x * 32 + threadIdx.x;
  int y = blockIdx.y * 32 + threadIdx.y;
  for (int j = 0; j < 32; j += 8) {
    if (x < width && y + j < height) {
      tile[threadIdx.y + j][threadIdx.x] =
        input[static_cast<size_t>(y + j) * width + x];
    }
  }
  __syncthreads();
  x = blockIdx.y * 32 + threadIdx.x;
  y = blockIdx.x * 32 + threadIdx.y;
  for (int j = 0; j < 32; j += 8) {
    if (x < height && y + j < width) {
      output[static_cast<size_t>(y + j) * height + x] =
        tile[threadIdx.x][threadIdx.y + j];
    }
  }
}

// input and output must be separate width * height allocations.
// Call with positive dimensions whose rounded-up coordinates fit in int.
void launchTranspose(const float* input, float* output,
                     int width, int height, cudaStream_t stream) {
  const dim3 block(32, 8);
  const dim3 grid((width - 1) / 32 + 1, (height - 1) / 32 + 1);
  transposeShared<1><<<grid, block, 0, stream>>>(
    input, output, width, height);
}

The launch helper shows the padded variant. As in SAXPY, check cudaGetLastError() after submission and check the synchronization or completion operation before consuming the result. The helper leaves that completion policy to its caller so it can participate in a stream of work.

Follow one value to check the indexing. Thread (tx, ty) loads input row 32by + ty + j, column 32bx + tx, into the same local tile coordinates. During the store phase, reading tile[threadIdx.x][threadIdx.y + j] exchanges those local coordinates. Swapping the block indices exchanges the tile's global location too.

For a rectangular edge tile, the output guard tests output column against the original height and output row against the original width. Those conditions are also the validity conditions for the input value at the swapped local coordinates. Every valid read therefore has a corresponding initialized shared word, even though invalid input positions were never loaded.

Account at a consistent scale

The stepper counts one warp's one load and store instruction on an interior tile, not a whole block. Our 256 threads execute four load-loop iterations and four store-loop iterations. Across eight warps and four iterations, the ideal full-tile sector counts are 128 load sectors in every variant, 1,024 store sectors for the naive version, and 128 store sectors for either shared version.

Those are per-instruction sector totals. Some sectors of the naive stores may be revisited by other instructions, so the total is not a claim that 1,024 unique sectors reach DRAM. Padding changes the shared read's service rounds from 32 to one in the scalar model. Wall-clock performance still depends on caches, instruction throughput, barriers, occupancy and the matrix dimensions.

This is the same kind of reasoning you used for SAXPY, with one extra ownership exchange. First determine what each lane must load and store. Group those addresses into global sectors. If you rearrange through shared memory, calculate the bank mapping and prove the synchronization. Then measure the remaining bottleneck.

For a real application, ask whether you can avoid the transpose altogether. A later kernel might consume the original layout with altered indexing, or the transpose might be fused into a producer or consumer. That can eliminate an entire pass over the arrays. The padded kernel is a useful building block when the rearrangement itself is necessary.

Check yourselfWhy does a 33-column shared tile help this transpose?

Level 09/09Silicon

Where the threads actually live

We have treated the SM as a place that holds blocks and issues warp instructions. Inside it are several execution pipelines and a large amount of live thread state. The A100 SM is organized into four processing partitions, each with a warp scheduler. NVIDIA's A100 architecture whitepaper, pages 20–22, shows the partition structure and the arithmetic, load/store and special-function units.

On this architecture, a scheduler can issue an instruction from an eligible warp each cycle under the appropriate pipeline conditions. A warp's instruction may take multiple cycles to execute across the available functional units. Warp width, arithmetic-lane count and instruction latency are different quantities; “32 threads in a warp” does not mean every kind of warp instruction completes in one clock.

A functional view of an A100 SM and its memory path. The register state lets many warps remain resident while a scheduler chooses an eligible instruction.

The FP32 pipelines perform ordinary floating-point arithmetic such as our array update. INT32 pipelines support integer arithmetic, including part of address calculation. Load/store units issue memory operations and move values between the memory hierarchy and register state. Special-function units handle selected operations such as approximate transcendental functions. Different instruction mixes place different demands on those units.

The register file is central to the bargain. A100 has 65,536 32-bit registers per SM, 256 KiB of raw register storage. Keeping many warps' values on chip lets the scheduler switch to another ready warp without writing the old warp's state to DRAM. Those registers are also an expensive resource: storage is only part of the cost, because the execution pipelines need bandwidth to read and write operands.

Partitioning helps organize that operand traffic and the execution machinery. It also means aggregate register capacity is not the whole allocation story. That is one reason the occupancy query uses architecture-specific rules. A scalar division of total registers by registers per block captures a budget, but it can miss how the machine allocates that budget.

A cache attacks latency by making some accesses finish sooner. Many resident warps attack latency by making another independent instruction available while an access is outstanding. GPUs use both. The balance is a design choice: more cache can reduce traffic for reuse-heavy code, while more register capacity can preserve more independent work. Neither resource helps a workload that has no corresponding reuse or independence.

Our two examples expose the distinction. SAXPY streams through values once, so a larger cache cannot manufacture reuse. It needs enough requests in flight and efficient coverage of each sector. Transpose benefits from a small programmer-managed staging area because it changes the relationship between the lanes and their destination addresses. Shared memory makes that exchange explicit and bounded.

Throughput has a power budget

Moving operands, accessing banks, reading register ports and charging long wires all consume energy. Scattered loads activate memory machinery for bytes you never use. Bank conflicts spend extra service rounds on the same logical operation. A layout improvement can reduce that internal work even when the number of useful array elements and arithmetic operations stays fixed.

Adding execution units alone would not solve these problems. They need operands, issue bandwidth and enough independent work. Larger caches and register files consume area and power too. The architecture distributes a finite transistor and energy budget across execution, state and movement; the programmer's layout determines how productively that machinery is used.

Tensor cores are a separate specialized execution path. Their role in matrix operations, and the arithmetic-intensity argument behind the roofline, are covered in LLM inference. For this dive, the ordinary warp instruction already explains the important relationship: one instruction selects participating lanes, and those lanes create an address pattern that the memory system must serve.

Trace our array update one last time. The host submits a grid. Blocks become resident where resources fit. Each warp's bounds check selects valid lanes. Its loads turn 32 logical addresses into sectors, and its arithmetic consumes register operands when the dependencies are ready. If a request misses the caches, it continues to DRAM while another eligible warp can issue. The final store completes each thread's part of the array.

That gives you a practical order for investigation: check correctness and dependencies, confirm the compiled resource use, inspect eligible versus stalled warps, calculate sector and bank patterns, then compare profiler counters with that explanation. A good kernel is one whose work, state and addresses match the machine's opportunities.

The GPU's advantage comes from keeping many useful instruction chains in flight and feeding them efficiently. Your thread layout connects those two jobs.

End of GPU kernels · 9 levels, Idea to Silicon

Where to next

Practical notes

  1. Note3 min read

    Measure LLM tokens per second without mixing workloads

    Report prefill throughput, decode throughput and time to first token separately. A bandwidth estimate gives an optimistic bound for small-batch dense decoding.

Browse all 34 divesAll topics