Why Do GPU Memory Accesses Need to Be Coalesced?
Because global memory is read in fixed 32-byte pieces, not one word at a time. When the 32 threads of a warp read 32 neighboring 4-byte words, the hardware combines (coalesces) them into four 32-byte transactions and uses every byte. When the threads read scattered addresses, each may need its own transaction, so the GPU moves up to eight times more data than the program uses, and a memory-bound kernel slows down by about the same factor.
What you'll learn
- The GPU memory hierarchy, from registers to global memory
- How a warp's loads turn into 32-byte transactions
- The efficiency of contiguous, misaligned and strided access, worked through
- Layout changes that make accesses coalesce
Conventions and assumptions
NVIDIA GPUs of compute capability 6.0 or later, where the CUDA C++ Best Practices Guide states that a warp's accesses coalesce into as many 32-byte transactions as are needed to serve all its threads. Each thread reads one 4-byte float.
The memory hierarchy
| Memory | Where | Shared by | Speed |
|---|---|---|---|
| Registers | On the SM | One thread | Fastest |
| Shared memory and L1 | On the SM | A thread block (shared) or the SM (L1) | Very fast, a few dozen cycles |
| L2 cache | On the chip | All SMs | Slower |
| Global memory (HBM or GDDR) | Off the chip | The whole GPU | Hundreds of cycles, but very high bandwidth |
Worked example: four access patterns
| Pattern (lane k reads) | 32-byte sectors | Bytes moved | Efficiency |
|---|---|---|---|
| a[k], aligned | 4 | 128 | 100% |
| a[k + 1], off by one float | 5 | 160 | 80% |
| a[2k], stride 2 | 8 | 256 | 50% |
| a[32k], stride 32 | 32 | 1,024 | 12.5% |
In each case the warp uses 128 bytes, 32 floats. In the aligned case those are exactly four sectors. Shifted by one float, they straddle a fifth sector. With stride 2 the warp's addresses span 256 bytes, so eight sectors move and half of each is wasted. With stride 32 every lane lands in its own 128-byte region: 32 sectors, of which one-eighth is used.
Making accesses coalesce
- Map consecutive threads to consecutive addresses: index arrays with threadIdx.x in the fastest-changing position.
- Prefer a structure of arrays (x[], y[], z[]) to an array of structures ({x, y, z}[]), so a warp reading x touches only x.
- For a matrix stored by rows, have neighboring threads read neighboring columns of the same row.
- When a strided pattern can't be avoided, as in a transpose, load a tile into shared memory with coalesced reads and rearrange it there.
Common mistakes
- Indexing a row-major matrix as a[threadIdx.x * width + j], so neighboring threads are a whole row apart.
- Using an array of structures and reading one field, which strides by the size of the structure.
- Assuming the L1 cache rescues a strided pattern. The transaction size stays 32 bytes either way.
- Judging a kernel by FLOP/s when it is limited by memory bandwidth.
Check your understanding
- Q1. A warp reads 32 consecutive 8-byte doubles, aligned. How many sectors? Answer: 8, since 256 bytes / 32 = 8, at 100% efficiency.
- Q2. Lanes read a[4k] as floats. Sectors and efficiency? Answer: 16 sectors, 512 bytes moved for 128 used, 25%.
- Q3. Why does an array of structures often coalesce badly? Answer: a warp reading one field skips over the other fields, which is a strided access.
Related
Sources
Last verified
The 32-byte transaction rule checked against the CUDA C++ Best Practices Guide, section 10.2.1. Every number in this post is computed in the site's tests (lib/gpuNumbers.test.js). Checked on 14 November 2026.