Why Do GPU Memory Accesses Need to Be Coalesced?

Because global memory is read in fixed 32-byte pieces, not one word at a time. When the 32 threads of a warp read 32 neighboring 4-byte words, the hardware combines (coalesces) them into four 32-byte transactions and uses every byte. When the threads read scattered addresses, each may need its own transaction, so the GPU moves up to eight times more data than the program uses, and a memory-bound kernel slows down by about the same factor.

What you'll learn

  • The GPU memory hierarchy, from registers to global memory
  • How a warp's loads turn into 32-byte transactions
  • The efficiency of contiguous, misaligned and strided access, worked through
  • Layout changes that make accesses coalesce

Conventions and assumptions

NVIDIA GPUs of compute capability 6.0 or later, where the CUDA C++ Best Practices Guide states that a warp's accesses coalesce into as many 32-byte transactions as are needed to serve all its threads. Each thread reads one 4-byte float.

The memory hierarchy

MemoryWhereShared bySpeed
RegistersOn the SMOne threadFastest
Shared memory and L1On the SMA thread block (shared) or the SM (L1)Very fast, a few dozen cycles
L2 cacheOn the chipAll SMsSlower
Global memory (HBM or GDDR)Off the chipThe whole GPUHundreds of cycles, but very high bandwidth
Coalesced vs scattered accessWhen 32 threads read 32 consecutive 4-byte words, the hardware serves them with one 128-byte transaction; when the addresses are scattered, it needs up to 32 separate transactions.32 threadsconsecutive1 transaction128 bytes32 threadsscatteredup to 32 transactions
Same 32 loads, very different traffic: one 128-byte transaction when addresses are consecutive, up to 32 when they are scattered.

Worked example: four access patterns

Pattern (lane k reads)32-byte sectorsBytes movedEfficiency
a[k], aligned4128100%
a[k + 1], off by one float516080%
a[2k], stride 2825650%
a[32k], stride 32321,02412.5%

In each case the warp uses 128 bytes, 32 floats. In the aligned case those are exactly four sectors. Shifted by one float, they straddle a fifth sector. With stride 2 the warp's addresses span 256 bytes, so eight sectors move and half of each is wasted. With stride 32 every lane lands in its own 128-byte region: 32 sectors, of which one-eighth is used.

Making accesses coalesce

  • Map consecutive threads to consecutive addresses: index arrays with threadIdx.x in the fastest-changing position.
  • Prefer a structure of arrays (x[], y[], z[]) to an array of structures ({x, y, z}[]), so a warp reading x touches only x.
  • For a matrix stored by rows, have neighboring threads read neighboring columns of the same row.
  • When a strided pattern can't be avoided, as in a transpose, load a tile into shared memory with coalesced reads and rearrange it there.

Common mistakes

  • Indexing a row-major matrix as a[threadIdx.x * width + j], so neighboring threads are a whole row apart.
  • Using an array of structures and reading one field, which strides by the size of the structure.
  • Assuming the L1 cache rescues a strided pattern. The transaction size stays 32 bytes either way.
  • Judging a kernel by FLOP/s when it is limited by memory bandwidth.

Check your understanding

  • Q1. A warp reads 32 consecutive 8-byte doubles, aligned. How many sectors? Answer: 8, since 256 bytes / 32 = 8, at 100% efficiency.
  • Q2. Lanes read a[4k] as floats. Sectors and efficiency? Answer: 16 sectors, 512 bytes moved for 128 used, 25%.
  • Q3. Why does an array of structures often coalesce badly? Answer: a warp reading one field skips over the other fields, which is a strided access.

Related

Sources

Last verified

The 32-byte transaction rule checked against the CUDA C++ Best Practices Guide, section 10.2.1. Every number in this post is computed in the site's tests (lib/gpuNumbers.test.js). Checked on 14 November 2026.