How Does a GPU Run Thousands of Threads at Once?
A GPU runs thousands of threads by giving each one very little hardware and switching between them constantly. Threads are grouped into blocks, each block runs on one streaming multiprocessor (SM), and inside an SM the threads run in groups of 32 called warps that execute one instruction at a time together. When a warp waits for memory, the SM issues instructions from another warp that is ready, so the arithmetic units stay busy without the large caches and complex control a CPU core uses.
What you'll learn
- The hierarchy: grid, thread blocks, warps and threads, and where each runs
- How a kernel maps one thread to one data element
- How many blocks and warps a real launch creates
- How switching between warps hides memory latency
Conventions and assumptions
NVIDIA's CUDA terms, which are the most widely documented; AMD and others use different names (wavefronts, compute units) for the same ideas. The facts about SMs, blocks and warps come from the CUDA Programming Guide. Latency figures in the last section are round illustrative numbers, not a particular GPU.
The hierarchy
| Level | What it is | Runs on |
|---|---|---|
| Grid | Every thread of one kernel launch | The whole GPU |
| Thread block | Up to 1,024 threads that can share memory and synchronize | One SM, start to finish |
| Warp | 32 threads of a block that issue together | One SM's warp schedulers |
| Thread | One instance of the kernel, with its own registers | One lane of a warp |
A GPU contains tens to over a hundred SMs. The hardware hands thread blocks to SMs as space frees up, so the same program scales from a small GPU to a large one without changes: a larger GPU simply runs more blocks at once.
A kernel: one thread per element
__global__ void add(const float *a, const float *b, float *c, int n) {
int i = blockIdx.x * blockDim.x + threadIdx.x; // one element per thread
if (i < n) c[i] = a[i] + b[i]; // the last block is partly idle
}
// Launch: enough 256-thread blocks to cover n elements
add<<<(n + 255) / 256, 256>>>(a, b, c, n);Worked example: adding two million-element vectors
- n = 1,000,000 with 256 threads per block needs ceil(1,000,000 / 256) = 3,907 blocks.
- That launches 3,907 × 256 = 1,000,192 threads, 192 more than there are elements, which is why the kernel checks i < n.
- The last block covers elements 999,936 to 999,999: only 64 of its 256 threads do any work.
- Each block is 256 / 32 = 8 warps, so the launch creates 3,907 × 8 = 31,256 warps for the SMs to work through.
Hiding latency by switching warps
A read from GPU memory takes hundreds of clock cycles. A CPU core hides that with big caches and by running ahead out of order. A GPU instead keeps many warps resident on each SM, each with its own registers, so switching costs nothing: every cycle the scheduler picks a warp whose next instruction is ready. As a rough illustration, if a load takes 400 cycles and each warp has 8 independent instructions to issue before it needs the result, an SM needs about 400 / 8 = 50 warps' worth of work in flight to never sit idle. That is why GPUs want far more threads than they have arithmetic units, and why the number of resident warps (occupancy) matters.
Common mistakes
- Launching too few threads. A few thousand threads can leave most SMs idle; GPUs need tens of thousands.
- Forgetting the bounds check, so the extra threads in the last block write past the array.
- Assuming threads in different blocks can synchronize with each other during a kernel. Only threads within a block can (clusters aside).
- Picking a block size that isn't a multiple of 32, which leaves part of the last warp of every block empty.
Check your understanding
- Q1. How many blocks of 128 threads cover 1,000 elements? Answer: 8, since ceil(1000 / 128) = 8, with 24 idle threads.
- Q2. How many warps is a 1,024-thread block? Answer: 32.
- Q3. Why does a GPU need many more threads than it has arithmetic units? Answer: so that while some warps wait for memory, others are ready to issue.
Related
Sources
Last verified
SM, block and warp facts checked against the CUDA Programming Guide. Every number in this post is computed in the site's tests (lib/gpuNumbers.test.js). Checked on 14 November 2026.