When Is a GPU Faster Than a CPU?

A GPU is faster when three things hold: the work splits into many independent pieces, each piece does enough arithmetic per byte of memory it touches, and the job is big enough that copying data to the GPU and back doesn't eat the gain. A large matrix multiply meets all three. A one-off vector addition meets only the first, and on a typical system the CPU finishes it before the data has even crossed to the GPU.

What you'll learn

  • How CPU and GPU designs trade latency for throughput
  • Arithmetic intensity and the roofline model
  • A worked comparison of a vector add and a matrix multiply
  • Why the PCIe transfer often decides the answer

Conventions and assumptions

The device numbers are round illustrative values, not a specific product: a GPU with 1,000 GB/s of memory bandwidth and 20 TFLOP/s of float32 compute, a CPU with 50 GB/s of memory bandwidth, and a PCIe 4.0 x16 link, which carries 16 GT/s × 16 lanes × 128/130 ÷ 8 = 31.5 GB/s each way. FLOP counts use float32 and count a multiply-add as 2 FLOPs.

Latency vs throughput

CPU coreGPU
GoalFinish one thread fastFinish many threads in total
Hardware per threadLarge caches, branch prediction, out-of-order executionSmall: registers and a lane of a warp
Hides memory latency byCaches and running aheadSwitching to another warp
Good atBranchy, serial, latency-sensitive codeRegular, data-parallel code

Arithmetic intensity and the roofline

Arithmetic intensity is FLOPs done per byte moved from memory. A kernel can't run faster than the memory can feed it, so attainable speed = min(peak compute, intensity × memory bandwidth). The crossover, peak compute ÷ bandwidth, is 20 TFLOP/s ÷ 1,000 GB/s = 20 FLOP per byte for the example GPU. Below that a kernel is memory bound; above it, compute bound.

Worked example 1: vector add of 1 million floats

  • c[i] = a[i] + b[i]: 1 FLOP per 12 bytes (two 4-byte loads, one 4-byte store), an intensity of 1/12 ≈ 0.083 FLOP per byte.
  • Memory bound: at most 0.083 × 1,000 GB/s = 83.3 GFLOP/s, far below the 20 TFLOP/s peak.
  • The kernel moves 12 MB: 12 µs on the GPU.
  • But a, b and c must cross PCIe: 12 MB ÷ 31.5 GB/s = 0.381 ms.
  • The CPU does the same 12 MB of memory traffic in 12 MB ÷ 50 GB/s = 0.24 ms, so if the data starts in CPU memory, the CPU wins.

Worked example 2: multiplying two 4096 × 4096 matrices

  • 2n³ = 2 × 4096³ = 137.4 GFLOP of arithmetic.
  • At least 3n² × 4 bytes = 201.3 MB of data (A, B and C), an intensity of about 683 FLOP per byte, well above the ridge of 20, so compute bound.
  • GPU compute time is at least 137.4 GFLOP ÷ 20 TFLOP/s = 6.87 ms; moving the matrices over PCIe adds about 6.39 ms.
  • A CPU core running at even 100 GFLOP/s would need 1.37 s. Here the GPU wins by roughly a factor of 100, transfer included.

Common mistakes

  • Timing only the kernel and leaving out the copies to and from the GPU.
  • Moving data back and forth between steps instead of keeping it on the GPU for the whole pipeline.
  • Expecting a speedup on small problems, where launch overhead and transfers dominate.
  • Porting branchy, serial code and expecting the GPU to run it fast.

Check your understanding

  • Q1. What is the arithmetic intensity of y[i] = a × x[i] + y[i] in float32? Answer: 2 FLOPs per 12 bytes, about 0.17 FLOP per byte, so memory bound.
  • Q2. On the example GPU, above what intensity is a kernel compute bound? Answer: 20 FLOP per byte.
  • Q3. Why can a GPU lose on a fast kernel? Answer: when moving the data over PCIe takes longer than the CPU takes to do the whole job.

Related

Sources

Last verified

CPU and GPU architecture described from the CUDA Programming Guide's hardware model; device figures are stated assumptions. Every number in this post is computed in the site's tests (lib/gpuNumbers.test.js). Checked on 14 November 2026.