Latency vs bandwidth

Chapter 0.2 left us with DRAM: dense, slow, and dealing in bursts. But slow how?When someone says memory is "fast," they are smuggling two completely different claims past you. The architecture texts keep the two words strictly apart: latency (or response time) is the time between the start and the completion of one event: one load, one disk access. Bandwidth (or throughput) is the total amount of work done per unit time, bytes per second once the stream is flowing.¹

Latency is the length of the pipe, paid once per request. Bandwidth is the widthof the pipe, a steady-state rate. They are set by different physics, they improve at wildly different rates, and they are bought with different money. This is the first idea of the course's spine, latency ≠ bandwidth ≠ capacity, and from here to the last chapter on LLM decoding, confusing them is the most expensive mistake you can make.

The classic proof that they are independent: a station wagon full of backup tapes hurtling down the highway (Tanenbaum's formulation). Terabytes arrive: colossal bandwidth. But the first byte takes a day. Nobody wants to play a video game over a station wagon, and nobody wants to ship a datacenter's backups over a phone line. Different jobs bind on different numbers.

The two numbers are so unlike each other that they are not even measured the same way. Latency is measured with dependent loads: pointer chasing, where each 64-byte line holds the address of the next, so no access can begin until the previous one returns; the chain forces you to feel every round trip. Bandwidth is measured with independent streaming loads whose results feed nothing, so the hardware is free to overlap as many as it can.² Hold on to that asymmetry: dependence exposes latency, independence buys bandwidth. It is the mechanical heart of chapter 0.9 and of every GPU in Topic 1.

SourcesCAQA 6e §1.4 · pp. 20–21Bakhvalov 2e §4.10 · pp. 88–91CS:APP 3e §6.1 · pp. 617 ff.

For a single transfer of N bytes, the total time is one pipe-length wait plus the flow time:

T(N) = latency + N / bandwidth

Two regimes hide in that sum. When N is small, the second term vanishes and latency is the whole bill: ask for 64 bytes from DRAM (90 ns away, 76.8 GB/s wide on our reference Intel Core i9-12900K) and the transfer itself contributes a laughable 0.8 ns. When N is large, the first term vanishes and bandwidth is the whole bill: stream 64 MB and the 90 ns you waited up front rounds to nothing.

The border between the regimes is worth knowing by heart: N* = latency × bandwidth, the number of bytes the pipe could have carried during one wait. For our reference DRAM that's 90 ns × 76.8 bytes/ns ≈ 6.8 KB. Transfers much smaller than N* are latency's hostages; transfers much bigger are bandwidth's. Hardware designers call this quantity the bandwidth-delay product, and every trick in this course, from cache lines and DRAM bursts to GPU coalescing and KV-cache batching, is an attempt to stop paying pipe-length prices for sub-N* payloads.

Here is the asymmetry that shapes the next forty years of hardware (and chapter 0.4): bandwidth improves much faster than latency. CAQA's milestone data across four decades of technology puts numbers on it: microprocessors and networks gained 32,000–40,000× in bandwidth but only 50–90× in latency; main memory and disks gained 400–2,400× in bandwidth against a pitiful 8–9× in latency.³ The book distills it into a rule of thumb worth memorizing verbatim: "bandwidth grows by at least the square of the improvement in latency."⁴

The reason is structural, not accidental. Bandwidth is parallelism: add wires, add channels, add banks (0.2), run the bus faster. That means you can buyit with transistors and pins, which is exactly what Moore's law kept handing out. Latency is a round trip through physical distance, wire capacitance, and DRAM sense-amplifier settling (0.2 again), none of which shrink on demand. One improves like a budget; the other like a law of nature. The trend lineage runs through Patterson's 2004 essay, aptly titled Latency Lags Bandwidth.⁵

Consequence: hardware increasingly spendsbandwidth to hide latency, fetching 64 bytes when you asked for 4, prefetching lines you haven't requested yet, running thousands of GPU threads so someonealways has data in flight. Once you see "burn width to hide length" as a move, you will recognize it in every chapter that follows.

SourcesCAQA 6e §1.4, Fig. 1.9 · pp. 20–21

On real silicon the two numbers split further, and the splits are instructive. The ~90 ns we quote is idle (or unloaded) latency: one polite pointer-chase on an otherwise quiet machine. Under load, with other cores hammering the same memory controller, requests queue, and observed latency climbs well above idle; the quoted number is a floor, not a promise.⁶

Bandwidth has the mirror-image caveat: one core cannot reach it alone. On Bakhvalov's measured Alder Lake (i7-1260P, DDR4-2400, 2 channels), a single P-core sustains about 25 GB/s from DRAM while the theoretical maximum is 38.4 GB/s; it takes all the cores together to push ~34 GB/s.⁷Why can't one core do it? Because bandwidth is only realized by keeping many transfers in flight at once, and a single core can track only so many outstanding misses. The number of in-flight bytes needed is our friend N* wearing its other hat: this is memory-level parallelism (MLP), and its shortage on a single thread is precisely the opening a GPU exploits with ten thousand threads (Topic 1).

Same idle latency, same DIMMs, yet three different numbers depending on how you ask: dependent chain (90+ ns per access), one streaming core (~⅔ of peak), every core streaming (~90% of peak). A spec sheet gives you two scalars; the machine gives you a surface. Bakhvalov's advice after measuring his own laptop: memorize your machine's hierarchy numbers; they are the mental model every later analysis leans on.⁸