The problem › Two exponentials, different exponents
A four-page paper predicts the next thirty years
In December 1994, Wm. Wulf and Sally McKee wrote a four-page note titled Hitting the Memory Wall: Implications of the Obvious. Its opening is disarmingly honest: "This brief note points out something obvious — something the authors 'knew' without really understanding." The obvious thing: microprocessor speed and DRAM speed were bothimproving exponentially, but "the exponent for microprocessors is substantially larger than that for DRAMs. The difference between diverging exponentials also grows exponentially."
You already have the vocabulary for this from 0.2 and 0.3: CPU speed is transistor switching (which scaled beautifully), while DRAM latency is capacitor sensing across physical distance (which barely scales at all; 0.3 put it at 8–9× over four decades). Two curves, both up and to the right, one much steeper, and the gap between them compounds every single year. Their question was simply: how big, how soon?
The mechanism › The argument
Even a perfect cache can't save you
The paper's engine is one line, "an old friend" as they call it: the average time to access memory, where t_c and t_m are the cache and DRAM access times and p is the probability of a cache hit:
t_avg = p × t_c + (1 − p) × t_m
Then come the assumptions, and they are stacked against the paper's own thesis. First, cache speed scales with processor speed (true for on-chip cache), so everything is normalized to t_c = 1 CPU cycle. Second, the generous one: the cache is perfect. No capacity misses, no conflict misses; the only misses are compulsoryones, first-ever touches of a line, which they bound at 1% or less. So (1 − p) is small. But it is not zero, and "therefore as t_c and t_m diverge, t_avgwill grow and system performance will degrade. In fact, it will hit a wall."
Where is the wall, exactly? In most programs 20–40% of instructions reference memory; taking the favorable end, every 5th instruction touches memory on average, so the wall is the moment t_avg exceeds 5 instruction times. Past it, "system performance is totally determined by memory speed; making the processor faster won't affect the wall-clock time to complete an application." And the exits are bricked up by construction: "we have already assumed a perfect cache, so a bigger/smarter one won't help. We're already using the full bandwidth of the memory, so prefetching or other related schemes won't help either."
The inputs were the era's trend rates: DRAM speeds improving 7% per year, microprocessor performance 80% per year (Baskett's estimate), and a next level of hierarchy starting four times slower than cache. From these they projected the average cost of a memory access in CPU cycles:
| year | avg cycles per memory access |
|---|---|
| 2000 | 1.52 |
| 2005 | 8.25 |
| 2010 | 98.8 |
"Under these assumptions, the wall is less than a decade away."The paper's sensitivity analysis makes the trap airtight: the starting miss cost and the exact hit rate move the date, not the destiny. At a 99.8% hit rate and a conservative miss cost of 4 cycles, the 5-cycle wall arrives in 11–12 years; at 99%, within the decade; at 90%, within five years. However you tune it, "in 10–15 years each memory access will cost, on average, tens or even hundreds of processor cycles." Their verdict: "we are going to hit a wall in the improvement of system performance unless something basicchanges" and, closing the paper, "it would appear that we do not have a great deal of time."
Predict first
Under their assumptions, an average memory access in 2010 would cost about how many CPU cycles?
interactive · coming soon
WallScrubber: the gap, decade by decade
Planned interactive: a timeline scrubber from 1980 to today plotting CPU performance and DRAM latency on a log axis. Drag through the decades and watch the gap open; overlay the year caches were forced to go multi-level.
SourcesWulf & McKee 1995 SIGARCH CAN 23(1) · pp. 20–23CAQA 6e §1.4 · pp. 18–21
The number › Your turn
Compound the gap yourself
Compute the number › diverging exponentials, 1994 rates
| CPU performance growth | ×1.80 per year (Baskett estimate) |
| DRAM speed growth | ×1.07 per year [Hen90] |
| gap growth per year | 1.80 ÷ 1.07 ≈ ×1.68 |
| compounded over 5 years | 1.68⁵ ≈ ×13 |
| compounded over 10 years | 1.68¹⁰ ≈ ×180 |
| Result | the CPU–DRAM gap grows ≈ 180× per decade |
The individual yearly rates look harmless: 80% vs 7% sounds like "both improving." Divide and compound, and one decade turns a manageable gap into a chasm. This arithmetic, a small ratio relentlessly compounded, is the single reason the memory hierarchy exists, and it's a two-line calculation you can now redo on any pair of trend lines you're suspicious of.
The full story › what they got right, what they hedged, and what actually happened
The hedge:the authors knew the graveyard they were walking into. "Over the past thirty years there have been several predictions of the eminent cessation of the rate of improvement in computer performance. Every such prediction was wrong," they wrote, wrong because each hinged on unstated assumptions overturned by later events, and admitted "our prediction of the memory wall is probably wrong too."The "most convenient resolution," they noted, would be "a cool, dense memory technology whose speed scales with that of processors," of which they were aware of none. It still doesn't exist.
Right, eerily:their speculative escape routes read like this course's table of contents. They asked whether it was "time to forgo the model that access time is uniform to all parts of the address space" and whether "the compiler [could] explicitly manage a smaller amount of higher speed memory," which is, word for word, GPU shared memory (Topic 2). They asked for "new ideas for how to trade computation for storage," which is FlashAttention's recomputation and speculative decoding (Topic 3). They even asked whether compulsory misses could be driven to zero with special "first write" instructions.
Missed: the wall was never broken; it was hidden. Deeper hierarchies (two cache levels became three), out-of-order cores overlapping misses with useful work (0.9), and above all parallelism: multicore CPUs and then GPUs stopped trying to make one access fast and kept thousands in flight, spending the bandwidth that kept scaling 32,000× (0.3's asymmetry) to hide the latency that improved 8–9×. Meanwhile the wall grew a second face the paper never considered: DRAM capacity growth collapsed too, from quadrupling every three years to nearly flat.Latency hidden, never beaten: that move is Topic 1's founding idea.
Two exponentials with different exponents means a gap that compounds ≈180× per decade, and even a perfect cache only delays the reckoning, because (1 − p) is small but never zero. The industry never tore the wall down; it learned to hide it behind hierarchy and parallelism, and the rest of this course is a tour of the hiding places.