So you read my last post about running open models locally, got excited, and decided to spin up your own LLM on your machine. Then reality hit: you need bigger hardware, especially a lot more memory.
Why do these models demand so much memory? And why is memory so expensive right now?
Parameters: The Root of the Problem
If you’ve spent any time around AI, you’ve heard about parameters. They’re the closest analogy we have to synapses in the brain: more parameters generally mean a more capable model. Frontier models now operate in the trillions (Claude’s Mythos is rumored around 10 trillion).
Thanks to distillation and quantization, we can run surprisingly strong models on consumer hardware. But the memory requirements remain enormous. An 80B‑parameter model still needs roughly 128 GB of RAM to run comfortably.
Memory has always been the bottleneck. Amir Gholami’s excellent piece on the AI Memory Wall explains the underlying physics and architecture constraints in detail:
(https://medium.com/riselab/ai-and-memory-wall-2cb4265cb0b8)
https://github.com/amirgholami/ai_and_memory_wall/blob/main/imgs/pdfs/ai_and_compute.pdf
Memory Prices: The Hyperscale Squeeze
I recently came across projections for 2026 memory consumption in data centers:
70% of all memory chips produced will be absorbed by AI data centers.
That number is staggering.
Source:
https://accuristech.com/blog/ai-data-center-electronic-component-supply/
Now combine that with the surge in U.S. hyperscale construction spending:
Notice that the chart captures only a small part of total data center investment, since IT hardware represents a significant additional cost. (https://ourworldindata.org/grapher/monthly-spending-data-center-us)
The picture becomes clear:
- Models need massive memory.
- Hyperscalers are building at unprecedented speed.
- Demand is compounding faster than supply can respond.
Given this dynamic, I don’t expect memory supply constraints to resolve anytime soon. Memory production is becoming one of the defining choke points of the entire ecosystem.
The Bottom Line
Running open models locally is absolutely possible, and increasingly attractive, but memory is the tax you pay. At the hyperscale level, that tax is becoming one of the most important economic and strategic factors in AI infrastructure.
The “memory wall” isn’t theoretical. It’s here, and it’s shaping everything from home‑lab experimentation to trillion‑dollar data‑center buildouts.