DumpsterCluster: From Dumpster Diving to Serving LLaMA-70B on $60 GPUs

128 used $60 GPUs serve LLaMA-70B cheaply, but only clean, cheap electricity makes it green.

223K

tokens/s

LLaMA-8B prefill-heavy throughput, 128 V100s

Beats one B200 at a fraction of hardware cost

How it works

Retired GPUs pile up
Datacenters retire working V100s every few years. They are cheap but have small memory, slow links and poor energy efficiency.
Deep pipeline on used parts
Built a 128-V100 cluster from second-hand parts. A Rust engine puts each layer on its own GPU; CPUs handle transfers.
Cheap, but carbon-heavy
Throughput is competitive and capital cost far lower. On average grids, carbon per token is about 4x (8B) to 40x (70B) higher.

From the paper