Decoded
← Feed
arXiv:2609.12551DCAI11 sep 2026

RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems

1 min readHN ×8PDF ↗
DEEPSEEK V4 FLASH0.45DEEPSEEK V4 PRO1GLM-5.35.42KIMI K316.47020 MEMORY TRAFFIC PER TOKEN
fig. memory traffic per tokenDeepSeek V4 reads far less memory per decoded token

A simulator language lets an AI agent redesign LLM serving setups without running real code.

39.5×

Peak decode throughput gap

DeepSeek V4 Flash versus Kimi K3 at 1M context

How it works

  1. Profiling hits limits

    AI optimizers tune existing software by profiling it, so designs the stack can't run well never get found.

  2. Simulate architectures instead

    RoofLang describes workloads and hardware as graphs, allows only meaning-preserving rewrites, and scores them with a roofline simulator.

  3. Insights and new designs

    It shows compact KV caches drive DeepSeek V4's speed; a Claude Code agent found layouts up to 50.1% better.

tracenew actionsComputegraphHardwaregraphSafe graphrewritesPlacementRooflinesimulatorAI optimizeragenttracenew actionsComputegraphHardwaregraphSafe graphrewritesPlacementRooflinesimulatorAI optimizeragent

What they found

Which comes out on top on Memory traffic per token?

Bigger batches amortize weight reads

try it

Drag batch size up and watch decode throughput per GPU climb then flatten.

Batch sizeDecode throughputweight reads dominateKV reads dominateV4 Pro peak, 1M context
Decode throughput
low
here
weight reads dominate
Weight reads get shared across a batch; KV-cache reads don't. Small KV caches allow bigger batches.Illustrative model, not the paper's data

Terms

6 terms

Roofline model Estimates op time as the slowest of compute, memory, or network limits.

Like an architect testing floor plans in a physics simulator before hiring builders, instead of only remodeling the house already standing.

Where it breaks
The simulator assumes ideal overlap and load balance, unlike real construction crews.

Limits

  • Initial compute and hardware graphs must be verified correct by humans.
  • Assumes perfect MoE load balance and omits KV appending and speculative decoding.
  • Simulating Kimi K3 on 128 GPUs can take hours and hundreds of GB.
From the paper