The Thinking Leakage in Reinforcement Learning for LLMs

RL in no-think mode mostly wakes up a model's existing thinking mode instead of teaching new reasoning.

75%

RL gain removed by subspace patching

Undoing just an 8-dimensional think subspace erases most of RL's accuracy gain

How it works

Suspicious no-think gains
Qwen3 models trained with RL in no-think mode score higher. But think and no-think modes share the same weights.
Probe weights, activations, heads
Train Qwen3-1.7B/4B/8B with GRPO, track think-likeness, then scale updates, patch activation subspaces and single attention heads.
Gain is borrowed thinking
Scaled updates approach but never beat think-mode accuracy; patching a think subspace removes 75% of the gain.

Which comes out on top on Share of RL gain removed?

try it

Drag the scaling factor c up and watch accuracy climb.

Update scale c1 x

Accuracy flattens near think-mode's 0.74: the update points toward thinking already present.Illustrative model, not the paper's data

6 terms

Hybrid reasoning model One model, like Qwen3, that can answer with or without a long reasoning trace.

A pianist told to sight-read without practicing slowly sneaks back her rehearsed habits; her score improves, but she hasn't learned new pieces.

Where it breaks

The model's think mode is a hidden switch, not conscious habit.

From the paper