Skip to main content
Lab Grimoire
TW EN
Coffee
LLM Internals

AI Does Not Need to Think at Length for Every Question

# AI Does Not Need to Think at Length for Every Question The useful question is not whether longer reasoning is always better, but how much test-time compute a

Author
CY
Published
Updated
AI Does Not Need to Think at Length for Every Question

The useful question is not whether longer reasoning is always better, but how much test-time compute a task deserves. A 2026 math-reasoning study trained a model to select among three response modes. On held-out MATH500, the policy used 41% fewer response tokens, with accuracy slightly below its base model [1].

Fixed long reasoning is not a universal solution

The best test-time scaling method can depend on prompt difficulty; one 2024 study reported more than 4x efficiency for its compute-optimal strategy versus a best-of-N baseline in a specific comparison [2]. That result is not a universal fourfold saving.

The 2026 policy selected a first response token for NoThink, Short, or Long. It was trained with GRPO, shaped rewards, and mode-specific token caps, without a separate router [1]. In the authors' three-seed MATH500 evaluation, base accuracy was 0.796 and adaptive-policy accuracy was 0.782; mean response length fell from 4,796 to 2,811 tokens, a reported 41% reduction [1]. The authors also reported 76% fewer tokens on GSM8K and higher accuracy than baselines at similar response length [1].

A self-routing model selects NoThink, Short, or Long and the figure lists author-reported MATH500 and GSM8K results

Figure 1. Programmatic redraw of the method and reported results in source [1]; not an independent reproduction.

Reasoning budget at a glance: what self-routed thinking length achieved and where it stops

More thinking can also overrun

A 2025 study across models and benchmarks observed initial gains from additional thinking followed by a decline, which the authors called overthinking [3]. This examined extended reasoning traces, not the 2026 adaptive router, so the two results should not be conflated.

The new routing result covers one 1.5B model trained on a mathematical distribution and three seeds; it does not establish similar savings for writing, code, or general chat [1]. Figure 2 keeps the three studies separate rather than treating them as a shared benchmark.

Three test-time-compute studies with their separate findings and extrapolation limits

Figure 2. Evidence boundaries across three separate studies; not a pooled benchmark or shared performance curve [1][2][3].

For a product or local-LLM evaluation, compare accuracy by task difficulty and record tokens and latency together. Treat difficulty-based routing as a candidate policy to validate on representative tasks, not a universal default [1][2][3].

References

  1. Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation
  2. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
  3. Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models

Frequently Asked Questions

Does this mean reasoning models are more accurate when they think less?

No. The 2026 study evaluated one 1.5B math-trained model on math benchmarks. Its adaptive policy scored 0.782 on MATH500 versus 0.796 for the base model. Fewer tokens did not mean higher accuracy in this comparison [1].

Does 41% refer to an accuracy improvement?

No. It is the reported reduction in mean response tokens, from 4,796 to 2,811. The authors separately reported MATH500 accuracy of 0.782 for the adaptive policy and 0.796 for the base model [1].

Do these findings apply to writing, coding, or general chat?

That has not been established. The main self-routing result covers one 1.5B model trained on a mathematical distribution and math benchmarks; it is not a general-chat or coding evaluation [1].

Does the 2025 overthinking study disprove the self-routing approach?

No. It studied extended reasoning traces and observed gains that later declined in tested settings. It was not a direct test of the 2026 adaptive router [3].

Found this useful? RSS · X · Buy me a coffee