Uncertainty quantification for LLMs is a key analysis path in direction of addressing hallucination and different points that restrict their dependable deployment. On this work, we present that reasoning hint size is a straightforward and helpful confidence estimator in massive reasoning fashions. By way of complete experiments throughout a number of fashions, datasets, and prompts, we present that hint size performs in comparable however complementary methods to different zero-shot confidence estimators akin to verbalized confidence. Our work reveals that reasoning post-training basically alters the connection between hint size and accuracy, going past prior work that had proven that post-training causes traces to develop longer typically (e.g., “overthinking”). We examine the mechanisms behind hint size’s efficiency as a confidence sign, observing that the impact stays even after adjusting for confounders akin to downside problem and GRPO-induced size bias. We determine high-entropy or “forking” tokens as enjoying a key function within the mechanism. Our findings exhibit that reasoning post-training enhances uncertainty quantification past verbal expressions, and set up hint size as a sensible confidence measure for giant reasoning fashions.
- † College of Southern California
- ‡ Stanford College
