On-policy distillation provides dense, per-token supervision for coaching reasoning fashions; nonetheless, it stays unclear below which situations this sign is useful and below which it’s detrimental. Which trainer mannequin ought to be used, and within the case of self-distillation, which particular context ought to function the supervisory sign? Does the optimum alternative differ from one token to the following? At current, addressing these questions usually requires expensive coaching runs whose combination efficiency metrics obscure the dynamics on the degree of particular person tokens. We introduce a training-free diagnostic framework that operates on the highest decision: per token, per query, and per trainer. We derive a great per-node gradient outlined because the parameter replace that maximally will increase the coed’s likelihood of success. We then develop a scalable targeted-rollout algorithm to estimate this gradient effectively, even for lengthy chains of intermediate ideas. The gradient alignment rating, outlined because the cosine similarity between this supreme gradient and any given distillation gradient, quantifies the extent to which a selected configuration approximates the best sign. Throughout a spread of self-distillation settings and exterior trainer fashions, we observe that distillation steerage reveals considerably increased alignment with the best on incorrect rollouts than on right ones, the place the coed already performs properly and the trainer’s sign tends to turn into noisy. Moreover, we discover that the optimum distillation context relies upon collectively on the coed mannequin’s capability and the goal process, and that no single universally efficient configuration emerges. These findings inspire using per-task, per-token diagnostic analyses for distillation.
