Coverage gradient algorithms have pushed many latest developments in language mannequin reasoning. An interesting property is their means to be taught from exploration on their very own trajectories, a course of essential for fostering numerous and artistic options. As we present on this paper, many coverage gradient algorithms naturally scale back the entropy—and thus the range of explored trajectories—as a part of coaching, yielding a coverage more and more restricted in its means to discover. On this paper, we argue that entropy needs to be actively monitored and managed all through coaching. We formally analyze the contributions of main coverage gradient goals on entropy dynamics, establish empirical components (corresponding to numerical precision) that considerably influence entropy habits, and suggest specific mechanisms for entropy management. These embrace REPO, a household of algorithms that modify the benefit operate to control entropy, and ADAPO, an adaptive uneven clipping strategy. Fashions skilled with our entropy-preserving strategies keep range all through coaching, yielding closing insurance policies which might be extra performant and retain their trainability for sequential studying in new environments.
- †MIT
- ‡ Equal contribution
- ** Work achieved whereas at Apple
