We research outlier tokens in Diffusion Transformers (DiTs) for picture era. Prior work has proven that Imaginative and prescient Transformers (ViTs) can produce a small variety of high-norm tokens that entice disproportionate consideration whereas carrying restricted native data, however their function in generative fashions stays underexplored. We present that this phenomenon seems in each the encoder and denoiser of contemporary Illustration Autoencoder (RAE)-DiT pipelines: pretrained ViT encoders can produce outlier representations, and DiTs themselves can develop inside outlier tokens, particularly in intermediate layers. Furthermore, merely masking high-norm tokens doesn’t enhance efficiency, indicating that the issue is just not solely brought on by a couple of excessive values, however is extra carefully associated to corrupted native patch semantics. To handle this situation, we introduce Twin-Stage Registers (DSR), a register-based intervention for each parts: educated registers when out there, recursive test-time registers in any other case, and diffusion registers for the denoiser. Throughout ImageNet and large-scale text-to-image era, these interventions constantly scale back outlier artifacts and enhance era high quality. Our outcomes spotlight outlier-token management as an vital ingredient in constructing stronger DiTs.
- †Rice College
- * Equal contribution
