AdaGC Stabilizes Large Language Model Training

Researchers propose AdaGC, a new adaptive gradient clipping scheme, to improve the stability of large language model pretraining. They observed that loss spikes, disruptions in training, arise from multiple factors including data issues and computational errors.

AdaGC addresses this by bounding gradient norms based on a tensor’s historical clipped values. The team designed AdaGC to work with any optimization algorithm and minimize additional memory use and communication demands, especially during distributed training.

Testing on Llama-2 7B, Mixtral 8x1B, and ERNIE 10B-A1.4B models showed AdaGC eliminated training instabilities and improved downstream accuracy compared to GlobalGC by up to 2.48%. The code is publicly available on GitHub.