The increasing demand for improving deep learning model performance has led to a
paradigm shift in supporting low-precision computation to harness the robustness of deep …

The role of numerical accuracy in training and evaluating neural network-based potential
energy surfaces is examined for different experimental observables. For observables that …

In recent years, large language models (LLMs) have achieved remarkable success in
natural language processing (NLP). LLMs require an extreme amount of parameters to …

The demand for inference on extremely large scale LLMs has seen enormous growth in the
recent months. It made evident the colossal shortage of dedicated hardware capable of …

High-quality data is crucial for the pre-training performance of large language models.
Unfortunately, existing quality filtering methods rely on a known high-quality dataset as …

Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with
their sophisticated capabilities. Training these models requires vast GPU clusters and …

Pretraining transformers are generally time-consuming. Fully quantized training (FQT) is a
promising approach to speed up pretraining. However, most FQT methods adopt a quantize …

The increasing scale of Transformer models has led to an increase in their pre-training
computational requirements. While quantization has proven to be effective after pre-training …

To efficiently train large-scale models, low-bit gradient communication compresses full-
precision gradients on local GPU nodes into low-precision ones for higher gradient …

Recent work has shown that 8-bit floating point (FP8) can be used for efficiently training
neural networks with reduced computational overhead compared to training in FP32/FP16 …