Ladder: Enabling Efficient {Low-Precision} Deep Learning Computing through Hardware-aware Tensor Transformation
The increasing demand for improving deep learning model performance has led to a
paradigm shift in supporting low-precision computation to harness the robustness of deep …
paradigm shift in supporting low-precision computation to harness the robustness of deep …
Numerical Accuracy Matters: Applications of Machine Learned Potential Energy Surfaces
The role of numerical accuracy in training and evaluating neural network-based potential
energy surfaces is examined for different experimental observables. For observables that …
energy surfaces is examined for different experimental observables. For observables that …
Achieving Peak Performance for Large Language Models: A Systematic Review
In recent years, large language models (LLMs) have achieved remarkable success in
natural language processing (NLP). LLMs require an extreme amount of parameters to …
natural language processing (NLP). LLMs require an extreme amount of parameters to …
Accurate Block Quantization in LLMs with Outliers
N Trukhanov, I Soloveychik - arXiv preprint arXiv:2403.20137, 2024 - arxiv.org
The demand for inference on extremely large scale LLMs has seen enormous growth in the
recent months. It made evident the colossal shortage of dedicated hardware capable of …
recent months. It made evident the colossal shortage of dedicated hardware capable of …
ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws
R Li, Y Wei, M Zhang, N Yu, H Hu, H Peng - arXiv preprint arXiv …, 2024 - arxiv.org
High-quality data is crucial for the pre-training performance of large language models.
Unfortunately, existing quality filtering methods rely on a known high-quality dataset as …
Unfortunately, existing quality filtering methods rely on a known high-quality dataset as …
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with
their sophisticated capabilities. Training these models requires vast GPU clusters and …
their sophisticated capabilities. Training these models requires vast GPU clusters and …
Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization
Pretraining transformers are generally time-consuming. Fully quantized training (FQT) is a
promising approach to speed up pretraining. However, most FQT methods adopt a quantize …
promising approach to speed up pretraining. However, most FQT methods adopt a quantize …
Exploring Quantization for Efficient Pre-Training of Transformer Language Models
The increasing scale of Transformer models has led to an increase in their pre-training
computational requirements. While quantization has proven to be effective after pre-training …
computational requirements. While quantization has proven to be effective after pre-training …
LoCo: Low-Bit Communication Adaptor for Large-scale Model Training
To efficiently train large-scale models, low-bit gradient communication compresses full-
precision gradients on local GPU nodes into low-precision ones for higher gradient …
precision gradients on local GPU nodes into low-precision ones for higher gradient …
Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point
Recent work has shown that 8-bit floating point (FP8) can be used for efficiently training
neural networks with reduced computational overhead compared to training in FP32/FP16 …
neural networks with reduced computational overhead compared to training in FP32/FP16 …