Deep Neural Networks (DNNs) are becoming an important tool in modern computing
applications. Accelerating their training is a major challenge and techniques range from …

A variety of methods have been applied to the architectural configuration and learning or
training of artificial deep neural networks (DNN). These methods play a crucial role in the …

We investigate the optimal model size and number of tokens for training a transformer
language model under a given compute budget. We find that current large language models …

We investigate the optimal model size and number of tokens for training a transformer
language model under a given compute budget. We find that current large language models …

Federated learning is an emerging distributed machine learning framework for privacy
preservation. However, models trained in federated learning usually have worse …

Scalability and privacy are two critical concerns for cross-device federated learning (FL)
systems. In this work, we identify that synchronous FL–cannot scale efficiently beyond a few …

We study empirical scaling laws for language model performance on the cross-entropy loss.
The loss scales as a power-law with model size, dataset size, and the amount of compute …

Large-scale model training has been a playing ground for a limited few requiring complex
model refactoring and access to prohibitively expensive GPU clusters. ZeRO-Offload …

In this paper, the problem of energy efficient transmission and computation resource
allocation for federated learning (FL) over wireless communication networks is investigated …

Federated Learning enables visual models to be trained in a privacy-preserving way using
real-world data from mobile devices. Given their distributed nature, the statistics of the data …