Distributed deep learning (DL) has become prevalent in recent years to reduce training time
by leveraging multiple computing devices (eg, GPUs/TPUs) due to larger models and …

Huge scale machine learning problems are nowadays tackled by distributed optimization
algorithms, ie algorithms that leverage the compute power of many devices for training. The …

The scale of modern datasets necessitates the development of efficient distributed
optimization methods for machine learning. We present a general-purpose framework for …

The rapid growth of memory and computation requirements of large language models
(LLMs) has outpaced the development of hardware, hindering people who lack large-scale …

Coordinate descent with random coordinate selection is the current state of the art for many
large scale optimization problems. However, greedy selection of the steepest coordinate on …

We describe a new software framework for fast training of generalized linear models. The
framework, named Snap Machine Learning (Snap ML), combines recent advances in …

Coordinate descent methods minimize a cost function by updating a single decision variable
(corresponding to one coordinate) at a time. Ideally, we would update the decision variable …

To alleviate hardware scarcity in training large deep neural networks (DNNs), particularly
large language models (LLMs), we present FusionLLM, a decentralized training system …

Long training times have increasingly become a burden for researchers by slowing down
the pace of innovation, with some models taking days or weeks to train. In this paper, a new …

In this paper we propose a novel parallel stochastic coordinate descent (SCD) algorithm
with convergence guarantees that exhibits strong scalability. We start by studying a state-of …