Deep Neural Networks (DNNs) are becoming an important tool in modern computing
applications. Accelerating their training is a major challenge and techniques range from …

Convolutional neural networks (CNN) is a specialized case of artificial neural networks
(ANN) and finds its application in computer vision and parallel distributed computing for …

Artificial intelligence (AI) has become a cornerstone of modern technology, revolutionizing
industries from healthcare to finance. Convolutional neural networks (CNNs) are a subset of …

Transformers are one of the most important machine learning workloads today. Training one
is a very compute-intensive task, often taking days or weeks, and significant attention has …

Graph Neural Networks (GNNs) are powerful and flexible neural networks that use the
naturally sparse connectivity information of the data. GNNs represent this connectivity as …

Machine Learning (ML) and Internet of Things (IoT) are complementary advances: ML
techniques unlock the potential of IoT with intelligence, and IoT applications increasingly …

Rematerialization and offloading are two well known strategies to save memory during the
training phase of deep neural networks, allowing data scientists to consider larger models …

Data-parallelism has become an established paradigm to train DNNs that fit inside GPU
memory on large-scale HPC systems. However, model-parallelism is required to train out-of …

Accelerating large-scale CNN training is needed to keep training times reasonable as
datasets grow larger and models become more complex. Existing frameworks primarily …

Abstract We present the Sequential Aggregation and Rematerialization (SAR) scheme for
distributed full-batch training of Graph Neural Networks (GNNs) on large graphs. Large …