Hardware accelerators based on field programmable gate array (FPGA) and system on chip
(SoC) devices have gained attention in recent years. One of the main reasons is that these …

In recent years, data are typically distributed in multiple organizations while the data security
is becoming increasingly important. Federated learning (FL), which enables multiple parties …

Synchronous strategies with data parallelism are widely utilized in distributed training of
Deep Neural Networks (DNNs), largely owing to their easy implementation yet promising …

To speed up the training of massive deep neural network (DNN) models, distributed training
has been widely studied. In general, a centralized training, a type of distributed training …

We propose VirtualFlow, a system leveraging a novel abstraction called virtual node
processing to decouple the model from the hardware. In each step of training or inference …

Synchronous parameters algorithms with data parallelism have been successfully utilized to
accelerate the distributed training of deep neural networks (DNNs). However, a prevalent …

The recent unprecedented success of deep learning (DL) in various fields is underlied by its
use of large-scale data and models. Training a large-scale deep neural network (DNN) …

In this paper, we consider the distributed local stochastic gradient descent (SGD) algorithm
by parallelizing multiple devices in the setting of stochastic convex optimization (SCO). The …

As the cost of deep learning training increases, using heterogeneous GPU clusters is a
reasonable way to scale cluster resources to support distributed deep learning (DDL) tasks …

The bulk synchronous parallel (BSP) is a celebrated synchronization model for general-
purpose parallel computing that has successfully been employed for distributed training of …