Ubiquitous sensors and smart devices from factories and communities are generating
massive amounts of data, and ever-increasing computing power is driving the core of …

Deep learning is becoming a mainstream technology for speech recognition at industrial
scale. In this paper, we provide an overview of the work by Microsoft speech researchers …

DNN training is extremely time-consuming, necessitating efficient multi-accelerator
parallelization. Current approaches to parallelizing training primarily use intra-batch …

Scaling up deep neural network capacity has been known as an effective approach to
improving model quality for several different machine learning tasks. In many cases …

Automatic Speech Recognition (ASR), which is aimed to enable natural human–machine
interaction, has been an intensive research area for decades. Many core technologies, such …

We show empirically that in SGD training of deep neural networks, one can, at no or nearly
no loss of accuracy, quantize the gradients aggressively—to but one bit per value—if the …

We introduce a new method for scaling up distributed Stochastic Gradient Descent (SGD)
training of Deep Neural Networks (DNN). The method solves the well-known communication …

This monograph provides an overview of general deep learning methodology and its
applications to a variety of signal and information processing tasks. The application areas …

PipeDream is a Deep Neural Network (DNN) training system for GPUs that parallelizes
computation by pipelining execution across multiple machines. Its pipeline parallel …

Transformers are one of the most important machine learning workloads today. Training one
is a very compute-intensive task, often taking days or weeks, and significant attention has …