Recent years have witnessed an exponential increase in the use of mobile and embedded
devices. With the great success of deep learning in many fields, there is an emerging trend …

The growing energy and performance costs of deep learning have driven the community to
reduce the size of neural networks by selectively pruning components. Similarly to their …

The use of object detection algorithms has become extremely important in autonomous
vehicles. Object detection at high accuracy and a fast inference speed is essential for safe …

Neural network pruning offers a promising prospect to facilitate deploying deep neural
networks on resource-limited devices. However, existing methods are still challenged by the …

The attention mechanism is becoming increasingly popular in Natural Language Processing
(NLP) applications, showing superior performance than convolutional and recurrent …

Abstract Knowledge distillation is a widely applicable technique for training a student neural
network under the guidance of a trained teacher network. For example, in neural network …

Structured pruning of filters or neurons has received increased focus for compressing
convolutional neural networks. Most existing methods rely on multi-stage optimizations in a …

Many neural network pruning algorithms proceed in three steps: train the network to
completion, remove unwanted structure to compress the network, and retrain the remaining …

State-of-the-art CNN based recognition models are often computationally prohibitive to
deploy on low-end devices. A promising high level approach tackling this limitation is …

High network communication cost for synchronizing gradients and parameters is the well-
known bottleneck of distributed training. In this work, we propose TernGrad that uses ternary …