The field of deep learning has witnessed significant progress, particularly in computer vision
(CV), natural language processing (NLP), and speech. The use of large-scale models …

Training large, deep neural networks to convergence can be prohibitively expensive. As a
result, often only a small selection of popular, dense models are reused across different …

We propose a parameter sharing method for Transformers (Vaswani et al., 2017). The
proposed approach relaxes a widely used technique, which shares parameters for one layer …

Recent advances in vision Transformers (ViTs) have come with a voracious appetite for
computing power, high-lighting the urgent need to develop efficient training methods for …

The field of deep learning has witnessed significant progress in recent times, particularly in
areas such as computer vision (CV), natural language processing (NLP), and speech. The …

This report describes Microsoft's machine translation systems for the WMT21 shared task on
large-scale multilingual machine translation. We participated in all three evaluation tracks …

Transformer-based neural models are used in many AI applications. Training these models
is expensive, as it takes huge GPU resources and long duration. It is challenging because …

We propose a scalable framework for the learning of high-dimensional parametric maps via
adaptively constructed residual network (ResNet) maps between reduced bases of the …

Recently, deep models have shown tremendous improvements in neural machine
translation (NMT). However, systems of this kind are computationally expensive and memory …

The large attention-based encoder-decoder network (Transformer) has become prevailing
recently due to its effectiveness. But the high computation complexity of its decoder raises …