In recent years, natural language processing (NLP) has got great development with deep
learning techniques. In the sub-field of machine translation, a new approach named Neural …

This paper demonstrates that multilingual denoising pre-training produces significant
performance gains across a wide variety of machine translation (MT) tasks. We present …

Despite the recent success of deep neural networks in natural language processing and
other spheres of artificial intelligence, their interpretability remains a challenge. We analyze …

Pretrained masked language models (MLMs) require finetuning for most NLP tasks. Instead,
we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are …

Deep learning has improved performance on many natural language processing (NLP)
tasks individually. However, general NLP models cannot emerge within a paradigm that …

Computer vision has benefited from initializing multiple deep layers with weights pretrained
on large supervised training sets like ImageNet. Natural language processing (NLP) …

Layer normalization (LayerNorm) is a technique to normalize the distributions of
intermediate layers. It enables smoother gradients, faster training, and better generalization …

Unsupervised representation learning algorithms such as word2vec and ELMo improve the
accuracy of many supervised NLP models, mainly because they can take advantage of large …

Cross-domain alignment between two sets of entities (eg, objects in an image, words in a
sentence) is fundamental to both computer vision and natural language processing. Existing …

Neural Machine Translation (NMT) can be improved by including document-level contextual
information. For this purpose, we propose a hierarchical attention model to capture the …