Universally modeling all typical information extraction tasks (UIE) with one generative
language model (GLM) has revealed great potential by the latest study, where various IE …

A great number of deep learning based models have been recently proposed for automatic
music composition. Among these models, the Transformer stands out as a prominent …

Code representation learning, which aims to encode the semantics of source code into
distributed vectors, plays an important role in recent deep-learning-based models for code …

Code prediction, more specifically autocomplete, has become an essential feature in
modern IDEs. Autocomplete is more effective when the desired next token is at (or close to) …

Self-attention mechanism in graph neural networks (GNNs) led to state-of-the-art
performance on many graph representation learning tasks. Currently, at every layer …

We propose SentiBERT, a variant of BERT that effectively captures compositional sentiment
semantics. The model incorporates contextualized representation with binary constituency …

Automatic captioning of images is a task that combines the challenges of image analysis
and text generation. One important aspect of captioning is the notion of attention: how to …

Learning distributed representation of source code requires modelling its syntax and
semantics. Recent state-of-the-art models leverage highly structured source code …

Abstract We introduce Transformer Grammars (TGs), a novel class of Transformer language
models that combine (i) the expressive power, scalability, and strong performance of …

The success of large-scale contrastive vision-language pretraining (CLIP) has benefited
both visual recognition and multimodal content understanding. The concise design brings …