Pre-trained Transformer-based models have achieved state-of-the-art performance for
various Natural Language Processing (NLP) tasks. However, these models often have …

Fairness and environmental impact are important research directions for the sustainable
development of artificial intelligence. However, while each topic is an active research area in …

Recent papers have shown that large pre-trained language models (LLMs) such as BERT,
GPT-2 can be fine-tuned on private data to achieve performance comparable to non-private …

Recognizing causal precedence relations among the chemical interactions in biomedical
literature is crucial to understanding the underlying biological mechanisms. However …

Training large language models can consume a large amount of energy. We hypothesize
that the language model's configuration impacts its energy consumption, and that there is …

The ability of pre-trained BERT model to achieve outstanding performances on many
Natural Language Processing (NLP) tasks has attracted the attention of researchers in …

The remarkable success of large transformer-based models such as BERT, RoBERTa and
XLNet in many NLP tasks comes with a large increase in monetary and environmental cost …

Large pretrained language models (PreLMs) are rev-olutionizing natural language
processing across all benchmarks. However, their sheer size is prohibitive for small …

Distilling state-of-the-art transformer models into lightweight student models is an effective
way to reduce computation cost at inference time. The student models are typically compact …

Abstract Natural Language Processing (NLP) tasks in non-dominant and low-resource
languages have not experienced significant progress. Although pre-trained BERT models …