The rapid growth of Large Language Models (LLMs) has been a driving force in
transforming various domains, reshaping the artificial general intelligence landscape …

It is well known that filtering low-quality data before pretraining language models or
selecting suitable data from domains similar to downstream task datasets generally leads to …

We describe our team's contribution to the STRICT-SMALL track of the BabyLM Challenge.
The challenge requires training a language model from scratch using only a relatively small …

Answering complex logical queries over incomplete knowledge graphs (KGs) is challenging.
Most previous works have focused on learning entity/relation embeddings and simulating …

Language models strongly rely on frequency information because they maximize the
likelihood of tokens during pre-training. As a consequence, language models tend to not …