End-to-end transformer-based models in textual-based NLP

A Rahali, MA Akhloufi - AI, 2023 - mdpi.com
Transformer architectures are highly expressive because they use self-attention
mechanisms to encode long-range dependencies in the input sequences. In this paper, we …

Vision-language models in remote sensing: Current progress and future trends

X Li, C Wen, Y Hu, Z Yuan… - IEEE Geoscience and …, 2024 - ieeexplore.ieee.org
The remarkable achievements of ChatGPT and Generative Pre-trained Transformer 4 (GPT-
4) have sparked a wave of interest and research in the field of large language models …

Efficient and effective text encoding for chinese llama and alpaca

Y Cui, Z Yang, X Yao - arXiv preprint arXiv:2304.08177, 2023 - arxiv.org
Large Language Models (LLMs), such as ChatGPT and GPT-4, have dramatically
transformed natural language processing research and shown promising strides towards …

Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation

Y Sun, S Wang, S Feng, S Ding, C Pang… - arXiv preprint arXiv …, 2021 - arxiv.org
Pre-trained models have achieved state-of-the-art results in various Natural Language
Processing (NLP) tasks. Recent works such as T5 and GPT-3 have shown that scaling up …

Towards artificial general intelligence via a multimodal foundation model

N Fei, Z Lu, Y Gao, G Yang, Y Huo, J Wen, H Lu… - Nature …, 2022 - nature.com
The fundamental goal of artificial intelligence (AI) is to mimic the core cognitive activities of
human. Despite tremendous success in the AI research, most of existing methods have only …

Lilt: A simple yet effective language-independent layout transformer for structured document understanding

J Wang, L Jin, K Ding - arXiv preprint arXiv:2202.13669, 2022 - arxiv.org
Structured document understanding has attracted considerable attention and made
significant progress recently, owing to its crucial role in intelligent document processing …

Chinese clip: Contrastive vision-language pretraining in chinese

A Yang, J Pan, J Lin, R Men, Y Zhang, J Zhou… - arXiv preprint arXiv …, 2022 - arxiv.org
The tremendous success of CLIP (Radford et al., 2021) has promoted the research and
application of contrastive learning for vision-language pretraining. In this work, we construct …

Cpt: A pre-trained unbalanced transformer for both chinese language understanding and generation

Y Shao, Z Geng, Y Liu, J Dai, H Yan, F Yang… - Science China …, 2024 - Springer
In this paper, we take the advantage of previous pre-trained models (PTMs) and propose a
novel Chinese pre-trained unbalanced transformer (CPT). Different from previous Chinese …

Chinesebert: Chinese pretraining enhanced by glyph and pinyin information

Z Sun, X Li, X Sun, Y Meng, X Ao, Q He, F Wu… - arXiv preprint arXiv …, 2021 - arxiv.org
Recent pretraining models in Chinese neglect two important aspects specific to the Chinese
language: glyph and pinyin, which carry significant syntax and semantic information for …

CLUE: A Chinese language understanding evaluation benchmark

L Xu, H Hu, X Zhang, L Li, C Cao, Y Li, Y Xu… - arXiv preprint arXiv …, 2020 - arxiv.org
The advent of natural language understanding (NLU) benchmarks for English, such as
GLUE and SuperGLUE allows new NLU models to be evaluated across a diverse set of …