The high computational and memory requirements of generative large language models
(LLMs) make it challenging to serve them cheaply. This paper aims to reduce the monetary …

As deep learning model sizes expand and new GPUs are released every year, the need for
distributed training on heterogeneous GPUs rises to fully harness under-utilized low-end …

Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with
their sophisticated capabilities. Training these models requires vast GPU clusters and …

Distributed full-graph training of Graph Neural Networks (GNNs) has been widely adopted to
learn large-scale graphs. While recent system advancements can improve the training …

Large-scale deep learning models contribute to significant performance improvements on
varieties of downstream tasks. Current data and model parallelism approaches utilize model …

Recent foundation models are capable of handling multiple machine learning (ML) tasks
and multiple data modalities with the unified base model structure and several specialized …

Training large language model (LLM) is a computationally intensive task, which is typically
conducted in data centers with homogeneous high-performance GPUs. This paper explores …

Large language models (LLMs) iteratively generate text token by token, with memory usage
increasing with the length of generated token sequences. The unpredictability of generation …

Navigating the intricacies of data management in the era of Large Language Models (LLMs)
presents both challenges and opportunities for database and data management …

Training transformer models requires substantial GPU compute and memory resources. In
homogeneous clusters, distributed strategies allocate resources evenly, but this approach is …