Benchmarking large language models on cmexam-a comprehensive chinese medical exam dataset

J Liu, P Zhou, Y Hua, D Chong, Z Tian… - Advances in …, 2024 - proceedings.neurips.cc
Recent advancements in large language models (LLMs) have transformed the field of
question answering (QA). However, evaluating LLMs in the medical field is challenging due …

The (r) evolution of multimodal large language models: A survey

D Caffagni, F Cocchi, L Barsellotti, N Moratelli… - arXiv preprint arXiv …, 2024 - arxiv.org
Connecting text and visual modalities plays an essential role in generative intelligence. For
this reason, inspired by the success of large language models, significant research efforts …

The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?

Q Zhao, M Xu, K Gupta, A Asthana, L Zheng… - arXiv preprint arXiv …, 2024 - arxiv.org
Large vision-language models (LVLMs), designed to interpret and respond to human
instructions, occasionally generate hallucinated or harmful content due to inappropriate …

Adashield: Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting

Y Wang, X Liu, Y Li, M Chen, C Xiao - arXiv preprint arXiv:2403.09513, 2024 - arxiv.org
With the advent and widespread deployment of Multimodal Large Language Models
(MLLMs), the imperative to ensure their safety has become increasingly pronounced …

Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation

Y Gou, K Chen, Z Liu, L Hong, H Xu, Z Li… - arXiv preprint arXiv …, 2024 - arxiv.org
Multimodal large language models (MLLMs) have shown impressive reasoning abilities,
which, however, are also more vulnerable to jailbreak attacks than their LLM predecessors …

Safety of Multimodal Large Language Models on Images and Text

X Liu, Y Zhu, Y Lan, C Yang, Y Qiao - arXiv preprint arXiv:2402.00357, 2024 - arxiv.org
Attracted by the impressive power of Multimodal Large Language Models (MLLMs), the
public is increasingly utilizing them to improve the efficiency of daily work. Nonetheless, the …

Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt

Z Ying, A Liu, T Zhang, Z Yu, S Liang, X Liu… - arXiv preprint arXiv …, 2024 - arxiv.org
In the realm of large vision language models (LVLMs), jailbreak attacks serve as a red-
teaming approach to bypass guardrails and uncover safety implications. Existing jailbreaks …

Direct large language model alignment through self-rewarding contrastive prompt distillation

A Liu, H Bai, Z Lu, X Kong, S Wang, J Shan… - arXiv preprint arXiv …, 2024 - arxiv.org
Aligning large language models (LLMs) with human expectations without human-annotated
preference data is an important problem. In this paper, we propose a method to evaluate the …

GradSafe: Detecting Unsafe Prompts for LLMs via Safety-Critical Gradient Analysis

Y Xie, M Fang, R Pi, N Gong - arXiv preprint arXiv:2402.13494, 2024 - arxiv.org
Large Language Models (LLMs) face threats from unsafe prompts. Existing methods for
detecting unsafe prompts are primarily online moderation APIs or finetuned LLMs. These …

SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model

Y Zhang, L Chen, G Zheng, Y Gao, R Zheng… - arXiv preprint arXiv …, 2024 - arxiv.org
The emergence of Vision Language Models (VLMs) has brought unprecedented advances
in understanding multimodal information. The combination of textual and visual semantics in …