Understanding design collaboration between designers and artificial intelligence: a systematic literature review

Y Shi, T Gao, X Jiao, N Cao - Proceedings of the ACM on Human …, 2023 - dl.acm.org
Recent interest in design through the artificial intelligence (AI) lens is rapidly increasing.
Designers, as a special user group interacting with AI, have received more attention in the …

Never-ending learning of user interfaces

J Wu, R Krosnick, E Schoop, A Swearngin… - Proceedings of the 36th …, 2023 - dl.acm.org
Machine learning models have been trained to predict semantic information about user
interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently …

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Y Jiang, E Schoop, A Swearngin, J Nichols - arXiv preprint arXiv …, 2023 - arxiv.org
Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused
understanding of images and language, but many perform poorly on UI tasks due to the lack …

BrushLens: hardware interaction proxies for accessible touchscreen interface actuation

C Liang, Y Iravantchi, T Krolikowski, R Geng… - Proceedings of the 36th …, 2023 - dl.acm.org
Touchscreen devices, designed with an assumed range of user abilities and interaction
patterns, often present challenges for individuals with diverse abilities to operate …

AI Assistance for UX: A Literature Review Through Human-Centered AI

Y Lu, Y Yang, Q Zhao, C Zhang, TJJ Li - arXiv preprint arXiv:2402.06089, 2024 - arxiv.org
Recent advancements in HCI and AI research attempt to support user experience (UX)
practitioners with AI-enabled tools. Despite the potential of emerging models and new …

Generating Automatic Feedback on UI Mockups with Large Language Models

P Duan, J Warner, Y Li, B Hartmann - … of the CHI Conference on Human …, 2024 - dl.acm.org
Feedback on user interface (UI) mockups is crucial in design. However, human feedback is
not always readily available. We explore the potential of using large language models for …

Interactive Flexible Style Transfer for Vector Graphics

J Warner, KW Kim, B Hartmann - Proceedings of the 36th Annual ACM …, 2023 - dl.acm.org
Vector graphics are an industry-standard way to represent and share visual designs.
Designers frequently source and incorporate styles from existing designs into their work …

Learning from unique perspectives: User-aware saliency modeling

S Chen, N Valliappan, S Shen, X Ye… - Proceedings of the …, 2023 - openaccess.thecvf.com
Everyone is unique. Given the same visual stimuli, people's attention is driven by both
salient visual cues and their own inherent preferences. Knowledge of visual preferences not …

UniAR: Unifying Human Attention and Response Prediction on Visual Content

P Li, J He, G Li, R Bhargava, S Shen… - arXiv preprint arXiv …, 2023 - arxiv.org
Progress in human behavior modeling involves understanding both implicit, early-stage
perceptual behavior such as human attention and explicit, later-stage behavior such as …

Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM

Z Liu, C Chen, J Wang, M Chen, B Wu… - Proceedings of the CHI …, 2024 - dl.acm.org
Mobile apps have become indispensable for accessing and participating in various
environments, especially for low-vision users. Users with visual impairments can use screen …