Screen recognition: Creating accessibility metadata for mobile applications from pixels

X Zhang, L De Greef, A Swearngin, S White… - Proceedings of the …, 2021 - dl.acm.org
Many accessibility features available on mobile platforms require applications (apps) to
provide complete and accurate metadata describing user interface (UI) components …

Webui: A dataset for enhancing visual ui understanding with web semantics

J Wu, S Wang, S Shen, YH Peng, J Nichols… - Proceedings of the …, 2023 - dl.acm.org
Modeling user interfaces (UIs) from visual information allows systems to make inferences
about the functionality and semantics needed to support use cases in accessibility, app …

Machine learning-based prototyping of graphical user interfaces for mobile apps

K Moran, C Bernal-Cárdenas, M Curcio… - IEEE Transactions …, 2018 - ieeexplore.ieee.org
It is common practice for developers of user-facing software to transform a mock-up of a
graphical user interface (GUI) into code. This process takes place both at an application's …

Reverse engineering mobile application user interfaces with remaui (t)

TA Nguyen, C Csallner - 2015 30th IEEE/ACM International …, 2015 - ieeexplore.ieee.org
When developing the user interface code of a mobile application, in practice a big gap exists
between the digital conceptual drawings of graphic artists and working user interface code …

Screen parsing: Towards reverse engineering of ui models from screenshots

J Wu, X Zhang, J Nichols, JP Bigham - The 34th Annual ACM …, 2021 - dl.acm.org
Automated understanding of user interfaces (UIs) from their pixels can improve accessibility,
enable task automation, and facilitate interface design without relying on developers to …

A systematic literature review of applications of the physics of notations

D Van Der Linden, I Hadar - IEEE Transactions on Software …, 2018 - ieeexplore.ieee.org
INTRODUCTION: The Physics of Notations (PoN) is a theory for the design of cognitively
effective visual notations, emphasizing the need for design grounded in objective and …

It takes two to tango: Combining visual and textual information for detecting duplicate video-based bug reports

N Cooper, C Bernal-Cárdenas… - 2021 IEEE/ACM …, 2021 - ieeexplore.ieee.org
When a bug manifests in a user-facing application, it is likely to be exposed through the
graphical user interface (GUI). Given the importance of visual information to the process of …

App store effects on software engineering practices

AA Al-Subaihin, F Sarro, S Black… - IEEE Transactions …, 2019 - ieeexplore.ieee.org
In this paper, we study the app store as a phenomenon from the developers' perspective to
investigate the extent to which app stores affect software engineering tasks. Through …

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Y Jiang, E Schoop, A Swearngin, J Nichols - arXiv preprint arXiv …, 2023 - arxiv.org
Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused
understanding of images and language, but many perform poorly on UI tasks due to the lack …

Who will leave the company?: a large-scale industry study of developer turnover by mining monthly work report

L Bao, Z Xing, X Xia, D Lo, S Li - 2017 IEEE/ACM 14th …, 2017 - ieeexplore.ieee.org
Software developer turnover has become a big challenge for information technology (IT)
companies. The departure of key software developers might cause big loss to an IT …