AutoTask: Executing Arbitrary Voice Commands by Exploring and Learning from Mobile GUI

L Pan, B Wang, C Yu, Y Chen, X Zhang… - arXiv preprint arXiv …, 2023 - arxiv.org
Voice command interfaces (VCIs) have gained increasing importance, enabling hands-free
and eyes-free interaction with digital devices. However, the inherent complexity in …

GPTVoiceTasker: LLM-powered virtual assistant for smartphone

MD Vu, H Wang, Z Li, J Chen, S Zhao, Z Xing… - arXiv preprint arXiv …, 2024 - arxiv.org
Virtual assistants have the potential to play an important role in helping users achieves
different tasks. However, these systems face challenges in their real-world usability …

GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning

MD Vu, H Wang, J Chen, Z Li, S Zhao, Z Xing… - Proceedings of the 37th …, 2024 - dl.acm.org
Virtual assistants have the potential to play an important role in helping users achieves
different tasks. However, these systems face challenges in their real-world usability …

VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning

Y Song, Y Bian, Y Tang, G Ma, Z Cai - Proceedings of the 37th Annual …, 2024 - dl.acm.org
Mobile task automation is an emerging field that leverages AI to streamline and optimize the
execution of routine tasks on mobile devices, thereby enhancing efficiency and productivity …

Navigating Interfaces with AI for Enhanced User Interaction

Y Song, Y Bian, Y Tang, Z Cai - arXiv preprint arXiv:2312.11190, 2023 - arxiv.org
This study introduces an innovative framework designed to automate tasks by interacting
with UIs through a sequential, human-like problem-solving approach. Our approach initially …

PromptRPA: Generating Robotic Process Automation on Smartphones from Textual Prompts

T Huang, C Yu, W Shi, Z Peng, D Yang, W Sun… - arXiv preprint arXiv …, 2024 - arxiv.org
Robotic Process Automation (RPA) offers a valuable solution for efficiently automating tasks
on the graphical user interface (GUI), by emulating human interactions, without modifying …

Test2VA: Reusing GUI Test Cases for Voice Assistant Features Development in Mobile Applications

G Weaver, X Qin - arXiv preprint arXiv:2407.18155, 2024 - arxiv.org
Voice Assistant (VA) in smartphones has become very popular with millions of users
nowadays. A key trend is the rise of custom VA embedding, which enables users to perform …

Building Python Application for Webmail Interfaces Navigation Using Voice Recognition Technology

M Alkhattali, M Dow, K Azwee… - International Journal of …, 2023 - papers.ssrn.com
Abstract Voice Recognition Technology (VRT) has played a crucial role in technology
development, finding extensive use in the development of humanitarian assistance …

iTutor: A Generative Tutorial System for Teaching the Elders to Use Smartphone Applications

R Zou, Z Ye, C Ye - Adjunct Proceedings of the 36th Annual ACM …, 2023 - dl.acm.org
We present iTutor, a generative tutorial system for promoting smartphone use proficiency
among elders. iTutor is unique because it can dynamically generate tutorials based on …

[PDF][PDF] EcoTarium Explorer: Increasing Accessibility at the Worcester EcoTarium

J Fox - 2024 - digital.wpi.edu
Today's technology provides a significant opportunity for those with accessibility needs to
feel more connected to the world around them. Museums, zoos, and similar institutions have …