Conversations with GUIs

In Proceedings of the ACM SIGCHI Conference on Designing Interactive Systems, DIS 2021.

Annotated datasets of application GUIs contain a wealth of information that can be used for various purposes, from providing inspiration to designers and implementation details to developers to assisting end-users during daily use. However, users often struggle to formulate their needs in a way that computers can understand reliably. To address this, we study how people may interact with such GUI datasets using natural language. We elicit user needs in a survey (N=120) with three target groups (designers, developers, end-users), providing insights into which capabilities would be useful and how users formulate queries. We contribute a labelled dataset of 1317 user queries, and demonstrate an application of a conversational assistant that interprets these queries and retrieves information from a large-scale GUI dataset. It can (1) suggest GUI screenshots for design ideation, (2) highlight details about particular GUI features for development, and (3) reveal further insights about applications. Our findings can inform design and implementation of intelligent systems to interact with GUI datasets intuitively.

Page teaser
In the Media

Press Release:
in English
in Finnish

Presentation Video:
on Youtube (6 minutes)

Twitter thread:


PDF, 2.5 MB
Kashyap Todi, Luis A. Leiva, Daniel Buschek, Pin Tian, Antti Oulasvirta. 2021. Conversations with GUIs. In Proceedings of the ACM SIGCHI Conference on Designing Interactive Systems (DIS ’21).

author = {Todi, Kashyap and Leiva, Luis A. and Buschek, Daniel and Tian, Pin and Oulasvirta, Antti},
title = {{Conversations with GUIs}},
year = {2021},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {},
doi = {10.1145/3461778.3462124},
booktitle = {Proceedings of the ACM SIGCHI Conference on Designing Interactive Systems},
keywords = {Conversational Assistants, Chatbots, GUI, Dataset, NLP, NLU},
series = {DIS '21'}}

Our survey material and labelled dataset of 1317 user queries is available in an open repository:

The source code of the Hey GUI chatbot prototype is available on GitHub:


For questions and further information, please contact:
Kashyap Todi

Acknowledgements: This work has been funded by the Finnish Center for Artificial Intelligence (FCAI), Academy of Finland projects "Human Automata" and "BAD", and the Bavarian State Ministry of Science and the Arts and coordinated by the Bavarian Research Institute for Digital Transformation (bidt).