Computational Interaction: Theory and Practice

Otmar Hilliges

ETH Zurich, Switzerland

Per Ola Kristensson

Cambridge University, United Kingdom

Antti Oulasvirta

Aalto University, Finland

John Williamson

University of Glasgow, United Kingdom

Computational Interaction: Theory and Practice

This course introduces computational methods for HCI.

As interfaces become more sophisticated, designing and adapting them requires an exponentially expanding set of design decisions. Computational methods can be used to synthesise elements of interfaces, to learn interaction structure from observations and to infer user intentions in a noisy world. Computational approaches empower HCI researchers to building sophisticated, robust interfaces quickly and reliably.

Summary & objectives

The course will cover:

• Optimisation: algorithmically finding interface configurations which satisfy constraints and maximize user-related criteria.
• Inference: a principled and robust approach to designing a transformation from input to useful action.
• Learning: principled ways to automatically learn and improve performance from data without explicit programming.

This course will:

• demonstrate how computational approaches can turn new technologies into viable interfaces while reducing inconsistency;
• extend researchers' capabilities to build robust interactions across a wide range of contexts and devices;
• show how computational approaches can focus interaction design on the interesting work of specifying the questions, and letting computational methods resolve the answers.

Pre-requisites

To get the most out of this course, participants should have working knowledge of undergraduate mathematics, particularly basic probability and linear algebra (even if it is a vague and rusty memory!).

Software and preparation

All of the materials will be delivered as interactive Jupyter notebooks, which interleave notes and live coding examples. All coding examples will be in Python, and familiarity with Python is strongly recommended. We also provide the materials for download and use on your own machine before and after the conference course sessions.

Layouts design with combinatorial optimisation

Antti Oulasvirta
This part introduces combinatorial optimisation methods: algorithmically searching for combinations of design decisions that best fulfill given goodness criteria. After an overview, three hands-on exercises focus on core requirements: definition of design tasks as search problems, algorithmic approaches to searching the defined design spaces, and developing objective functions using design knowledge and theories.

Participants will:

• learn basic skills in formulating UI design problems such that an algorithm can help in exploring it;
• understand the requirements of optimisation approach;
• understand how it can be tied to user-centered design and data-driven design.

Vision-based interaction with deep learning

Otmar Hilliges
Recovering intent from low-level sensor data is a core problem in many interactive systems. This section of the course will introduce the participants to the basic problems and available solutions surrounding human action recognition from sensor data using deep learning. After an overview of relevant HCI problems we will give an introduction to succesful machine-learning models and how to apply these to common tasks such as static and dynamic gesture recognition.

Participants will:

• understand which types of HCI problems lend themselves well to data-driven approaches
• know how to collect and label data for training and testing
• have a basic understanding of deep learning applied in this context
• understand successful models (in the HCI context) in more detail.

Probabilistic decoding for intelligent text entry

Per Ola Kristensson
This section will view text entry design as fundamentally a problem of decoding the user's intended text from noisy observations from a variety of sensors, such as touchscreens or depth sensors. The section will introduce key concepts in statistical language processing and give an overview of how to design and implement a probabilistic decoder for text entry. The section will illustrate how such decoding enable the design of several new text entry methods for a variety of use cases.

Participants will

• understand how designing and implementing probabilistic decoders can result in faster, more flexible and more accurate text entry systems;
• be aware of key design decisions that affect decoding performance.

Citing

To cite this course, use the following BibTeX entry:

                    @inproceedings{williamson2018computational,
                      title={Computational Interaction: Theory and Practice},
                      author={Williamson, John and Oulasvirta, Antti and Hilliges,
                              Otmar and Kristensson, Per Ola},
                      booktitle={Proceedings of the 2018 CHI Conference Extended Abstracts
                                 on Human Factors in Computing Systems},
                      year={2018},
                      organization={ACM}
                    }