Ability-Based Optimization of Touchscreen Interactions

IEEE Pervasive Computing, 2018
AI improves touchscreen interfaces for impaired users

User interfaces are usually designed to an average user. However, some users have disabilities or deficits that make standard user interfaces difficult to use. We propose interface optimization as a tool to solve this problem. Interface optimization means changing the user interface to suit the individual needs of the user. Our proposed optimization process has many benefits:


  • The optimization improves performance by reducing errors and improving task efficiency
  • User interfaces can be customized to different impairments like dyslexia, tremor, or memory deficits
  • The optimizer works without extensive amounts of training data
  • The customization is automatic
The layput optimizing process

Example problems that can be resolved with interface optimization:

  • Essential tremor hinders the user’s ability to point accurately, leading to enormous difficulties with touch-based interfaces. An optimizer can increase the size of the user interface elements and group functions together to adapt to the screen size constraints.
  • Dyslexia makes proofreading typed text and reading words of the user interface more time consuming and error-prone. An optimizer can adjust the number of text in the user interface and introduce aids for making sure that the typed text is correct.
  • Dementia decreases the ability to think and remember, making the use of most everyday user interfaces difficult or impossible. An optimizer can suggest designs, which minimize the memory load and require as little previous knowledge from the user as possible. They prioritize frequent or important tasks.
Abstract

The paper examines a computational design approach for improving user interface designs for people with sensorimotor and cognitive impairments. In ability-based optimization, designs are created by an optimizer and evaluated against model of an individual performing tasks. Alternative designs can be explored and adapted to an individual’s abilities. In this paper, we explore text entry on touchscreen devices as the case. Individual abilities are parametrically expressed as part of a task-specific cognitive model, and the model estimates how the individual might adapt her interaction to the task. Optimized designs can potentially improve speed and reduce error for users with tremor and dyslexia. Ability-based optimization does not necessitate extensive data-collection and could be applied both automatically and manually by users, designers, or caretakers.

The layout optimizing process
Press Releases
  1. in English
  2. in Finnish

Publication
paper

PDF, 0,5 MB
Sarcar, S., Jokinen, J., Oulasvirta, A., Wang, Z., Silpasuwanchai, C., Ren, X., 2018.
Ability-Based Optimization of Touchscreen Interactions.
IEEE Pervasive Computing 17.1, pp. 15-26. 

	    
@article{sarcar2018ability,
  title={Ability-Based Optimization of Touchscreen Interactions},
  author={Sarcar, Sayan and Jokinen, Jussi PP and Oulasvirta, Antti and Wang, Zhenxin and Silpasuwanchai, Chaklam and Ren, Xiangshi},
  journal={IEEE Pervasive Computing},
  volume={17},
  number={1},
  pages={15--26},
  year={2018},
  publisher={IEEE}
}

A link to the IEEE Pervasive Computing vol. 17 issue 1 website.

Material

Data:

Media:

  • The design space for ordinary tasks, such as typing, can easily get very large. In such cases, computational optimization with a psychological user model is the only feasible way to design interfaces for individual abilities.
    Design space Design space

  • Changing the parameters of the model corresponds to having users with different abilities. This results in both subtle and large changes in the optimal layout design, as seen in these test results.
    Design space

  • These videos show the model's simulation of typing for two different users. The first video simulates an average user without impairments. In the second video, the parameters of the typing model have been changed to correspond to a person with essential tremor. Red circle shows the gaze location and yellow shows the location of the finger.
Frequently Asked Questions

Q: Can the optimizer be extended to other kinds of impairments than dyslexia, memory deficits and tremor?
A: Any impairment that can be specified and parameterized with a generative model can be targeted. In addition, the design space must of course contain designs that increase performance for users with these impairments. If these conditions are met, the optimizer can be used to design layouts for the user with the given impairment(s).

Q: How much is performance improved with the optimized layouts?
A: The predicted improvement with the optimized layout and for the assumed user with tremor was about 16% increase of typing speed, and even more importantly, the prohibitive error rate (over half of the typed keys are incorrect) was decreased to an acceptable amount. In the pilot user tests, one of the user learned to use the optimized layout with typing speed increase of 14% and error rate decrease from 7% to 3%. The other pilot user did not learn to use the layout efficiently during the 30 minute experiment, although even for them, the error rate did decrease.

Q: Can the optimizer be calibrated to an individual user?
A: Yes. This requires either specifically tailored calibration tasks or online user data collection. In the paper, we focused on a stereotypical user with a fairly clearly defined impairment. However, an interactive system with a built-in model of the user and an optimizer can first be used to calibrate the model and then optimize the system.
Contact

For questions and further information, please contact:

Jussi P.P. Jokinen

Email:
jussi.jokinen (at) aalto.fi

Phone:
+358 45 196 1429




Sayan Sarcar

Email:
sayan.sarcar (at) kochi-tech.ac.jp

Phone:
+81 90 94513410





Xiangshi Ren

Email:
xsren (at) acm.org

Acknowledgements:This work has received funding from the joint JST–AoF project "User Interface Design for the Ageing Population" (AoF grant 291556) as an activity of FY2014 Strategic Inter- national Collaborative Research Program (SICORP), and from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement 637991).