How We Type: Eye and Finger Movement Strategies in Mobile Typing

In Proceedings of the ACM Conference on Human Factors in Computing Systems, CHI 2020.
What do you know about typing on a mobile soft keyboard?
  • Rivalry for attention between screen areas:
    • Even though mobile devices are small, one cannot monitor the keyboard and the text display at the same time. --> A strategy must be selected that determines which to give attention and when.
  • Speed-accuracy tradeoff:
    • A strategy strikes a compromise between the cost of NOT correcting errors early and the time lost in glancing at the text display, when the fingers cannot be guided. More errors --> more attention to text display --> worse WPM.
Abstract

Relatively little is known about eye and finger movement in typing with mobile devices. Most prior studies of mobile typing rely on log data, while data on finger and eye movements in typing come from studies with physical keyboards. The paper presents new findings from a transcription task with mobile touchscreen devices. Movement strategies were found to emerge in response to sharing of visual attention: attention is needed for guiding finger movements and detecting typing errors. In contrast to typing on physical keyboards, visual attention is kept mostly on the virtual keyboard, and glances at the text display are detrimental to performance. When typing with two fingers, users, while making more errors, manage to detect and correct them more quickly, which explains part of the known superiority of two-thumb typing over one-finger typing. We release the extensive dataset on everyday typing on smartphones.

Materials

All data and videos are open for anyone to use.

The HOW-WE-TYPE-MOBILE Dataset
Data
  • Finger movement data with x-y-z coordinates of the markers on the index finger (one-finger typing), and both thumbs (two-finger typing).
  • Gaze data with x-y coordinates mapped onto the smartpone screen.
  • Typing log marked with conditions and target sentence.
    • Videos
    Press Releases
    1. in English
    2. in Finnish

    Publication
    paper

    PDF, 3.1 MB
    Jiang, X., Li, Y., Jokinen, J. P. P., Hirvola, V., Oulasvirta, A., & Ren, X. 2020. How We Type: Eye and Finger Movement Strategies in Mobile Typing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). 

    	    
@inproceedings{Jiang2020,
    author = {Jiang, Xinhui and Li, Yang and Jokinen, Jussi P.P. and Hirvola, Viet Ba and Oulasvirta, Antti and Ren, Xiangshi},
    title = {How We Type: Eye and Finger Movement Strategies in Mobile Typing},
    year = {2020},
    isbn = {9781450367080},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3313831.3376711},
    doi = {10.1145/3313831.3376711},
    booktitle = {Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems},
    pages = {1–14},
    numpages = {14},
    keywords = {eye-hand coordination, text input, eye movement, mobile device, finger movement},
    location = {Honolulu, HI, USA},
    series = {CHI ’20}
}
    Errata

    This Errata lists an error and its correction for this paper.

    Location: Page 8, Figure 7
    Error: Wrong figure and title
    Correction:

    Figure 7: Eye–hand distance and average WPM per sentence
    Figure 7: Eye–hand distance and average WPM per sentence

    Frequently Asked Questions
    Q: What kind of setup did you have for this? real phone? head-worn eye tracker?
    A: Yes, we used a head-worn eye tracker (SMI W2), marker-based optical motion tracking (OptiTrack Prime13), and key logging. --> Synchronized and registered in the same coordinate system.

    Q: Will typing posture affect performance?
    A: Yes. There are different constraints for typing with different postures. For example, in two-thumb typing, the movement range for the thumbs is limited as the hand is also used for holding the device. So the user has to adapt their finger movement and finger-key mapping strategy to the postures. Also, an unfamiliar posture will slow down the typing. So, in our study, we applied two of the most frequently used posture for typing (one-index-finger typing and two-thumb typing), in order to eliminate the effect of unfamiliar postures.

    Q: Is it related to the skill of PC touch typing?
    A: To some extent, yes. As both physical keyboard and touchscreen keyboard have a QWERTY layout, the user could take advantage of this knowledge and use them across different devices. According to an ongoing study about fast typists, the fast typists on a touchscreen keyboard also showed a good performance on the physical keyboard.

    Q: How accurate is this eye-tracking?
    A: The accuracy of the eye tracker is within 0.5°. For analysis purposes, we mapped the gaze points to the smartphone screen area using spatial transformation. We manually checked the data quality, which looks reasonable. Adding that the main interest for us is to see the area of interest during typing, like keyboard area and the text-input area, it can provide enough information in an acceptable accuracy.

    Q: Did you observe any differences between typing and gesturing on a touchscreen keyboard?
    A: No, our study didn't cover the gesture typing condition. But as we can see, the gesture typing is different from touch-based typing because the device will interpret the whole finger movement path as an input. In that case, all the finger movement on the screen has a meaning. The user has to be more careful about the finger control. But in touch-based typing, the fingers are free to move in the area before touching the screen. As the constraints are different, the performance may also be different.
    Contact

    For questions and further information, please contact:

    Xinhui Jiang

    Email:
    jiangxinhui0077@gmail.com

    Jussi Jokinen

    Email:
    jussi.jokinen@aalto.fi

    This work has received funding from the joint JST–AoF project "User Interface Design for the Ageing Population" (AoF grant 291556) as an activity of FY2014 Strategic International Collaborative Research Program (SICORP), and from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement 637991).