Neuromechanics of a Button Press

In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2018), ACM Press.
Aalto University User Interfaces group
KAIST Interactive Media Lab
button

Pressing a button appears easy, but the brain needs a probabilistic internal model to control a press. A new theory exposes significant improvements to button design that help gamers and musicians. The project was triggered by admiration of our remarkable capability to adapt button-pressing It created detailed simulations of button-pressing with the goal of producing human-like presses. People push a button on a remote controller differently than a piano key. The press of a skilled user is surprisingly elegant when looked at terms of timing, reliability, and energy use. People successfully press buttons without ever knowing the inner workings of a button. It is essentially a black box to our motor system. On the other hand, we also fail to activate buttons, and some buttons are known to be worse than others. To improve buttons, we need to understand the motor control processes and the effects of design parameters like button dimensions, materials, force curves, activation point, and feedback.

Press releases

Article and Citation
paper
Best Paper Honorable Mention

PDF (3.2 MB)
Oulasvirta, A., Kim, S., & Byungjoo, L. 2018.
Neuromechanics of a Button Press
In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), ACM Press, pp. 4099-4112.
Best Paper Honorable Mention


Abstract: To press a button, a finger must push down and pull up with the right force and timing. How the motor system succeeds in button-pressing, in spite of neural noise and lacking direct access to the mechanism of the button, is poorly understood. This paper investigates a unifying account based on neuromechanics. Mechanics is used to model muscles controlling the finger that contacts the button. Neurocognitive principles are used to model how the motor system learns appropriate muscle activations over repeated strokes though relying on degraded sensory feedback. Neuromechanical simulations yield a rich set of predictions for kinematics, dynamics, and user performance and may aid in understanding and improving input devices. We present a computational implementation and evaluate predictions for common button types. 

    	    
    @inproceedings{oulasvirta2018neuromechanics,
         title = {Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18)},
         author = {Oulasvirta, Antti and Kim, Sunjun and Lee, Byungjoo},
         title = {Neuromechanics of a Button Press},
         pages = {4099--4112},
         year = {2018},
         doi = {10.1145/3173574.3174082}
         organization = {ACM Press}}
Overview of theory
Neuromechanics views button-pressing as optimal control limited by physical, neural, and sensory bounds. It proceeds from the biological fact that the central nervous system has no direct access to the physical body (here, finger) it is moving. It is enveloped by neural interfaces. Our account of the motor controller rests on the convergence of two principles:

Perceptual control: With no direct access to the button, the motor system relies on its limited sensations to adapt the motor commands. We formulate a perceptual control objective for button-pressing: the goal of the motor system is to control its own sensations arising as a consequence of the button press. In particular, we model timing estimates that a user makes based on noisy sensory signals.

Probabilistic internal model: To control a button, the brain must learn to predict the consequences of the actions it takes. We build on previous work on predictive processing and the Bayesian brain hypothesis to posit a probabilistic internal model. It tries to learn to predict the perceptual effects that its open-loop commands have over episodes of attempts with a button.

NEUROMECHANIC: A computational implementation of the model
Improving button designs

Impact Activation is a new technique to activate push- and touch buttons

One implication of the theory is that activating the button at the moment when the sensation is strongest will help users better rhythm their keypresses. To test this hypothesis, we created a new method for changing the way buttons are activated. The technique, published at CHI 2018, is called Impact Activation. Instead of activating the button at first contact, it activates it when the button cap or finger hits the floor with maximum impact. The technique was 94% better in rapid tapping than the regular activation method for a push-button (Cherry MX switch) and 37% than a regular touchscreen button using a capacitive touch sensor.

Code: NEUROMECHANIC

MATLAB Neuromechanic

NEUROMECHANIC is a computational implementation released as a modeling workbench used in MATLAB. Button designs and user tasks can be explored by manipulating parameters as presented above in the Simulations section of paper. After optimization, the model outputs such values as means and statistical indices for task-level performance (button activation and perceptual error), cognition (p-centers), and dynamics and kinematics (FD, DV, pulp contraction, and muscle forces). A visual simulation of the dynamics is available also. A database of 20 FD curves of commercial button types is available too (see Data).

Data


Single subject human data

Media: Photography and videos of button-pressing


Pushing a button
These media are licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0 or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
Contact

For more information, please contact:

Antti Oulasvirta

Email:
antti.oulasvirta (at) aalto.fi

Phone:
+358 (09) 47001

Acknowledgements: This work was funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement 637991) and by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Planning (NRF-2017R1C1B2002101). We thank Jong-In Lee for help with the video.