Finger-Aware Shortcuts

Trigger different shortcut commands by pressing the same key with different fingers, hands, or hand postures.

Finger-Aware Shortcuts Concepts
Finger-Aware Shortcuts Concepts
Finger-Aware Shortcuts Concepts
Finger-Aware Shortcuts Concepts
  1. 1

    People can use different hand postures to press the same key quickly and accurately across the keyboard using both hands.

  2. 2

    People like the index finger and the thumb with other fingers closed most. They are the fastest too!

  3. 3

    People don't like the ring finger and the middle finger both with other fingers closed. You could probably imagine why.

  4. 4

    Using different postures is as fast as different modifier keys when pressing the same key to activate commands!

We evaluate and demonstrate finger, hand, and posture identification as keyboard shortcuts. By detecting the hand and finger used to press a key, and open or closed hand postures, a key press can have multiple command mappings. A formative study reveals performance and preference patterns when using different fingers and postures to press a key. The results are used to develop a computer vision algorithm to identify fingers and hands on a keyboard captured by a built-in laptop camera and reflector. This algorithm is built into a background service to enable system-wide Finger-Aware Shortcut keys in any application. A controlled experiment uses the service to compare the performance of Finger-Aware Shortcuts with existing methods. The results show Finger-Aware Shortcuts are comparable with a common class of shortcuts using multiple modifier keys. Finally, application demonstrations illustrate different use cases and mappings for Finger-Aware Shortcuts and extend the idea to two-handed key presses, continuous parameter control, and menu selection.

A formative study

We let our participants use 20 different postures (2 hands x 5 fingers x 2 poses) to press different areas of the keyboard:

Hand Posture: Left Thumb, Hand Closed

Left Thumb, Hand Closed

Hand Posture: Left Index, Hand Closed

Left Index, Hand Closed

Hand Posture: Left Middle, Hand Closed

Left Middle, Hand Closed

Hand Posture: Left Ring, Hand Closed

Left Ring, Hand Closed

Hand Posture: Left Little, Hand Closed

Left Little, Hand Closed

Hand Posture: Left Thumb, Hand Open

Left Thumb, Hand Open

Hand Posture: Left Index, Hand Open

Left Index, Hand Open

Hand Posture: Left Middle, Hand Open

Left Middle, Hand Open

Hand Posture: Left Ring, Hand Open

Left Ring, Hand Open

Hand Posture: Left Little, Hand Open

Left Little, Hand Open

People can do this accurately: we had an error rate of only 1.9%. We also looked at the time and preference of each posture:

Formative Study Results
Formative Study Results
Formative Study Results

A 3-tier posture selection guideline

There are symmetric patterns across the two hands, so we concluded with a 3-tier posture guideline in terms of finger and hand pose:

  • Recommended: Index (Open, Closed), Thumb (Closed)
  • Intermediate: Little (Open, Closed), Ring (Open), Middle (Open), Thumb (Open)
  • Avoided: Ring (Closed), Middle (Closed)

Note: Participants with bigger hands complained the thumb with hand open hitting the screen when pressing the upper part of the keyboard. Therefore it's ruled out from the recommended posture list. The middle finger with hand closed has bad connotations in many cultures, strong enough in the interview to be put into the avoided posture list.

We mounted a mirror to reflect the light path of the built-in RGB camera in the laptop to watch the rectified keyboard and applied computer vision and machine learning techniques to identify the hand, finger, and pose.

In short, we used

  • Gaussian naïve Bayes classifier to isolate hand contour
  • Yoruk et al.’s approach for fingertip localization
  • Hand and finger identification with $1 recognizer and heuristics
  • Kalman filters to track fingertips over frames
  • Cocoa event interception to modify system-wide key press behaviour

Keyboard shortcut mappings

We compared Finger-Aware Shortcuts against 3 conventional keyboard shortcut mappings. We mapped two set of 4-character words to different keyboard shortcut activation methods:

Different Keys (3KEY)

  • edge → E
  • envy → R
  • exam → K

Note: This is widely used in applications that require little text entry, e.g., Adobe Photoshop.

Different Keys + Same Command Modifier Key (3KEY1MOD)

  • wool → Command ⌘ + W
  • wack → Command ⌘ + E
  • whim → Command ⌘ + K

Note: This is to simulate the most common keyboard shortcut mapping, where the most frequent command is mapped to Command + the first letter of the command name, e.g. Command + C for Copy. Then you run out of the key, you need to map other commands that start with the same letter to other keys, e.g., Command + X for Cut.

Same Key + Different Modifier Keys (1KEY3MOD)

  • ruby → Command ⌘ + R
  • rack → Command ⌘ + Shift ⇧ + R
  • reef → Option ⌥ + R

Note: This is usually used to map a family of related commands such as Command + F for Find, Command + Shift + F for Find in Project, and Option + F for Find and Replace in a text editor.

Same Key + Different Finger-Aware Postures (FINGERAWARE)

  • kite → Right Index Finger, Open Hand + K
  • kart → Right Index Finger, Closed Hand + K
  • keel → Right Middle Finger, Open Hand + K

Note: 2 postures were selected from the recommended posture list and 1 posture was selected from the intermediate posture list.

Experiment Task

The participants were shown the command and they were required to activate the corresponding keyboard shortcut after doing a text entry task or a trackpad pointing task.

There was a cheat sheet that could be activated by the space key to help the participant memorize the mappings.

Comparison

We found that Finger-Aware Shortcuts are comparable to the commonly used Same Key + Different Modifier Keys in terms of both error rate and speed. This is encouraging considering

  • Finger-Aware Shortcuts are new to the participants. It takes time to develop motor memory.
  • False negatives of the hand tracking algorithm caused high frustration.
  • Same Key + Different Keys mapping is a well practised task.
Performance Study Results

Finger-Aware Shortcuts can be used in a variety of applications.

Alternate Modifiers

Certain postures could be treated as a modifier key. For example, you could use the right thumb with hand closed to press the C key to replace Command + C. You could also use the right index finger with hand closed to press the B key to replace Command + Shift + B.

Complementary Commands

A family of commands such as changing the heading style in a text editor to Header 1, Header 2, and Header 3 could be activated using the same H key with different fingers.

Up/Down Control

Two fingers could activate adjust up and down of any related value. For example, you can adjust the text size using your left hand index and little finger with hand closed.

Continuous Input

Not limited to discrete input, Finger-Aware Shortcuts can be further extended to continuous input. For instance, you can adjust the RGB value of a colour by pressing R, G, or B keys using the right index finger and an extended thumb. Then using the thumb to adjust these values.

Context Menu

Similar to the gesture in the continuous input demo, you can select items in a context menu.

Two-Handed Posture

You can also use two hands to press the same key together to activate some rare commands such as Log Out.

  Jingjie Zheng, Daniel Vogel. 2016. Finger-Aware Shortcuts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 4274-4285. DOI=http://dx.doi.org/10.1145/2858036.2858355

@inproceedings{Zheng:2016:FS:2858036.2858355,
    author = {Zheng, Jingjie and Vogel, Daniel},
    title = {Finger-Aware Shortcuts},
    booktitle = {Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems},
    series = {CHI '16},
    year = {2016},
    isbn = {978-1-4503-3362-7},
    location = {Santa Clara, California, USA},
    pages = {4274--4285},
    numpages = {12},
    url = {http://doi.acm.org/10.1145/2858036.2858355},
    doi = {10.1145/2858036.2858355},
    acmid = {2858355},
    publisher = {ACM},
    address = {New York, NY, USA},
    keywords = {finger identification, keyboard shortcuts},
} 
        

If you have any question, feel free to contact:

Jingjie Zheng

  jingjie.zheng [at] uwaterloo.ca

M.Math. Student, David R. Cheriton School of Computer Science, University of Waterloo

Daniel Vogel

  dvogel [at] uwaterloo.ca

Assistant Professor, David R. Cheriton School of Computer Science, University of Waterloo