It's Time for Better No-Hands Computer Interfaces

by . .

computing hands linguistics rsi

Introduction

I have typing-induced RSI aggravated by a fall off a bicycle. I've struggled for months with available voice-input software. I've been forced to place nearly all of my personal projects on hold, and I'm uncertain how I will perform a normal job without further degrading my hands.

I will not stand for this. Computers can do anything. My mission in 2021 is to build the best possible no-hands interface for textual work at the computer.

I'm currently seeking funding which would allow my mission to become my job. Failing that, I will at least find mentors to guide me as I create this interface myself.

Hundreds of Thousands of Injuries

It's hard to get a straight answer on how many people suffer injuries that would impair normal use of a keyboard and mouse.

For my purposes here, I'll offer two approximations of a lower bound on potential customers.

Injury Prevention (academic journal)

Global rates of bony hand trauma have decreased slightly over the last 27 years (table 1). In 2017, an estimated 178.9 (95% UI [uncertainty interval] 145.8 to 216.8) age-standardised hand and wrist fractures per 100,000 individuals occurred worldwide, representing a 2.6% decrease from 1990.

https://injuryprevention.bmj.com/content/26/Supp_1/i115.

This figure, 178.9/100k people, is the age-standardized incident rate (ASIR) for the whole world. The only North America figure (from 2017) is the ASIR for bony hand trauma in "High Income North America." That ASIR is 210.7 (95% UI 167.4 to 262.8).

The US Census currently estimates US population at ~330,786,186.

Absent other information, I'll assume that the "High Income" subpopulation has bony hand trauma at the same rate as other income brackets. Multiply US population by the ASIR of 210.7/100,000 ...

(* 330786186 (/ 210.7 100000))

... and we get 696,966 -- just shy of 700,000 US citizens -- with bony hand trauma.

Bureau of Labor Statistics

In 2019, the BLS reported 339,800 cases of workplace injury to the upper extremities (shoulder, arm, wrist, and hand.) Here's an abridged version of a table from their data reporting tool. Workplace injuries alone total ~340,000 cases. How many more happen outside work?

Number of nonfatal occupational injuries and illnesses involving days away from work by selected worker and case characteristics and part of body, All U.S., all ownerships, 2019
Characteristic All parts of body UPPER EXTREMITIES
(code 4XXXXX)
2019 2019

Total:

 1,108,300 339,800

Sex:

Men

 662,740 210,230

Women

 441,500 128,520

Nature of injury, illness:

Fractures

 100,480 42,050

Sprains, strains, tears

 372,240 100,500

Amputations

 6,250 6,100

Cuts, lacerations, punctures

 100,400 69,120

Cuts, lacerations

 78,960 56,560

Punctures (except gunshot wounds)

 21,450 12,560

Bruises, contusions

 99,070 21,260

Chemical burns and corrosions

 4,740 840

Heat (thermal) burns

 16,650 8,920

Multiple traumatic injuries

 32,380 3,920

With sprains and other injuries

 17,710 1,710

With fractures and other injuries

 4,350 1,090

Soreness, pain

 212,190 49,210

Carpal tunnel syndrome

 4,630 4,630

Tendonitis

 1,670 1,360

All other

 157,600 31,900

Not Just Carpal Tunnel, Not Just RSI

Carpal Tunnel Syndrome is the most well-known repetitive motion/strain injury (RMI/RSI), but there are others, like Cubital Elbow Syndrome, or De Quervain’s Tendosynovitis (inflammation of the thumb tendons, aka "texting thumb.")

However, while RSI is frequently caused by keyboarding and mousing, many other kinds of injury can impair use of those devices.

Look again at the "Upper Extremities" column in the table from the Bureau of Labor Statistics.

9,760 heat and chemical burns. Over 6,000 amputations. Could you type with a chemical burn?

Current Software "Solutions" Aren't

Suppose you're injured to the point of being unable to use a keyboard and mouse.

"There's an app for that," right? Not so fast!

I merely have RSI -- no amputation here -- yet I get piss-poor service from existing voice-input and dictation software. Here are the problems that make existing software feel infuriatingly backward.

  • voice input, but zero ability to voice edit
  • no structured prose manipulation at the word/sentence/paragraph level
  • lack of feedback during and after user commands
  • long startup time (looking at you, macOS dictation)
  • high latency between end of speech and execution of command (or appearance of dictated text)
  • failure to adapt to user goals by squashing together "command" and "dictate" paradigms
  • no alternate input modalities to ease strain on user's voice

My Goal

Produce the best possible no-hands interface for desktop computing, with a focus on text/document work, and targeted at non-programmers.

The UI must leap out of the screen, guiding the user at every step with context menus. Its behavior will made visible with a display for "currently hearing" and "last recognized" commands.

User will be able to choose from multiple input modalities to accommodate their impairments. This goes beyond speech: there will be gesture commands, as well as low-cost physical tokens (stickers) recognizable by the same video system that recognizes gestures.

There will always be some chance of inaccurate speech-to-text recognition, but with smart design, the frustration of inaccurate recognition can be minimized. Dictation and command modes will be separate, with their own UI affordances. Gestures form a much smaller class than the enormous space of command words, so they can be recognized with higher accuracy. Finally, constant visual feedback will keep the user aware of how the app is interpreting their actions.

One of Many Mockups

Here's a single oh-so-early mockup of some intended UI flows for prose editing.

Comparisons to Other Apps

Coming soonish. Let's just say macOS dictation is pitiful.