Gesture set replacing frequent commands for fighting Repeated Strain Injuries

De Ensiwiki
Aller à : navigation, rechercher



Over the course of the last few years, we have developed a wide range of input technologies that can be integrated to our daily activities, such as eye movement recognition, gesture recognition, and voice commands, but, in practice, none of them replace the traditional mouse and keyboard inputs on those daily activities. From an objective point of view, the mouse and keyboard are efficient, people are used to them, and their capabilities are good enough to be used for daily activities. However, reseach has shown that our morphology is not designed to use keyboard and mouse for long periods of time. Nowadays there is practical evidence that there are health problems having continuous usage of computers as main cause, the most common of them being repetitive strain injuries (RSI). In order to prevent them or to lessen the pain, several hardware and software technologies have been developed. In addition, medical research has shown that doing a set of simple exercises are highly effective at their prevention and their treatment. In this project we combine both gesture detection and traditional inputs, in order to reduce the risks of having RSIs without replacing the widespread keyboard and mouse inputs.


Repetitive Strain Injuries

Repetitive Strain Injuries (RSI) describes the pain in muscles, nerves and tendons caused by repetitive tasks or forceful exertions. The most common symptoms of RSI are usually associated with edema, tendinitis and carpal tunnel syndrome, among others.

Since the 1970s there has been an increase in RSIs because of the widespread use of typewriters/computers in the workplace that require long periods of repetitive motions in a fixed posture.

State of the art

At present time, there are some hardware devices for gesture recognition and some software programs to remind the user of doing stretches. However, a tool that combines both things has yet to be developed.

Leap Motion

Leap Motion is a device with hand and finger recognition. This device has two monochromatic IR cameras and three infrared LEDs, with a range of approximately 1 meter with accuracy of approximately 0.7 millimeters. The refresh rate is high, being able to have a frame rate up to 200 FPS.

The Leap Motion API allows to fully recognize arms, fingers and hands, as long as the fingertips are visible to the camera.

Leap Motion example

While the Leap Motion successfully recognizes some gestures with high accuracy, other gestures make some fingertips "hidden" for the camera. These gestures cannot be recognized with a Leap Motion, the main reason why it was not used.

Microsoft Kinect

Kinect is another motion device, consisting of a infrared camera, with an effective range of 0.7 to 6 meters.

Unlike the Leap Motion device, the Kinect is well suited for body recognition in larger areas, with lower resolution. However, the refresh rate is also significantly lower, having a frame rate ranging from 9 to 30 FPS.

Kinect example

The low frame rate of the Kinect infrared camera was a major issue when considering the use of it.


Workrave is a software program that helps with the prevention and treatment of RSIs. Among its functionalities, the program reminds the user after a certain period of time to take a small break or to do some stretches.

Workrave screenshots

This program, however, does not integrate any kind of gesture recognition, forcing the user to stop doing an activity in order to do the exercises.

Addressed problem

For the last 30 years, the usage of computers has increased exponentially, especially in the professional area. Unfortunately, the number of associated health issues due to their usage has also increased. Among them, one of the most common health issues are RSIs. [The objective behind this project is to reduce and/or eliminate the risk of getting an RSI.] Medical research has shown low-grade RSIs can be treated with simple hand/wrist exercises. However, in order to do them, the person has to take a short break from his activities. This usually deters people from doing the exercises, in order not to lose productivity.

In this project, we use these exercises as gestures for doing useful commands in the computer, as a way of preventing and/or treating RSIs while at the same time making the user productive, allowing him not to halt his workflow.


For RSI prevention purposes, it is generally recommended that you should take a five minute break after every 20 or 30 minutes of continuous activity.

In general, the exercises that require several repetitions are shorter. Because of that, we assign more frequent actions to these commands.

The gestures that we are going to use are:

Gesture set
Gesture Command
GestureRSI1.gif Opening the web browser
GestureRSI4.gif Opening an IDE
GestureRSI3.gif Compiling a program
GestureRSI2down.gif Scrolling down
GestureRSI2up.gif Scrolling up

Originally, we intended to use Leap Motion and Kinect for the gesture detection. Unfotunately, their limitations would have made the implementation more challenging. Instead, for this project, we decided to implement Wizard of Oz.

Our gesture-command mapping criteria focused mostly on users who perform daily activities with a duration of around eight hours a day, especially programmers. We mapped their most frequent input commands, into RSI exercises.


  1. 12 participants were asked to do 2 simple programming tasks using traditional keyboard and mouse inputs.
  2. The participants go to a website, where they briefly learn the commands related to each gesture.
  3. The participants are asked to do 2 simple programming tasks different than in step 1. In this step, they are asked to use gestures, without restricting the use of keyboard and mouse.
  4. The participants fill a survey focusing on the degree of acceptance of the gestures.

We use different exercises in order to minimize the learning curve effect.

Our participants were people who perform repetitive activities for around eight hours a day.


We already know that the selected gestures are effective in fighting RSIs thanks to several existing medical publications. Consequently, we did not evaluate the medical effects on the wrist or the reduction of RSIs in the user, especially since that is a long term research that can require years of use of traditional inputs.

Instead we would like to evaluate, how often the user performs common input activities and how comfortable they feel about using different input methods that don’t require an immediate response, that can take a moment to be processed and that are used in a range of 10 minutes.

RSIres1.png RSIres2.png RSIres3.png RSIres4.png


The system for the gesture set replacing frequent commands, is generally accepted for the people who tested it, and would be used in majority in a daily routine, mostly for health reasons.

However, there are objections in general about the process of learning the gestures and adapting them to the commands that are already learnt in traditional input devices as keyboard and mouse.

To get used to gesture commands additional tools could be required in order to assist the learning, estimulate the user, and track the usability of commands to get information about which of them should be used and how often are used


[1] Zhongping J., Ligang L., Yigang W., B-mesh: A modeling system for base meshes of 3d articulated shapes, Computer Graphics Forum 29 (7) (2010) 2169–2177

[2] Kobbelt L., Campagna S., Vorsatz J., Seidel H.-P.: Interactive multi-resolution modeling on arbitrary meshes. In Proc. of SIGGRAPH (1998), pp. 105–114