|Titre du projet||3Claws, the 3 fingers mouse controller|
|Cadre|| Projets de spécialité
|Encadrants||François Berard, Laurence Nigay ,Celine Coutrix (Tutor)|
- 1 Students
- 2 Subject
- 3 Prototyping
- 4 Testing Part
- 5 Conclusion
- 6 How to go further
- 7 See Also
- 8 References
Most part of the human population is using a touchpad or a mouse almost daily. These small devices, connected to our computer are really useful, but it remains one problem when they are used in addition to the keyboard: the homing time between these devices makes us waste a lot of time. The 3Claws project has been created for the AHCI MOSIG 2018-2019 Students’ Project by Danh-Chieu-Phu Huynh and Sébastien Riou. It's a new way of interaction provided to move the cursor on a computer screen by using only 2 cameras.
The main goal of this project is to avoid the homing time between the keyboard and the mouse/touchpad. The idea is to remove the maximum amount of time from the homing time. To do so the idea is quite simple, instead of going from one device to another, everything will now happen on the keyboard thanks to 2 cameras. We decided to use 2 different cameras, in order to detect 3 finger locations (Thumb, Index, and Middle finger). The objective of this project is to provide a user-friendly mouse-keyboard device that will help the user using tools when writing documents or filling a form in the fastest way for example by avoiding the homing time of the usual mouse devices.
State of the art
A lot of projects already tried to create a mouse controlled by hand gesture using a camera. The work of Hojoon Park  itself inspired from the work of Chu-Feng Lien  tends to implement a method for controlling mouse using a real-time camera. Unfortunately in addition to the fact that the final prototype was not stable because of the bad hand detection and hypothesis, this idea can't be applied in our context because it asked the user to remove his hand from the keyboard, the homing time will then be different than the one between keyboard and usual mouse, but it will exist anyway. In order to make our prototype more stable, we decided to use the tape idea of Abhik Banerjee, Abhirup Ghosh, Koustuvmoni Bharadwaj, Hemanta Saikia  even if the light would be a problem for future improvement, it would be a nice trade-off in order to test our human-centered technology. We also had the idea already presented by Kamran Niyazi, Vikram Kumar, Swapnil Mahe and Swapnil Vyawahare  using some threshold between the fingers distance to trigger click events.
All these researches have been realized using as first reference the “Computer vision based mouse”, Acoustics, Speech, and Signal  study.
To create the prototype of this project we chose to use 2 low-resolution webcams in order to the fastest possible in term of image frame processing.
- The first camera is placed on top of the screen and is oriented to see the keyboard from the top. It is used to determine the x and y position of the 3 fingers (thumb, index, middle finger) above the keyboard.
- The second one is used in order to detect clicks by a modification of the distance between fingers on the z-axis.
We also used colored tape to make finger detection easier.
For the final prototype, we finally have been able to do it with one camera only. To be more understandable the following explanations will suppose that we still have 2 cameras.
In order to activate the 3Claws option, we used a movement that looks « natural » for humans but uncommon when typing on a keyboard so that it’s not disturbing for the user but still easy to recognize. The 3 tracked fingers are activating the 3Claws when they are reaching a given threshold and when they become close enough as you can see here:
The 3Claws will then be disabled if one of these fingers is going beyond this threshold and become too far from another one:
Move Cursor with 3Claws
When 3Claws is activated we can move the cursor easily just by keeping the same position with fingers and by moving the hand. The translation from the movement of the hand on the keyboard to the screen will be done in a relative space so that we may need to deactivate and reactivate the 3Claws several times (such as with touchpad) but it should be basically more intuitive like this and avoid big movements.
Click with 3Claws
For the click, we tried to stick to the idea of the mouse and touchpad to stay more users friendly and to be understood more easily. In order to detect the clicking events with 3Claws we are using the side camera. To use the right click the basic idea is to keep the thumb and middle finger together and to release the index finger.
|Activated Mode||Right CLick|
In contrary, if we want to use the left click we need to keep the index and the thumb together and we just have to release the middle finger.
|Activated Mode||Left CLick|
The idea by using a side camera was to be sure that the click was really achieved by the user. With only a top camera it would have been hard to make the difference between clicking or fast typing only.
Most of the test parts have been designed by using a basic window. Only the way we use this window will change depending on the test we want to achieve. In the window of this app there are 3 different things:
- validation button
- empty text field
- text giving the word that has to be entered.
Each of the following tests will have to be done with the mouse, touchpad and finally 3Claws.
The first test is designed in order to test our project in “real” condition. The basic idea is to ask the user to go to this window, enter the asked text inside the input field and to finally validate by clicking the button. The window will then move to another place and he/she will have to repeat the operation. The goal is to compare the action time between the 3 tested devices.
In test 2, to avoid the errors or the reflexion time that could be due to an unexpected word we are keeping the same input word that has to be entered before validating and the user have to tap it one first time to launch the test to be sure he's used to it.
The third test consists only in clicking on the validate button, without taping the text. It will give the opportunity to test the system without having the need of disabling it so that the 3Clwas system will stay activated from the beginning to the end of the test.
Other Test suggestions
We were thinking about 2 other kinds of tests that we didn’t have time to implement.
- The first one would be to go from one place to another without clicking (to test the move only)
- The other one would be to test the 2 different clicks only (for example by using the right click depending on the color of the button).
Thanks to this project the homing time between taping something on the keyboard and having interaction with the cursor on the screen should be significantly reduced. We are aware of the fact that our prototype is not perfect yet so maybe the computations between the finger recognition and cursor actions can lead to error in the final results of our tests. We were also thinking about the fact that the click will be a bit hard to assimilate for the user. With mouse and touchpad, we're used to put our finger down to click, with the 3claws we have to put it up. Unfortunately using such a gesture is not possible with our base position so we hope it will not disturb the user during the recognition part. In our opinion, the 3Claws is really stronger than touchpad and mouse when there effectively have a homing time but it should be less efficient if it is used as a substitution for tasks that have to be done using cursor only. We think that the final results should be better than the mousepads that are sometimes hard to use when a huge distance separate the cursor from our goal, but the fact that actual mouses are really fast and accurate makes us think that the final results will be hardly better than mouse ones.
Analysis of the results
We can see on the Test 2 results that the difference of time between TouchPad and Mouse is not that significant, but we can analyze it in combination with Test 3 and it gives more interesting information.
Regarding Test 3 we clearly see that for tasks that need to interact with cursor only (it's to say no keyboard), the mouse is obviously really faster.
What we could assume is that for Test 2 results, the homing time between keyboard and touchpad is smaller than the one between the keyboard and mouse. Thanks to this reduction of the homing time the touchpad can stay competitive with the mouse, but it's really only if we use it in association with the keyboard.
How to go further
Drag n Drop and Selection
The drag n' drop or selection would be implemented easily by changing the "click" interaction by a "press & release" interaction instead, but it would ask for totally different tests. We then decided to focus only on one kind of interaction to detail it more carefully.
Something that could be useful when filling forms or writing documents would be the use of the slider. Unfortunately, it would take too much time to implement it and to test it for this project. The idea we proposed was to keep the index and the middle finger together and to move them away from the thumb. It could be compared to a smart gesture present in some touchpad, allowing to slide windows bar using 2 fingers.
More accurate camera
The prototype is based on low-resolution camera so the finger detection and movement detection could be more accurate and precise with a better resolution. For now, we only use tape to detect the different fingers. The color of the tape we want to detect change depending on the light for example. The best improvement would be to completely avoid the use of tape and to detect automatically finger, but it would also ask more computations.
Another problem is that it's hard to detect fingers under low light. A perfect improvement, for example, would be to use the same technology as Kinect2.0 depth sensors and to then compute the position of each finger. Computation would may be more complex but the data should be acquired in a faster way.
 A. Erdem, E. Yardimci, Y. Atalay, V. Cetin, A. E.: “Computer vision based mouse”, Acoustics, Speech, and Signal Processing, Proceedings(ICASS) IEEE International Conference, 2002
 Chu-Feng Lien: “Portable Vision-Based HCI – A Real time Hand Mouse System on Handheld Devices”, National Taiwan University, Computer Science and Information Engineering Department
Abhik Banerjee, Abhirup Ghosh, Koustuvmoni Bharadwaj, Hemanta Saikia :Mouse Control using a Web Camera based on Colour Detection, Department of Electronics & Communication Engineering,Sikkim Manipal Institute of TechnologyEast Sikkim, India
 Hojoon Park , A Method for Controlling Mouse Movement using a RealTime Camera ,Department of Computer Science Brown University, Providence, RI, USA
 Kamran Niyazi, Vikram Kumar, Swapnil Mahe and Swapnil Vyawahare : Mouse Simulation Using Two Coloured Tapes, Department of Computer Engineering, AISSMS COE, University of Pune, India