Development of a tool for sliding feet gestures in HCI in desktop environments

De Ensiwiki
Aller à : navigation, rechercher


Project schedule.png
Titre du projet Development of a tool for sliding feet gestures in HCI in desktop environments
Cadre Projets de spécialité

Encadrants Céline Coutrix

This project was realized in the MOSIG M2 course "Human Centered Interaction".
Student: Franziska Bärthele


Context

The task for the project was to develop a new technique for human computer interaction. Regarding the desk with a computer as a regular work environment one can see that while using the computer the feet are not doing anything up until now. While working with a computer there are several parts of the workflow which slow down the work. One example are unnecessary homing- processes, or tasks where the hands have to switch not only between keyboard and mouse, but also documents or other devices. To enhance the work with a computer and make it more efficient the focus of this project was on including the feet.

Addressed Problem

There has been work on feet interaction. There are several different devices who offer the functions of a mouse for working with the feet. One example is a trackball approach from Pakkanen and Raisamo[1]. Also movements/gestures of the feet on the floor have been tracked, Velloso et al.[2] and Simeone et al.[3] worked on this. There are also approaches which work with trackable shoes and interactive floors (VR) like Saunders[4] and LaViola[5]. Velloso et al. gave a wide overview of existing types of Feet Interaction and classified them[6]. Hoffmann showed already 1991 that feet are way slower than hands[7]. Accuracy and fatigue are other known bottlenecks of the work with feet.
A special use case for feet interaction is the work with other devices than general computers. These devices may require to be held with two hands, or require interaction with two hands. But how to give additional input, if the hands are already in use? Beside speech and other approaches feet shall be considered here.

Approach

To avoid fatigue in the use with feet the decision was to focus on a sliding movement on the floor. Therefore the feet don’t have to be lifted. These gestures have been studied already. But how to recognize a movement without demanding for high accuracy? This shall be solved with adequate haptic feedback. To give feedback to a sliding foot it shall hit a boundary, a wall. To then make the foot feel pushing a button there is a movable resistance which triggers a button. So the result of these ideas ends in building a frame around the feet standing on the floor. This frame has walls which offer enough haptical feedback to make the user understand whether a button has been pressed or not.
This approach obviously only works if the foot can really slide on the ground, what should be avoided by the soles of the shoes. The prototypes were built of cardboard, and in first attempts no material was found to make shoes slide on the ground. So it was decided to work with socks only. These and other reasons for developing the approach shall be explained on the basis of the built paper prototypes and its final version.

First Prototype

First Prototype

For a simple understanding of the tool it was decided to make it with 6 buttons. One for the toe, side and heel of each foot. In the first prototype only the toe and heel sides were realized. Further in detail one toe/heel side was cut out with and one without a spare part for recognizing the middle between right and left side. But this seemed to be not necessary and primarily unstable. This piece in the middle should not be repeated in the next try. There was no feedback except for the cardboard sides the feet touched, but without resistance. The floor was cut out to try the sliding effect on different floors, but it wasn’t significantly better, and it made the prototype unstable. So this was considered to be left out next time. Another point which was remarked with this prototype was that the tool was sliding away if one touched the sides, so finally it must be attached to / taped on the floor, or be steady by its own weight.

materials: cardboard and tape

Second Prototype

Second Prototype

For the heel and toe sides the options of opening in the middle or at the sides were considered now. Opening in the middle was way more stable, so this shall be kept. The sides open up at the toe side, as there is also the broadest (most outer) point of the foot. Feedback was added now. With attached rubber bands the sides are now coming back after being pushed. But the rubber bands were to strong for the case if the cardboard side wasn’t pushed exactly at the rubber band point. The cardboard buckled and the button would not have been pushed. So the sides were strengthened with sticks, which solved this problem easily. But after a few days using it the sticks began seperating again, so this is not a final solution. For a complete haptic feedback is the point of when the button was pushed missing, a very constant stop.

materials: cardboard, tape, sticks to strengthen the sides, rubber band, pins, paper clips

Third Prototype

Third Prototype: the side button is being triggered, one can see the rubber band and the outer frame which stops the button and gives haptic feedback

To the second one was added a slightly larger frame who should stop the buttons movements and give the final ‘push a button haptic feedback’. The second frame is around 1,5 – 3 centimetres away from the first. The feedback works like this.

materials: cardboard, tape, sticks to strengthen the sides, rubber band, pins, paper clips

Remarks through first person research

In this project it wasn't possible to run real, authentic or long-term user studies. However, to gather more data and impressions of the tool I tried it out as good as possible. This kind of research is called First Person research as Lucero [8] mentioned.

  1. often when you sit the feet are very close to the legs of your chair, the tool hasn’t enough space here. Or the user has to place the feet further away from the chair, which can lead to an uncomfortable or unproductive way of sitting (at the desk). A chair with legs far away from each other, where the tool fits in between can solve this problem.
  2. regular cardboard is too weak to be handled with feet for longer time (one time ‘playing’ with the feet with a corner of the prototype can damage it strongly), with other material like wood it wouldn't be necessary to strengthen the sides.
  3. height of frame: seems good at about 7cm (a range of 5 to 10 cm should work)
  4. width of frame: 40cm were used for the prototype, but it could be less
  5. length of frame: 40cm were used for the prototype, but it could be less
  6. width and length are strongly depending on the users feet size / size of shoe.
  7. My feet are about 26 cm long and rather thin (size 40). If distances are to tall to the frame, the movement for triggering the buttons was too, or unnecessary long. If the feet are tall the distances get tiny, and it would be more likely to trigger it by accident.
30cm feet in shoes in the Third Prototype

Another Remark:
A friend with 30cm long feet (really large, size 46-48, 30cm) tried this prototype with shoes on, you can see it on the photo. When I asked him whether he would trigger the buttons by accident the answer was yes, when I asked him whether the frame could be shorter the answer was no. But this is an extreme case.

Final, working Prototype

Working Prototype with Arduino Leonardo

The third prototype was reconstructed, but with toe and heel sides opening in the middle (better stability). The taller frame was attached to smaller one with cable tie. Now the functionality was added in the FabLab. For the program of the Arduino the libraries ‘Keyboard’ and ‘Bounce2’ were used. The haptic feedback and function call works really good and fast. Only the outer part of the buttons is a very fragile part of the prototype (cables hanging in the air). While using the prototype it is not hindering the user, but shifting the chair next to it or playing with the feet around it is really dangerous for these parts. They have to be hidden robust in a next version.

materials: cardboard, tape, sticks to strengthen the sides, rubber band, pins, paper clips, cables, 6 big buttons, 1 Arduino Leonardo, necessary materials for brazing and connecting, USB cable, (PC, actually it should work with all Linux and Windows PCs)

User Study

Mapping of the functions from the tool

The user study is only about acceptance, first impression of the tool, and possible optimizations of it. The only aim is to collect qualitative data. The assigned functions to the buttons were the result of a survey from 12 people about their mostly used keyboard shortcuts. Any print combination didn’t count because the functions are not consistent with all operating systems. The result was: 1. ctrl+v, 2. ctrl+c, 3. ctrl+x, 4. ctrl+f, 5. ctrl+a, 6. alt tab. See the photo for the mapping of functions and buttons.
The participant filled in a small demographic questionnaire first. I explained the prototype and it’s functions. The participant got a sheet with the information of which button triggers which function (see photo). After I let the users try them out in an example exercise, to see whether they understood how to use it. Then I gave one minute of play time to get a bit used to it. Then I gave them one small exercise which included all the functions the tool offers. Again I gave them time to try anything out before the participant filled in the questionnaire.
The final questionnaire consisted of four parts. The first one was a modified USE-Questionnaire, with the focus on only relevant points. The second is a modified AttrakDiff-Questionnaire. It is focused on the appropriate and relevant aspects. The third one is a reduced SUS-Questionnaire, here the inappropriate fields were left out. In the fourth part the participants were asked about an optimization of the prototype.
The study was conducted by only one person. It was not possible to collect all kind of data which would have been necessary. For example it wasn't possible to regard the participants feet, face and screen at the same time, but it could have given a better insight.

Sadly one button was already broken before the study, so the study included only 5 buttons.

Results

First Person Research with the working prototype

I used it four days with functions / keyboard shortcuts that I assigned to it how I thought I’d need it (alt tab, ctrl + f, s, x, c, v). During these four days I was programming the most time (about 8 hours a day). A typical part of the workflow was having an IDE and a browser opened. In the browser I searched for documentation or code examples, then copied them into my code, adapted it, saved the code, went back to the browser. I used mainly javascript so I could see my result in the browser. After one or two hours I was used to use the new tool for any cut, copy and paste process. Also the saving and window switching worked well. I didn’t use the find function often. But I loved, that when I had changed some code, that I could take my cup of tea in my left hand, saved and switched the window by feet, and my right hand could stay on the mouse, and the pointer in the middle of the screen. As it was easier to then refresh the webpage from the middle of the screen with the mouse, than if the pointer would come from the bottom of the screen (choosing the browser). At the beginning it was strange using the tool, but I got used to it, and could use it really fast after 4-5 hours of having it in use. A negative point is definitely that I was scared to break something, so standing up and sitting down got quiet complicated. I had the feeling that I could work really fast with it, and that some functions or workflows have been even faster with the tool than without. Maybe it was due to being surprised in a positive way, but still though one could do experiments in this direction.

User Study

Demography

All of the four participants had an age of 23 to 28. Their shoe sizes were between 42 and 45, the foot lengths were between 26 and 28 cm. All of them are rather heavy using their pc, from 7 to 9 hours a day. All of them use ‘some’ or ‘a lot’ of keyboard shortcuts. This shows already a data gap, all the participants have rather large feet (male sizes), no typical female feet have tried the prototype.

First Questionnaire

Three of four questions gained very good outcome, the following statements were agreed: ‘It is easy to learn to use it’, ‘ I am satisfied with it’, ‘I would recommend it to a friend’. Only the last point ‘both occasional and regular users would like it’ has a rather neutral result.

Second Questionnaire

The results of this questionnaire chose the following descriptive adjectives for the tool: understandable, exciting, predictable, fast, inventive, easy, clear, continuous workflow. Only the point ‘obstructive/supportive’ was answered rather neutral.

Third Questionnaire

The result here is also positive. All the six statements have been answered with a more or less positive value, but still positive Feedback. It was asked for the following impressions:

  • I think that I would like to use it frequently
  • I found it unnecessarily complex
  • Most people would learn to use it very quickly
  • I found it very awkward to use-cases
  • I felt very confident using it
  • I needed to learn a lot before I could get going

Fourth part: Prototype Optimization

Three out of four participants think that the frame should be more slim, the same amount states that it should be shorter. So the direction goes clearly to a shorter and thinner version. The last question asks about whether the participant thinks, that the buttons might be triggered by accident. Two participant answered ‘yes, occasional’, one ‘yes, often’, and one ‘no, not at all’. This result is very unclear, and doesn’t give a distinct result on the asked question.

comments of participants: In a last line it was asked for any other remarks, here are the most interesting notes, considering options not mentioned up until here:

  • “more feedback for copy” (-function) // this is the only function which does not provide any visual feedback
  • ”… I would love to have more options and more shortcuts…”

Discussion

For a project in this early stage, and in its unknown appearance for the participants the results can be seen as very good. It seems that the tool was accepted, and very fast and easy understood. Still there are a lot of topics who must be worked on:

Problems

  1. A perfect size of the tool will depend on the users size of feet. So in a next study either the participants have to be chosen after their size of shoes, or the prototype has to be built in at least two different sizes. Up until here it can only be stated that 40 x 40 cm have been to tall for the inner frame.
  2. These prototypes have all not be stable enough to be able to work on them for more than a few days. The attached sticks for stability started to detach after about three days of working with it. A suggestion would be wood, but this will lead to major changes in the way of how to build it.
  3. The participants realized quickly which points to touch to push the button. Still, it could be considered to make the whole wall sinking when pushing (and not only one side as up until now). This could make it even easier to push the right buttons.
  4. The legs of the chair can block the most comfortable position of the tool for the user. This has to be considered in future approaches.
  5. The tool has to be somehow attached to the floor, or heavy enough to not move when it is in use. (Actually it sometimes moves a bit when pushing a button too strong.)

Good Points

  1. All participants could accept the overall concept faster than expected. The only thing which had to be mentioned clearly is that the feet don’t have to be lifted, sliding is enough. No one mentioned anything uncomfortable except for the the too big frame.
  2. There is no disturbing time delay between the push of the button and the function call. If the participants said something about this or the haptic feedback it was in a positive way.
  3. The explanation of the basic function was very fast and well understood by all participants.
  4. To be able to keep on working with only one hand at the mouse and the feet is new. These six functions are already able to relieve one hand of the work, still being able to perform copy, paste, delete (ctrl+x works like delete if the clipboard is not used any further) workflows fast. This leads to new possibilities in working with additional devices with a new free hand.

Outlook

  1. The study doesn’t give an answer on the question of whether the buttons will be pushed by accident. But this is an important factor. Before designing a new prototype it might be better to get a result to this question with the actual prototype. The result might provoke new decisions on the prototype design.
  2. For good results of a next study the participants will need a lot of time to get comfortable with the tool. An obvious suggestion would be to hand them the tool for working with it on their regular workplace. As the functions called by the tool have to fit to the work the participants are doing, it was good if the participant could choose, and maybe even change the functions on his/her own. Therefore a small software would have to be provided.
  3. Two participants mentioned that they could handle more than six buttons with this design. If this would work in general has to be tested.
  4. The now used buttons do give a very quiet ‘click’ audio feedback. It might be useful to have a louder one, especially if any visual feedback of a function is missing.
  5. This tool does not replace a mouse nor a keyboard. But it should be considered for any use-case scenario where the hands are not available or using another device and where input is needed.

References

[1] Toni Pakkanen, Roope Raisamo: Appropriateness of Foot Interaction for Non-Accurate Spatial Tasks (2004); DOI: 10.1145/985921.986004
[2] Eduardo Velloso, Jason Alexander, Andreas Bulling, Hans Gellersen: Interactions under the Desk: A Characterisation of Foot Movements for Input in a Seated Position (2015); DOI: 10.1007/978-3-319-22701-6_29
[3] Adalberto L. Simeone, Eduardo Velloso, Jason Alexander, Hans Gellersen: Feet Movement in Desktop 3D Interaction (2014) DOI: 10.1109/3DUI.2014.6798845
[4] William Saunders, Daniel Vogel: Tap-Kick-Click: Foot Interaction for a Standing Desk (2016); DOI: 10.1145/2901790.2901815
[5] Joseph LaViola Jr., Daniel Feliz, Daniel Keefe, Robert C. Zeleznik: Hands-Free Multi-Scale Navigation in Virtual Environments (2001). In Proceedings of the Symposium on Interactive 3D Graphics (pp. 9-15)
[6] Eduardo Velloso, Dominik Schmidt, Jason Alexander, Andreas Bulling: The Feet in Human–Computer Interaction: A Survey of Foot-Based Interaction (2015); DOI: 10.1145/2816455
[7] Errol Hoffmann: A comparison of hand and foot movement times (1991); ERGONOMICS, 34:4, 397-406; DOI: 10.1080/00140139108967324
[8] Andrés Lucero, Kristina Höök et al.: A Sample of One: First-Person Research Methods in HCI (2019); DOI: 10.1145/3301019.3319996