Resolution of touch in shape changing display

De Ensiwiki
Aller à : navigation, rechercher

Students

This page presents the HCI project of Aveek Mukherjee and Charles Bailly.

Introduction

Context

Visual output has become a standard in a lot of systems nowadays. Computers, tablets and smartphones mainly use this modality to convey information to users. Audio is also largely used in combination with it, while haptic feedbacks are mainly limited to notifications (incoming call or message, for example). However, visual and audio based interactions are not always suitable depending on some busy and/or noisy environment. Driving, crossing a road, biking, being in a crowded tramway… These are simple examples where haptic feedback interactions could play a bigger role. Shape-changing displays is one possible technique of tactile human computer interaction. Several approaches has already been studied, and some of them are based on pins. Pins are small cells which can be raised-up (possibly to different heights) to convey information. Dynamic Braille keyboards, tactile mouses and Tactons [Brewster and Brown, 2004] are examples of systems using it.

VirTouch tactile mouse

Our work is based on the idea of a smartphone with a high-resolution shape-changing display, meaning that it has a high density of small pins. Such a system would be difficult to build concretely in the frame of this project, but it is still interesting to study it to see it’s usefulness.


Adressed problem

Our main focus was to study the precision humans can have when they interact with such a system using their fingers. At what moment (size, number and position of pins) users cannot perceive a group of pins but a complete shape? Are all pins necessary to represent a basic geometrical shape (for example, a plain square representing a button)? Which one(s) can be removed, which cannot? Do users miss particular pins (under error), “add” some of them under certain circumstances (over error)? These are the questions we studied in this project.


Our Approach

As explained above, we focus on a basic geometrical shape, a square, which would represent a button in the final system. In fact, we wanted a horizontally and vertically symmetrical shape, and the square was a good choice for that. Since having a dynamic prototype would have required a lot of time, we decided to work on static ones. We created several of them, varying the position, the size and the number of pins in order to test different patterns. In order to explain the configuration of pins in a shape, we will use the below notation:


Example.png

In the above configuration we have a shape where 10 pins per row and 10 such rows are present. White circle means pin in that position is absent, and Black circle means the pin is enabled. As shown in the previous drawing, all pins are cylindrical.


Shapes

We divided our shapes in four groups, one for each size of the square, as shown in the following table.

Group Number of pins (row x col) Diameter of pin (mm)
A 10 x 10 1
B 9 x 9 1
C 9 x 9 0.75
D 9 x 9 0.50

The size of the square and the diameter of pins were chosen according to our needs and the state of the art regarding the perception level of the human finger. It has been assumed that two points are distinguishable from one only when the two points are sufficiently separated to evoke spatially distinct foci of neural activity [Mountcastle and Bard, 1968; Vallbo and Johansson, 1978]. Therefore, two very close pins (which can be considered as points during the perception process) can be misperceived as a single one. Since the two points discrimination technique based on this principle has been called into question lately, we prefered to based on work on another approach, the Grating orientation discrimination [Van Boven and K. Johnson, 1994]. According to it, the spatial resolution of the human finger is equal to 0.94mm.

With this result in mind, we designed the first group of shapes, A, choosing 1mm diameter pins. The idea was that two non-consecutive pins should be distinguished. Therefore in all the shapes we tried to keep the distance between the pins to be greater than 0.94mm. The corresponding patterns (see the dedicated section below) are supposed to be “fairly easy” to perceive. In other words the users must perceive the pins distinctively. Since we needed an odd number of pins on a square side for some patterns, we also created the group B following the same principle. Then we wanted to gradually create shapes where the users must not be feel the individual pins distinctively. Thus, pin diameter in group D is half of those in B. We wanted to be sure to be quite below the threshold to see the effect of the scale on the global shape perception. We wanted to see where is the lower bound, below which the user cannot perceive the pattern and just feels a plane square. Finally, group C is something in between, since we wanted to have an “average” scale.

Patterns design

The patterns we designed are the followings:

A1
A2
A3
A4

The first pattern was created to check if users were able to perceive correctly borders, and the empty space remaining. The second was designed to see how pins at a special position (here, the corners) are understood. Then, with A3 and A4, we wanted to continue working on borders to see if some regular patterns were recognized or not.

B1, C1, D1
B2, C2, D2
B3, C3, D3
B4, C4, D4
B5, C5, D5

These patterns are the heart of our work: we created several times the same patterns at different scales (see the table about groups above). First, we designed a first serie to continue working on pins at a special position (one pin at the center of the square). Then, we complexified the pattern adding some pins to form a regular shape (2 and 3). Therefore, we were reducing each time the distance between two pins. The two last patterns were made to be difficult in purpose, especially the 5th.

B6
B7

B6 was the last pattern about pins at a special position: instead of having one pin at the center like B1, C1 and D1, we put one pin at the center of each border. Finally, we wanted to have a pattern which was not symetrical, so we designed B7. The idea was also to have a nearly complete shape with a hole inside.


Implementation

Once we finished to design the pattern, we laser cut our pieces and divided them on several plates of four or five squares, depending on their scale.

Single square
Plate of four squares


Evaluation

Users

All our work is based on the sensitivity of the finger, so we had to choose users who (hopefully) had more or less the same sensitivity. In practice, we excluded people with jobs or activities that can alter their finger sensitivity (manual workers, for example, or professional pianists).
In the end, we gathered 13 users between 15 and 67 years old. Most of them are students.

Experiment

We asked users to put the index of their dominant hand on one square at a time. They were free to move their finger to feel the shape as long as they were not looking at it . There was no limitation in time. When they felt ready, users were asked to mark the felt shape on a printed questionnaire with squares. Each square was displayed using the same notation as above, and all pins were blank at the beginning. To indicate that he felt something, a user just had to darken the corresponding pin on the paper. They were also encouraged to think aloud and to write their comments on each square when they had some.

Users where first trained with additional patterns to eliminate the learning effect and be sure they perfectly understood what they were expected to do. Once the experiment was done, We agglomerated each such data and created heatmaps for the felt touches for each individual shape.

Notions and notations

Here are some notations and formulas we will use in the next sections:

OverError indicates the number of extra pins perceived by the user as a factor of actual pins enabled. UnderError indicates how many pins the user could not perceive.


Over Error =  (upinlen - apinlen)/apinlen

Under Error = (apinlen - corlen)/apinlen

Where: upinlen = #pins the user felt , corlen = #pins the user got correct and apinlen = #pins that were actually in the shape


Total Error= OverError + UnderError

Diff Error = OverError - UnderError


Accuracy = (TotalPossiblePins - TotalError*apinlen)/TotalPossiblePins*100

Where TotalPossiblePins = #pins the could have been placed in the total shape

Density = apinlen / TotalPossiblePins

EmptySpace = (TotalPossiblePins  - apinlen) * AreaOfPin

AreaOfPin = \pi r^2

Where r is the radius of a pin

Results

HeatMaps

Heatmaps are a basic but consice way to present the data, and is convenient to observe some tendancies.

Borders

A1 - Pattern
Heatmap

This first square shows that borders were easily recognized at this scale. Also, nearly all users were able to perceive that the center was completely empty.


A3 - Pattern
Heatmap

We already observed some confusion here which we were not expecting. The distance between two pins on a border was quite greater than the perception threshold (2mm compared to 0.94mm). It seems that users perceived it, but were not able to precisely replace pins on the drawing. Therefore, we have a significant over error in theses areas. However, corners are once more well recognized.


A4 - Pattern
Heatmap

This pattern was understood by most users. Here, the placement of pins seemed easier for them, since they were all at a special location (corner or around the center of a border).


B6 - Pattern
Heatmap

This square was really similar to the previous one: we just have one pin less on each border (9*9 pins instead of 10*10). As expected, we observe something coherent with A4, where we just have a little bit of unprecision around the center of borders.

Therefore, users are able to perceive borders and big empty spaces quite accuratly, but can be a little unprecise when they have to find the precise position of pins. This phenomenon can lead to significant over error in these areas.

Pins at special positions

Several patterns were made to study if some positions for the pins were easier to perceive than others.

A2 - Pattern
Heatmap

Here, it is clear that corners were really easy to perceive for users, since we have no error at all.


B1, C1 and D1 pattern
Heatmap B1
Heatmap C1
Heatmap D1

One pin at the center of the pin is well recognized in the two first cases, but only half of users we able to do so in the last one. Moreover, the over error is quite significant here compared to previously. The scale may play an important role in this phenomenon, even if once again the distance between the central pin and the borders is larger than the theoretical threshold (1.5mm compared to 0.94mm).


Complex patterns

B2, C2 and D2 pattern
Heatmap B2
Heatmap C2
Heatmap D2

Here, we added to the previous pattern some pins on the diagonals of the square. What is surprising is that the intermediate scale, C, was the most successful one. The B2 pattern was not understood, and we again observe more over error on the left side. More studies would be necessary to explain this missunderstanding. It might come from our pieces, which were not perfect, or maybe from some learning effect. The results for D2 are going this way: they are quite close from the next serie of pattern. Maybe some users were biased (confirmation bias) and tried to find a pattern they already encontered.


B3, C3 and D3 pattern
Heatmap B3
Heatmap C3
Heatmap D3

This pattern was already quite hard for users at a big scale, and get worst with smaller ones. What is interesting is that effective pins were recognized, but the overall over error is significant, especially in the last case, when most people felt a plain shape.


B4, C4 and D4 pattern
Heatmap B4
Heatmap C4
Heatmap D4
B5, C5 and D5 pattern
Heatmap B5
Heatmap C5
Heatmap D5

These two patterns were designed to be difficult, and results are coherent with our expectations: even with large pins, users were confused and felt a nearly plain shape. This is even more true with the second one, where most users felt quickly a perfect square. Therefore, these patterns allow to reduce the number of pins without altering the perception of users. In addition, we found again the phenomenon where we are above the theoretical threshold of distinction (for group B), but we still observe confusion.


Errors, accuracy and density

All formulas used in this section are here.

Over error and Under error

TotalError Error Difference

First, we observe that over error is much more frequent than under error. It may be surprising since some pins can be hard to perceive, especially at small ones. However, when patterns were hard for users, they had the tendency to perceive it as perfect (plain shape), leading to a high over error. This is interesting because it confirms that some "incomplete" patterns (not all pins are used) can do the same as complete ones.: they felt more pins than the effective ones. We see that A3 has a high over error rate and under error rate. This is because in the formula of both OverErrorRate and UnderErrorRate we express these as a ratio with the number of enabled pins. A3 has a low density and so the error rates get amplified.

The Error difference can be used if we want to ignore exact positions of pins in the formula of Error. Probably if a user felt a pin but marked it in the wrong position it will contribute by 1 to both OverError and UnderError. In this case the Error difference can ignore these cases. However we see that there is not much difference in values of error by using Error difference instead of TotalError, confirming the fact that OverError is the dominant component in TotalError.

Accuracy vs Density

We plotted Accuracy vs the Density of the shapes. Please note there are multiple shapes with same density. Example B1, C1, D1 have same pins in exact positions leading to same density.

Accuracy vs density.png

The purpose of plotting vs density is to see if there is any pattern between the shapes of different sizes but same densities. We see that sets (B4,C4,D4) and (B5,C5,D5) have close Accuracies. Since they were nearly plain squares, users didn't made of lot of mistakes considering them as plain shapes.
There is bigger variation between sets (B1,C1,D1) , (B2,C2,D2) and (B3, C3, D3). Also, there is no repetitive ordering between B,C and D. Therefore, it seems that smaller scales don't always result in lower user accuracy.

Also note how the Accuracy values of A4 and B6 are close: as we could expect, similar shapes have similar accuracy.

Accuracy vs EmptySpace

We plotted Accurcy vs EmptySpace for each shape. Note that EmptySpace axis is in logarithmic scale.

Accuracy vs emptyspace.png

We see that there is a pattern between the 5 shapes of groups B,C and D. It seems that the accuracy reaches its minimum somewhere in the middle density shapes and then increases at both ends of the shapes. For example, the accuracy in group C is highest at shape C1 , then achieves the lowest at C4 and increases again for C5.
In fact, it highlights a phenomenon we already described previously. The more complex the pattern is, the more users make mistakes. However, there is a moment when users cannot understand the pattern and feel it as plain, but errors are compensated since the shape was nearly full.

Conclusion

The work we did within the frame of this project is only a first step of what we had in mind at the beginning (the smartphone with a high resolution tactile display based on pins). However, even if we could not build a working dynamic prototype, we were able to find some interesting preliminary results about some patters of pins and the accuracy of the human finger.
First, we showed that it is not necessary to use all pins to have users feeling a plain shape. Some complex patterns allow that, even if the distance between pins is sometimes greater than the theoretical threshold of distinction. However, some specific pins should not be removed (corners, borders and the center of the shape). Then, we also saw that users make more mistakes with complex patterns, until these error are compensated by a high-enough density of pins.

Further studies would be required to determine the precise bound of this phenomenon. Also, studying other shapes than the square would also be interesting.

References

[1] Stephen Brewster and Lorna M. Brown, “Tactons: structured tactile messages for non-visual information display”. AUIC '04 Proceedings of the fifth conference on Australasian user interface - Volume 28, pages 15-23. 2004

[2] Sean Follmer, Daniel Leithinger, Alex Olwal, Akimitsu Hogge and Hiroshi Ishii; "inFORM: Dynamic Physical Affordances and Constraints through Shape and Object Actuation". MIT Media Lab, 2014

[3] Robert W. Van Boven and Kenneth O. Johnson, “The limit of tactile spatial resolution in humans”. American Academy of Neurology, 1994

[4] R.W. Van Boven, R.H. Hamilton, T. Kauffman, J.P. Keenan and A. Pascual–Leone, "Tactile spatial resolution in blind Braille readers". 2000

[5] Eden Tan, "Estimating human tactile resolution limits for stimulator design", 1995