Augmented reality instructions for building tasks

De Ensiwiki
Aller à : navigation, rechercher
Project schedule.png
Titre du projet HCI Projects
Cadre HCI (MoSIG)

Encadrants Laurence Nigay




Some construction tasks or maintenance tasks can be difficult or prone to errors, even with a printed manual. If a mistake is made, users need to step back through instructions and deconstruct their previous work. These errors cause frustration and can also lead to piece damage or in some cases dangerous situations. Some use cases examples:

  • Furniture constructions: Errors may lead to unusability or accidents.
  • Electronic constructions: Errors may lead to short circuits or dysfunctions.
  • Cross-stitch embroidery: Errors have no major consequences.

Addressed problems

We made several hypotheses about this situation:

  • Printed manuals are not clear enough for some specific tasks (e.g. use piece 3a, not piece 3b or use the 15mm screw, not the 12mm one).
  • Users lose focus when they switch between printed manual and real construction.
  • Users may skip a part of a printed manual and realize later what happened.

The goal of our project is to let the user focus on the construction by displaying augmented reality instruction on it.

Anticipated results

We predict AR instructions will have no benefits for very simple tasks or can have a negative impact on them. On the contrary, it should greatly increase time performance and reduce errors for difficult tasks. Instead of moving eyes between manuals and operation regions, users can see both goal and objects intuitively with AR assistance hence they may also feel the task is easier with AR instructions.

State of the art

There exist many solutions to help people deal with this kind of tasks. But unfortunately, most of the solutions have limitations.

For instance, DuploTrack real-time system demonstrated by Ankit Gupta. It is a real-time system for authoring and guiding Duplo block assembly, composed of Kinect to track the model, screen to display visual feedback and an operation table. Since a Kinect camera has requirements for installing space, this device is not flexible enough. And noise in the camera data and structure mismatch have an influence on the visual feedback.

Mark Rice and his team proposed a study on AR interface for wire harness assembly. In this paper, he mentioned that camera alignment issues resulted in noticeable limitations during interaction in AR conditions. (Comparing Three Task Guidance Interfaces for Wire Harness Assembly).

Research is also done on comparing AR method with paper instructions. Jonas Blattgerste presents experiments by using Epson Moverio smart glasses, Microsoft HoloLens, smartphone and paper manuals. The experiment results show that participants use the least time to solve tasks with paper manuals, but they make fewer errors with AR-assisted on Microsoft HoloLens.(Comparing Conventional and Augmented Reality Instructions for Manual Assembly Tasks.)


Our approach

Prototype presentation

Final prototype

Users have to place objects on 2D grids of different resolutions using printed manual or augmented reality instructions on a phone. The goal is to reproduce the shown pattern quickly. We measured:

  • The time taken by the user to complete the task.
  • The number of errors.
  • The difficulty of the task from the user viewpoint

During experiments, two methods are introduced: user manuals and AR application. Users start with one method and one challenge set, and then the other method and the other challenge set. The starting method and set change from one user to another.

For the detailed protocol of the interaction, see Experiment protocol

Video of the experimentation using the AR app

Prototype alternatives

  • Initially, we thought about using the Meta2 headset of the lab instead of a mobile phone, due to technical difficulties we decided to use the latter.
  • Initially we planned to use a higher fidelity interaction with 3D building, using lego pieces but it was not feasible given the remaining time.
  • If we couldn't use an AR app. We would have used a transparent sheet to simulate the display of AR data. This solution is simple and require no coding but is less representative regarding to the use of a phone and the use of an AR app.

Thoughts about prototype choice

The simplicity of the interaction allows us to avoid open problems of computer vision and artificial intelligence for building an application prototype. Using a smartphone opens the question: Do users prefer using both hands for these tasks or using only one hand and having AR data?

Experiment protocol

We start by explaining the whole process to the users and respond to their questions. Secondly, we let them grasp objects and place them on the grid, then manipulate the AR app. After this step, users feel more comfortable with these actions.

Pieces are placed on the left or on the right side depending on user preference. This is especially important for the AR experiment where users have to move their free hand in front of the camera. Otherwise, the pattern around the grid could be hidden and camera will lose detection frequently.

Finally, when they are ready, we begin the experimentation.

Starting time is either:

  • When the user sees the pattern on the printed manual.
  • Before the user moves the phone to try to display AR instructions.

Ending time is either:

  • When the user says stop.
  • When the user has finished and forgot to say stop.

We count one error for each offset in x or y direction.

We used the Simple Ease Question (SEQ) protocol to mark the difficulty. We scaled it from 1 to 10 to fit better oral discussion and french notation system.

Technical description

The AR application detects a given pattern and replaces the content of the grid with instructions it displays a representation of parts to place on the grid.

We built a specific pattern for the app to recognise. There are 3 resolutions of grid: 10x10, 20x20 and 40x40. We also built two challenge datasets: set S1, set S2. Each set contains one challenge for each grid resolutions. Pieces to place on the grid are 4 pieces of paper of the same shape and different colors (red, green, blue, black).

20x20 grid, challenge sets and pieces
20x20 grid
challenge sets and pieces

The AR mobile app built with Unity. It can run on Android 4.0 or more. The application can be installed via this apk file. Grid sizes and challenge sets can be selected directly within the application.

  • Dev. environment: Windows10, Unity 2018.3.
  • Devices: Android device.
  • Requirements: Android version 4.0 or more.
Pictures of our AR App
AR data for 10x10 grid and 1st challenge set
AR data for 40x40 grid and 2nd challenge set

Strenghts and weaknesses of the prototype


  • Application is working.
  • User use the app intuitively.
  • Well organised, clear protocol, last 10 to 15 minutes per participant including explanations and debriefs.
  • Application is only a prototype and has some bugs.
  • Sometime the AR will disappear after hiding the pattern, the app has to recognise the pattern again to display AR content.
  • Pieces were small and difficult to grab. We should have used 3D shapes.


  • Users start with different challenges and different methods.
  • Use of SEQ gives interesting results.
  • The interaction is performed with a low fidelity prototype.


We performed the experiment on 24 users, most of them were Ensimag students.

  • 6 of them started with user manual and challenge 1.
  • 6 of them started with user manual and challenge 2.
  • 6 of them started with augmented reality and challenge 1.
  • 6 of them started with augmented reality and challenge 2.

Raw results can be accessed here. The first row corresponds to the difficulty perceived by the user from 1 to 10.

Sorted results and simple analysis can be found here in several tabs.

One thing to take into account is that the application is not perfect and some users lost several seconds because the app cannot detect the pattern and no squares displayed on the screen. This behaviour had impacted negatively time results and maybe difficulty feelings of users.

Talking with users gave us the following feedbacks too:

  • For 40x40 grid each of them prefers the AR app.
  • For 20x20 grid almost each of them shows their preference on the AR app, few of them said it didn't matter a lot.
  • For 10x10 grid some users favour one method while some prefer the other and the others thought it didn't matter.
  • Some users told us pieces where too difficult to place because they were flat and small.
  • Most of the users have an enjoyable experience on the app.


UM = User Manual AR = Augmented reality

Average difficulty (SEQ) 1 = very easy, 10 = very hard
Grid Resolution UM start UM UM start AR AR start UM AR start AR


2.917 2.167 1.667 2.167
20x20 5.583 4.917 2.083 3.083
40x40 8.750 7.833 2.583 3.917
Difficulty variance (SEQ) 1 = very easy, 10 = very hard
Grid Resolution UM start UM UM start AR AR start UM AR start AR


2.447 1.061 0.606 0.515
20x20 1.538 1.720 0.811 1.356
40x40 2.022 1.970 1.356 1.720
Mean difficulty
  • In average, users find the task easier with augmented reality, especially for more complex tasks.
  • Users are trained through the first method and it has an impact on their perception of difficulty, regardless they start with printed manual or augmented reality.
  • Less variance with augmented reality, users are more equals using this method in term of difficulty feeling than with the user manual.
Average time
Grid Resolution UM start UM UM start AR AR start UM AR start AR


21.833 19.564 25.541 22.854
20x20 45.445 45.455 28.768 30.827
40x40 103.208 80.088 27.206 31.554
Time variance
Grid Resolution UM start UM UM start AR AR start UM AR start AR


81.487 52.873 51.896 60.851
20x20 153.312 406.771 101.408 81.090
40x40 2588.210 1545.638 51.952 110.794
  • Users are more efficient with the manual for simple tasks, AR increases time performances for 13x13 or more.
  • Variance is less important with AR, users are more similar with AR: 2 possible causes: AR acts as a filter for participants resolving the most problems of the task hence users only perform a less important role in the task; Or participants aren't familiar with this interaction (relatively new for people)
  • Variance is more important when users have no experience with the task (10x10 UM/UM and AR/AR relative to 10x10 UM/AR and AR/UM) (training effect)
Mean time
  • For simpler tasks, the movement times are slightly different for participants with user manual start from user manual and augmented reality. Bus as task difficulty increases, the difference becomes more significant.
Mean time and standard derviation
Average errors
Grid Resolution UM start UM UM start AR AR start UM AR start AR


0.167 0.083 0 0
20x20 1 0.667 0.083 0
40x40 5.333 3.583 0.083 0.333
Errors variance
Grid Resolution UM start UM UM start AR AR start UM AR start AR


0.333 0.083 0 0
20x20 1.455 0.970 0.083 0
40x40 7.333 9.174 0.083 0.424
  • Fewer errors with AR, on all difficulties;
  • Training effect: Fewer errors when participants have already performed the task with another method
Mean errors
  • Higher standard deviation for user manuals with respect to AR, confirms the point mentioned before that user performs more differently with user manuals comparing to AR;
Mean errors and standard derviation

Evaluation conclusion

  • Regardless with which method and task users starts, they will train with the first task they do and perform better for the second task compared to those who started with the second task.
  • For easy tasks users are not quicker with AR but make fewer errors.
  • By interpolation we find that AR increases time performances at the resolution of 13x13 grids.
  • AR users results vary less than UM users. This is because AR computes the position of the objects when users with manual have to do it by themselves.


We were happily surprised to see training effects from one method to another.

We were worried our AR app may alter the task too much for this to happen. With the printed manual, the user has to count the cells to place pieces, with the AR app the task becomes a drag and drop task.