OPERA - Outil de formation Professionnelle et d’Enseignement basée sur la Réalité Augmentée

As part of their professional training, apprentices are often required to study, understand and reproduce tasks and activities while respecting rigorous processes, which often define the sequencing of the steps and tasks to be accomplished, and also define for each of them the needed tools, the targeted results as well as the related best practices. As examples of professions based on such processes, we can mention cooking or baking with the fulfillment of specific recipes, site surveillance with gestures and rigorous operations to be carried out, and table setting with codes and rules to respect.

The objective of the collaboration between Orif and “I3” is to develop a solution that is generic, innovative, and rich. First, the solution must be generic to be able to cover various trainings and adapt to various processes. Second, the solution must be innovative by allowing apprentices to practice real scenarios with strong immersion: apprentices must feel like in a real situation and have the feeling of being in normal working conditions. Finally, the solution must involve adequate and comprehensive training tools allowing to cover multiple use cases: time-free practice, time-constrained practice, examination, and so on.

To achieve the required solution and be able to use it in a given context, we adopted an Agile approach to validate the intermediate choices and compare them to real use cases. Initially, we worked on a generic model enabling the definition of the concept of “process” as well as “step” of a process. This modeling work had to be done in a generic and flexible way so as to ensure the need to generate the final solution. In a second step, we selected a particular training and we instantiated our sorting model to cover it. Finally, we developed and selected the virtual objects, the tools and the mechanisms covering all the stages of the selected training process. As for the process model, we opted for a representation in the form of an oriented graph with the steps as nodes and the precedence and dependency constraints as arcs. Thanks to this representation, we

are not limited to linear and sequential processes but cover more complex and more varied situations. Each node in our graph represents a specific step (in the process), which is in turn linked to a set of objects and tools allowing the apprentice to carry it out. Each step is therefore associated with:

• Theoretical resources allowing revisions and/or theoretical reminders to be done during the training sessions.
• Virtual interactive tools available during this stage and helping the apprentice to accomplish his task. For example, in the case of a recipe preparation process, unit conversion tools or dosing tools may be required.
• Virtual objects to be manipulated to accomplish the task. These objects strongly depend on the task and must be coupled with rules and controls, specific to each of them. For example, in the case of a training for table setting, a virtual object can be a plate to be placed by the apprentice. This requires controlling what plate is selected, the way in which it is placed on the table and finally its relative position in relation to the other elements of dressing.
• Review and tracking tools. Indeed, the developed solution must ensure a follow-up and a quantitative evaluation of the apprentice’s actions to be used for debriefing.

This model has been implemented in a generic way allowing it to be instantiated (applied) in different ways according to the targeted training needs.

Three major innovations have been introduced in this project. The first one is the genericity of the training tool so that it can be easily and quickly adapted to different training courses, the only constraint being that the training is based on a step-by-step process with a defined sequence and identifiable results. The second innovation is the intensive use of mixed reality to maintain a high level of realism while manipulating virtual objects. This mixed reality allows the apprentice to be in a real work environment and thus feel its constraints and limits. At the same time, mixed reality makes it possible to enrich this real framework with virtual objects that the apprentice can manipulate, move, and interact with them without being limited by the objects’ cost, their availability and the security aspects. The third innovation is gamification. Indeed, with the aim of increasing the acceptability and attractiveness of the training tool, we introduced various gamification aspects to enhance user motivation. Among the gamified aspects introduced we can mention:

• Scoring: accompanying the apprentice's operations with score and rewards.
• Missions: training can be performed in the form of missions, which can be carried out in single or group modes, and also in cooperative and/or competitive mode.
• The time constraint: some training sessions may be limited by time limit to intensify the challenge and put the apprentice in a stressful situation.

Currently the project is still in the development phase. The first usable prototype we have targeted is a training tool on the art of table setting, which will cover all the stages of table setting: tablecloths, napkins, cutlery, glasses, decor, and so on. It also allows to situate the apprentice in different real scenarios such as buffets, à la carte dinners, and group meals. The tool has three modes:

• The training mode consists of controlling the actions of the apprentice and guiding him via instructions, notifications, and theoretical reminders.
• The mission mode consists of controlling the apprentice without intervention while keeping logs allowing to debrief his work and note it.
• The exam mode is like the mission mode, but it adds the time constraint.

The tool has been developed with Unity and deployed on Microsoft Hololens 2 AR glasses. The objective is to be able to carry out tests with real users during next year’s fall semester (2022-2023).