Augmented-reality system for training of heavy-equipment operators in surface mining

The mining industry’s commitment to innovation reflects a history of adopting technological advancements to enhance workplace safety, environmental sustainability and vocational training. The objective of this research was to develop an augmented-reality (AR) system for heavy-equipment operators (HEOs) in surface mining. The developed system has the potential to enhance mine safety, training and data-driven decision-making, which presents a significant step toward a more sustainable, effective and technologically driven mining training, contributing to the industry’s evolution and growth. The AR training system leverages Microsoft’s Power Platform and HoloLens 2 capacities to provide operators with detailed and immersive training guides for three mining equipment, including bulldozers, motor graders and haul trucks. AR guides combine speech controls, 3D objects, informative images and explicative videos in the real working conditions to enhance the operator’s learning process. The system has the potential to modify the training structure based on user performance and provides an efficient approach to data collection during training. This project was developed and applied through a case study in a surface mine in the Southern United States. Background U.S. federal law outlines in the Code of Federal Regulations Title 30 the permissible training methods in mining, which includes classroom instruction, on-site training, interactive computer-based instruction or any other technology-based training methods. The U.S. Mine Safety and Health Administration (MSHA) recognizes computer-based and interactive training technologies as viable methods and establishes that the presence of a competent person may not necessarily be required in a training room. However, lack of structured training and inadequate equipment inspections continue to contribute to mining injuries. Thus, modernizing and innovating training programs that effectively engage inexperienced employees is still an urgent challenge for the sector. Heavy equipment in mining is frequently under extreme conditions, resulting in exponential wear, abrupt breakdowns or failures. Therefore, HEOs are required by law to inspect and ensure the equipment is free from defects prior to operation. Failure to conduct a proper pre-operation inspection may lead to machinery wear-related accidents, causing considerable injuries to employees and damage to the equipment, affecting the company’s safety goals, profitability and reputation. The complexity of equipment inspections relies on factors such as machinery type, operator experience, machine wear, allotted time, and knowledge gained through training. Hence, standardized training can bring a high value to the entire mining process, minimize operational errors and improve equipment efficiency. Methodology This research leverages the capacities of the Microsoft Power Platform and HoloLens 2 device to develop, display, store and analyze the AR training system. It uses cloud computing services such as Dynamics 365 Guides, Power BI, Power Apps and others to integrate a single business solution, providing robust security, reliability and scalability. HoloLens 2 facilitates interaction through hand movements, voice commands, spatialized sound, eye tracking and real-world anchoring, creating an immersive mixed-reality environment for effective operator engagement. The system comprises three AR training guides, an evaluation application and data management tools for bulldozers, motor graders and haul trucks. Each guide provides standardized step-by-step instructions supported by interactive content anchored to a unique QR code. The interaction with the system, including operator’s gaze patterns and task completion times, are recorded for analysis through the Admin Center and Power BI, providing training adoption insights and performance trends. Holographic images and videos offer dynamic instructions for complex procedures and emphasize safety compliance, while 3D content indicates specific components and action sequences in real working conditions. At the end of the training session, an evaluation application is displayed to assess the knowledge acquired through an actual inspection. This system not only enhances training effectiveness and safety compliance but also provides data-driven insights to refine training strategies over time. Fig. 1 AR training guides in the mining environment. Results and discussion This AR training system presents an innovative way to engage employees more effectively, store training records and obtain performance insights. The system demonstrated remarkable versatility and scalability due to QR-code technology, which allows training guides to adapt to different locations or mine sites and facilitates follow-ups at local and corporate levels. However, the precision of training may vary depending on equipment models, as component locations can differ. QR codes must be affixed in the same positions as during guide development to ensure accurate holographic overlays. The training is structured into sequential tasks and steps, reflecting the inspection workflow: Safety Task, Front View, Right View, Rear View, Left View, Cabin/Deck and a Pre-operation Inspection Test. Each training guide incorporates six primary tasks, with steps tailored to the complexity and location of equipment components. Trainees receive instructions right in the equipment, enhanced by holographic information developed in collaboration with mine personnel, as shown in Fig. 1. The dynamic content automatically appears and disappears as operators trigger voice commands, eliminating the need for manual interaction with the AR information. Fig. 2 Evaluation training app launched in an augmented-reality space. Quizzes integrated at the end of each task assess trainees’ comprehension before advancing to subsequent tasks. Besides, a final evaluation requires trainees to perform an actual pre-operation inspection, which is also recorded in compliance with MSHA regulations. Figure 2 illustrates this feature.The system provides detailed insights by users and mine location, including time spent per step and task complexity, as showcased in Fig. 3. Trainers can leverage this data to evaluate system efficacy, refine content and offer targeted guidance. Scalability remains a standout feature, enabling centralized management of training data across mines or even countries by using QR-code technology. Conclusion Safety and effective training are critical in the mining industry to ensure operational success. This research developed an AR training system for bulldozers, motor graders and haul trucks to provide step-by-step holographic instructions and enhance pre-operation inspection training. By integrating dynamic visual aids like holographic images, videos and 3D objects, the system creates a hands-on and immersive experience that improves safety, reduces costs and streamlines processes. Fig. 3 Historical data visualization of training sessions. The standardized and flexible AR training guides include in situ instructions, quizzes and evaluation tasks to ensure operators master procedures before operating a piece of heavy equipment, fostering competence and compliance with MSHA standards. Furthermore, digital and centralized data management tools allow decision-makers to analyze performance, address weaknesses and personalize guidance across multiple mine sites through portable QR codes. References A list of all references is available in the full paper.

Augmented-reality system for training of heavy-equipment operators in surface mining

Authored by: Juan David Valencia Quiceno, Vladislav Kecojevic, Amy McBrayer and Dragan Bogunovic

To view this article you must be a member of SME/UCA.

Gold retains key reserve status despite surging bond yields, central bankers say

China demands copper supply commitments for Anglo Teck merger approval, sources say

Leaders in mining excellence honored by NMA, federal agencies

Gold retains key reserve status despite surging bond yields, central bankers say

China demands copper supply commitments for Anglo Teck merger approval, sources say

Leaders in mining excellence honored by NMA, federal agencies

To view this article you must be a member of SME/UCA.

ICMM and INAP launch new tool to strengthen water pollution prevention in mining

Challenges in the battery raw materials supply chain: Achieving decarbonization from a mineral-extraction perspective

Related Articles

Rare earth elements in the global economy: Usage, recovery and the quest for supply security — A review

A systematic evaluation of localization signals in the advancing openpit mine Aitik

From foundation to future: Revisiting AI integration in mining engineering education through current perspectives of students, educators and industry

Iron ore tailings reprocessing by electrostatic separation plus two-stage froth flotation

Gold retains key reserve status despite surging bond yields, central bankers say

China demands copper supply commitments for Anglo Teck merger approval, sources say

Leaders in mining excellence honored by NMA, federal agencies

Gold retains key reserve status despite surging bond yields, central bankers say

China demands copper supply commitments for Anglo Teck merger approval, sources say

Leaders in mining excellence honored by NMA, federal agencies

Use Single Sign On