Enhancing robotic perception for an autonomous roof bolting using an event-based machine learning framework

The mining sector plays a pivotal role in the development of the world economy (WEF Mining, 2015). Underground mining is increasing to cater to the need for minerals to support the automotive and other industries as well as clean energy technologies (Agusdinata et al., 2022). The rapidly rising demand is spurring the mining industry toward deep underground mining (Ghorbani et al., 2023). However, underground mining presents significant challenges across various dimensions (Dempsey et al., 2018), particularly in human safety (Mahdevari et al., 2014) and in sustainability (Krzemien et al., 2016) and working conditions (Wong et al., 2017). According to a report issued by the U.S. Mine Safety and Health Administration (MSHA), the total number of fatal injuries for the first three quarters of 2023 was higher than that for the previous year (MSHA, 2023). Figure 1 A scenario of rock, strap and power cable area in the Edgar Mine in Idaho Springs, CO, where it is required to identify the drillable regions from nonrock regions. One of the hazards in the underground mining industry that can result in fatalities and serious injuries is falls on the roofs and ribs of underground mines (Kahraman et al., 2016). According to the MSHA, 26 fatal injuries had been reported as of Aug. 5, 2023 (MSHA, 2023). Roof fall hazards are addressed by roof bolting (Mark, 2022), which has been recognized by the Federal Coal Mine Health and Safety Act of 1969 as the exclusive method of providing support for underground entry (Peng, 2023). Roof bolting is a fundamental technique for ensuring the safety and stability of underground mine environments, particularly in areas where roof conditions are unstable or prone to collapses (Rostami et al., 2015). Roof bolting is still an active research area with many aspects that need addressing, including safety during bolting and increasing the efficiency of the bolting process. The roof bolting process mainly includes perforating the unsupported roof, followed by the insertion of a roof bolt and epoxy resin to fasten the overlying roof strata (Peng and Tang, 1984). For rock support, straps are used, resulting in an additional step of drilling through holes in the straps. Although roof bolting improves the safety of mine personnel and the sustainability of mines, the roof bolting process is risky. There are many safety hazards associated with the process of roof bolting, including fatal and nonfatal injuries (Peters, 1994). The fatal and nonfatal injuries associated with roof bolting are caused by six tasks: bolting, handling of materials, temporary roof support, drilling, tramming, and traversing (Sammarco et al., 2016). Airborne dust exposure is also a health hazard associated with roof bolting that has not been addressed jointly with the previous safety hazards in the literature (Rahimi et al., 2023). In response to the safety risks of roof bolters during roof bolting operations and on the basis of a recommendation by the National Institute for Occupational Safety and Health (NIOSH), the current research work was conducted to improve safety and reduce cost by proposing a machine learning model based on computer vision to automate roof bolting. Machine learning for roof bolting The application of machine learning in the mining industry is still growing compared to other fields (Azhari et al., 2023). Leveraging machine learning in the mining industry is essential in driving the industry to be an autonomous and more technologically advanced sector (Elwahab et al., 2023). Deep learning is a subfield of machine learning that enhances the performance of machine learning algorithms especially in computer-vision-based tasks. Autonomous technologies in the mining industry offer numerous advantages by minimizing workers’ exposure to dangerous conditions, increasing safety standards, lowering costs and enhancing efficiency (Hyder et al., 2019). An autonomous system needs to understand the scene and localize objects of interest. Therefore, the first task to automating roof bolting is the perception task to classify rocks from nonrocks and locate where to drill. Manual methods are still used for the localization of bolts in underground roof bolting (Gallwey et al., 2021). In the literature, Singh et al. (2021) and Qi et al. (2021) used machine learning algorithms to detect roof bolts that have been already been installed. However, none of the existing work has addressed the automatic detection of where to drill the holes for roof bolting. The automation of the roof bolting process using machine learning requires the distinguishing of rock from anything that is nonrock, such as the power cables of existing straps. Figure 1 shows a scenario of the working environment where it is required to identify rock areas from a strap and a power cable. In addition, it is required to identify the small holes inside the strap to drill and insert the bolts. Figure 2 A training image with its corresponding ground-truth training mask in Edgar Mine. This pair of images form the training image and the training label for the proposed semantic segmentation model that will produce a segmented image for a given color image. Computer-vision tasks are mainly classified as image classification, object detection and recognition, and image segmentation. Some key challenges for computer-vision tasks in underground mines are poor lighting conditions and dust, which degrade the quality of images and, in turn, degrade the performance of the machine learning algorithm. Preprocessing for image enhancement can be used to improve the performance of deep-learning algorithms (Qi et al., 2021). Image preprocessing is also useful for unsupervised domain adaption where a deeplearning model that has been trained on images captured by traditional cameras (also known as frame-based cameras) can be used for sparse images captured by event-based cameras. In this research, deep-learning-based binary-image semantic segmentation is used to classify pixels that belong to rocks from pixels that belong to nonrock objects as the first step toward autonomous roof bolting. Figure 2 shows a sample of a color image and its corresponding segmented mask. Event cameras Event cameras are bio-inspired cameras that capture the changes of pixel intensity asynchronously as a stream of events instead of collectively at fixed frame rates, which are streamed by regular cameras. An event (x, y, t) is expressed as: where e(x, y, t) is the event value of a pixel located at (x, y) at timestamp t, and ΔL (x, y, t) is the change of the pixel’s intensity between t − 1 and t. Event cameras are characterized by high dynamic range, high temporal resolution, and low blur for fast motion (Gallego, 2022). The high dynamic range arises from the circuitry of the event camera, which outputs the pixel intensity on a logarithmic scale. The high temporal resolution arises from the event camera’s stream of events instead of the whole frame. Due to these properties, event cameras are capable of responding to intensity changes of the scene when the object is moving fast, which means that event images almost do not suffer blur due to fast motion. The aim is to use this type of vision sensor to overcome the issues in environments with poor lighting and airborne dust, such as in underground mines. The principle of operation of event cameras is based on dynamic scenes in which the object of interest is moving, the camera is moving, or the lighting conditions are changing. Figure 3 shows two types of images captured by an event camera. Because images from event cameras differ from images from regular cameras, existing deeplearning algorithms cannot be implemented directly to tasks driven by event camera data. In addition, there is not enough training data for event images especially in underground mining environments (Alonso and Murillo, 2019). The main contributions of the current research work are: Building a deep-learning-based binaryimage semantic segmentation for underground roof bolting using color images.Adapting the deep-learning model to be able to implement binary semantic segmentation for both types of images: images from regular cameras, and images from event cameras. Figure 3 Two types of event camera images: (a) events image and (b) accumulated image. Figure 4 Proposed event-based machine learning framework for semantic segmentation. Images from a traditional camera are used to train the model, while the model can be used to segment images from traditional and event cameras. Proposed model The proposed model is based on formulating the autonomous roof bolting process as a binary model where the goal is to classify what is seen into two regions: rock regions, and nonrock regions. Among the three main tasks in computer vision, image semantic segmentation is the most appropriate method to classify rock areas from any nonrock areas. Image segmentation using a deeplearning convolutional neural network model is promising to provide a highly accurate segmentation model. For the current research work, a deep-learning convolutional neuralnetwork- based image segmentation model called Unet was selected (Ronneberger et al., 2015). Unet is a convolutionalneural- network deeplearning model used for semantic segmentation. The training data are images (color) while the training labels are labeled masks. A labeled mask is an image with a limited number of groups. Each group contains pixels of the same object. Figure 4 shows the proposed deep-learning-based semantic segmentation model using Unet that is composed of two parts: the training part, and the testing part. Training part. The training part of the proposed framework consists of the following main components: Data augmentation: The process of creating more training images to increase the performance of the deeplearning model. In computer-vision tasks, data augmentation is implemented using spatial transformation and intensity transformation (Shorten and Khoshgoftaar, 2019).Color to gray-scale: This image processing is used to adapt the deeplearning model so that the trained model can be used to predict masks for images that are captured by a regular camera and to predict masks for images that are captured by an event camera.UNet model: The semantic segmentation model is implemented using UNet architecture by Ronneberger et al. (2015). UNet is one of the most popular convolutional neural networks used for image segmentation. The architecture of the UNet model consists of two main parts: the encoder and the decoder. The encoder is a conventional convolutional neural network that consists of 3 × 3 convolution layers, linear rectified function, and 2 × 2 maximum pooling. The encoder downsamples the input images at a constant spatial rate with strides of two at each pooling layer, which reduces the computation cost of the model during the training. The decoder upsamples the extracted features by 2 × 2 from the encoder part. The output layer is a 1 × 1 convolution layer to classify each extracted pixel. Testing part. The testing part includes the trained semantic segmentation model, the color to gray-scale component, and the median filtering. The color to gray-scale preprocessing is used to predict masks for test images that are captured by regular cameras while the median filtering preprocessing is used to predict masks for accumulated images that are captured by event cameras. The median filtering on the accumulated events images is the key component to adapt the color-based deeplearning model to be used for the segmentation of images from traditional cameras and images from event cameras. The median filtering removes the noise from the accumulated images and preserves the edges. Experiment Two types of camera were used to collect data from Edgar Mine in Idaho Springs, CO: an Intel RealSense L515 LiDAR camera and two iniVation DVXplorer Mini event cameras. The sensor setup is shown in Fig. 5. The L515 LiDAR camera produces color images, 480 × 680 in size, which are used to train and test the proposed semantic segmentation model. The event camera produces accumulated images of the same size as the color camera. The hyperparameters for the deep-learning model are given in Table 1. Figure 5 Camera setup for collecting real data: two event cameras and L515 LiDAR camera. Table 1 Deep-learning model hyperparameters. Performance evaluation metrics As the segmentation task is a pixel-level classification, the data to be analyzed to evaluate the performance of the proposed model are represented by four variables: TP: True positive, where a pixel of a class of interest (a rock pixel) is correctly classified as pixel of rock region. TN: True negative, where a pixel of a noninterest class (a nonrock pixel) is classified correctly. FP: False positive, where a pixel of a noninterest class is incorrectly classified as pixel of a class of interest. In autonomous roof bolting, a pixel of nonrock region is classified as a pixel of a rock region. FN: False negative, where a pixel of an interest class is incorrectly classified as a pixel of a noninterest class. It is desired to increase the true positive and true negative metrics, while it is required to reduce the false positive and false negative metrics. The confusion matrix is used to show these four variables. From these four measurements, four metrics can be calculated to evaluate the performance of a deep-learning model — accuracy, precision, recall and F1 score — given by the following equations (Khalifa et al., 2020): Each of these four metrics provides specific insights into the performance of the deeplearning model. For example, the accuracy metric shows the correct prediction among all classes. The precision metric indicates the correct prediction of the positive class (rock) and is used to minimize the false positive, while the recall metric indicates how the machine learning model correctly identifies the true positive from all the actual positive samples and is used to minimize the false negative. The F1 score balances between the precision and recall. Results Model performance. Figure 6 shows the performance of the deep-learning-based binary semantic segmentation during the training and validation in terms of loss, while Fig. 7 shows the performance in terms of accuracy. Model testing. The model was tested on color images from a traditional camera and on accumulated images from an event camera. Model testing on images from traditional camera. Figure 8 shows the confusion matrix that visualizes the performance of the proposed deep-learning semantic segmentation model in terms of true positive, false positive, true negative and false negative. It also gives the normalized four metrics defined in Eqs. (2)-(5) as percentages. Figure 9 shows the predictions of the model on testing color images containing rocks and straps only, while Fig. 10 shows the predictions of the model on testing color images that contain rocks, straps and power cables. The images were collected from Edgar Mine. Figure 6 Training and validation learning curves of the losses for the proposed semantic segmentation model. The losses are decreasing and smooth after seven epochs. Figure 7 Performance of the proposed deep-learning-based semantic segmentation model in terms of accuracy during the training and validation. Figure 8 Confusion matrix of the proposed semantic segmentation model tested on unseen color images. Model testing on images from event camera. The same proposed deep-learning-based semantic segmentation model has been tested on accumulated images from an event camera. The accumulated images were collected through dynamic illumination using a blinking light as a method of stimulating the event camera sensors. Figure 11 shows the predictions of the model on testing images from an event camera. The scene contains rocks, straps with holes, and power cables. In addition, some of the testing images contain high dynamic range between lowcontrast regions and high-contrast regions. Figure 9 Binary semantic segmentation of images from a color camera into rock and strap regions. Figure 10 Semantic segmentation of images from a color camera into rock and nonrock with the ability to distinguish multiple nonrock objects. Figure 11 Semantic segmentation of accumulated event images from an event camera. Unseen accumulated images have been segmented to rock and nonrock regions by the proposed deep-learning semantic segmentation model. Discussion From the results of the proposed model, the proposed deep-learning-based semantic segmentation model affords acceptable results for segmenting the input images into two regions (binary segmentation): rock regions and nonrock regions. As can be seen from Figs. 7 and 8, the learning curves are smooth after seven training epochs. The training accuracy and validation accuracy of the proposed model are around 98 percent at a relatively small epoch 14, as shown in Fig. 7, which means that the model could learn fast. From Fig. 8, the model could provide high performance when it has been tested on new color images. The obtained evaluation metrics are: accuracy = 97.36 percent, precision = 99.55 percent, recall = 93.36 percent and F1 score = 96.36 percent. Comparing the learning accuracy in Fig. 7 with the testing accuracy in Fig. 8, the proposed model does not suffer from overfitting or underfitting, which means that the proposed model is robust and can be used in different underground mines. The model succeeded in segmenting color images captured by a traditional camera and classifying the pixels that belong to rocks from pixels that belong to straps. In addition, the model could locate the holes in the straps that belong to rock. Figure 9 shows the testing of the model for the scenario in which the scene contains rocks and multiple straps where some of the straps contain holes. As shown in Fig. 9, the proposed semantic segmentation model could classify small holes inside the straps as rocks, and this will also help for drilling holes in straps safely. Figure 10 shows the ability of the proposed model to distinguish power cables and straps as nonrock regions. Figure 11 shows testing of the proposed deep-learning semantic segmentation model on accumulated images collected by an event camera. Even though the deep-learning model has not trained on images from event cameras, it could provide acceptable results when tested on accumulated event images. As shown in Fig. 11, the model could distinguish straps and power cables as nonrock. This means that the proposed model is not only applicable for new test images, but also applicable for testing images that are different from those used for the training of the model. Conclusion This research work proposes an adaptive deep-learning-based semantic segmentation model to automate roof bolting in underground mine. It addressed the harsh environments in underground mines by using state-of-art neuromorphic-inspired sensors known as event cameras. The obtained results show that the proposed model could classify the scene in an underground mine into regions: rocks and nonrocks. This research work represents a basis for applying a deep-learning-based semantic segmentation model to digitize the mining industry for increased miner safety. Acknowledgment This work is supported by the National Institute for Occupational Safety & Health (NIOSH/Project 75D30121C12206). References WEF Mining (2015) Metals in a sustainable world 2050. In World Economic Forum: Geneva, Switzerland Datu Buyung Agusdinata, Hallie Eakin, and Wenjuan Liu (2022) Critical minerals for electric vehicles: A telecoupling review. Environmental Research Letters, 17(1):013005 Yousef Ghorbani, Glen T Nwaila, Steven E Zhang, Julie E Bourdeau, Manuel Cánovas, Javier Arzua, and Nooraddin Nikadat (2023) Moving towards deep underground mineral resources: Drivers, challenges and potential solutions. Resources Policy, 80:103222 Patrick Dempsey, Lydia Kocher, Mahiyar Nasarwanji, Jonisha Pollard, and Ashley Whitson (2018) Emerging ergonomics issues and opportunities in mining. International Journal of Environmental Research and Public Health, 15(11) Satar Mahdevari, Kourosh Shahriar, and Akbar Esfahanipour (2014) Human health and safety risks management in underground coal mines using fuzzy topsis. Science of the Total Environment, 488:85–99 Alicja Krzemien´, Ana Suárez Sánchez, Pedro Riesgo Fernández, Karsten Zimmermann, and Felipe González Coto (2016) Towards sustainability in underground coal mine closure contexts: A methodology proposal for environmental risk management. Journal of Cleaner Production, 139:1044–1056 Cuebong Wong, Erfu Yang, Xiu-Tian Yan, and Dongbing Gu (2017) An overview of robotics and autonomous systems for harsh environments. In 23rd International Conference on Automation and Computing (ICAC), pp. 1–6 U.S. Department of Labor, Mine Safety and Health Administration (2023) Mine safety and health at a glance 2023. https://www.msha.gov/sites/default/files/DataReports/Charts/FY2023 Sair Kahraman, Jamal Rostami, and Ali Naeimipour (2016) Review of ground characterization by using instrumented drills for underground mining and construction. Rock Mechanics and Rock Engineering, 49:585–602 U.S. Department of Labor, Mine Safety and Health Administration (2023) Fatality alert 2023. Report accidents and hazardous conditions: 1-800-746-1553 Christopher Mark (2002) The introduction of roof bolting to US underground coal mines (1948-1960): a cautionary tale Syd Peng (2023) Roof bolting and underground roof falls. Geohazard Mechanics, 1(1):32–37 Jamal Rostami, Sair Kahraman, Ali Naeimipour, and Craig Collins (2015) Rock characterization while drilling and application of roof bolter drilling data for evaluation of ground conditions. Journal of Rock Mechanics and Geotechnical Engineering, 7(3):273–281 Syd S Peng and DHY Tang (1984) Roof bolting in underground mining: a state-of-the-art review. International Journal of Mining Engineering, 2:1–42 Robert H Peters (1994) Improving safety at small underground mines. In: Proceedings of Bureau of Mines technology transfer seminar JJ Sammarco, A Podlesny, EN Rubinstein, and B Demich (2016) An analysis of roof bolter fatalities and injuries in US mining. Transactions of Society for Mining, Metallurgy, and Exploration 340(1):11 Elham Rahimi, Younes Shekarian, Naser Shekarian, and Pedram Roghanchi (2023) Investigation of respirable coal mine dust (rcmd) and respirable crystalline silica (rcs) in the US underground and surface coal mines. Scientific Reports, 13(1):1767 Faris Azhari, Charlotte C Sennersten, Craig A Lindley, and Ewan Sellers (2023) Deep learning implementations in mining applications: a compact critical review. Artificial Intelligence Review, pp. 1–36 Ahmed Abd Elwahab, Erkan Topal, and Hyong Doo Jang (2023) Review of machine learning application in mine blasting. Arabian Journal of Geosciences, 16(2):133 Zeshan Hyder, Keng Siau, and Fiona Nah (2019) Artificial intelligence, machine learning, and autonomous technologies in mining industry. Journal of Database Management (JDM), 30(2):67–79 Jane Gallwey, Matthew Eyre, and John Coggan (2021) A machine learning approach for the detection of supporting rock bolts from laser scan data in an underground mine. Tunnelling and Underground Space Technology, 107:103656 Sarvesh Kumar Singh, Simit Raval, and Bikram Banerjee (2021) A robust approach to identify roof bolts in 3d point cloud data captured from a mobile laser scanner. International Journal of Mining Science and Technology, 31(2):303–312 Yunliang Qi, Zhen Yang, Wenhao Sun, Meng Lou, Jing Lian, Wenwei Zhao, Xiangyu Deng, and Yide Ma (2021) A comprehensive overview of image enhancement techniques. Archives of Computational Methods in Engineering, pages 1–25 Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J. Davison, Jörg Conradt, Kostas Daniilidis, and Davide Scaramuzza (2022) Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):154–180 Inigo Alonso and Ana C Murillo (2019) Ev-segnet: Semantic segmentation for event-based cameras. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops Olaf Ronneberger, Philipp Fischer, and Thomas Brox (2015) U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241. Springer Connor Shorten and Taghi M Khoshgoftaar (2019): A survey on image data augmentation for deep learning. Journal of big data, 6(1):1–48 Nour Eldeen M Khalifa, Mohamed Hamed N Taha, Aboul Ella Hassanien, and Sarah Hamed N Taha (2020) The detection of COVID-19 in CT medical images: A deep learning approach. Big Data Analytics and Artificial Intelligence against COVID-19: Innovation Vision and Approach, pp. 73–90

Enhancing robotic perception for an autonomous roof bolting using an event-based machine learning framework

Authored by:
Rik Banerjee, Akram Marseet, Andrew J. Petruska

To view the past issues of the UCA magazine you must be a member of SME/UCA.

From sessions to strategy: Revealing long-term and emerging innovation trends in data analytics, intelligent systems and automation

Mining education gets federal boost as industry faces workforce crunch

Critical minerals management on the federal seabed

From sessions to strategy: Revealing long-term and emerging innovation trends in data analytics, intelligent systems and automation

Mining education gets federal boost as industry faces workforce crunch

Critical minerals management on the federal seabed

To view the past issues of the UCA magazine you must be a member of SME/UCA.

The road to zero: The 50‑year effort to eliminate roof fall fatalities from US underground coal mines

Preparing America’s infrastructure for the massive adoption of clean energy

Related Articles

Copper stockpiling, transformation capacity and industrial synchronization in the US

From sessions to strategy: Revealing long-term and emerging innovation trends in data analytics, intelligent systems and automation

Historic day in Washington DC for mining education

Critical minerals: Technical and policy considerations

From sessions to strategy: Revealing long-term and emerging innovation trends in data analytics, intelligent systems and automation

Mining education gets federal boost as industry faces workforce crunch

Critical minerals management on the federal seabed

From sessions to strategy: Revealing long-term and emerging innovation trends in data analytics, intelligent systems and automation

Mining education gets federal boost as industry faces workforce crunch

Critical minerals management on the federal seabed

Use Single Sign On