Using generative pre-trained transformer-4 (GPT-4), FFMPEG and Microsoft Azure to aid in creating a text-to-video generation tool to improve safety shares and incident descriptions in the mining industry

Traditional safety talks often rely on static images, PowerPoint slides or verbal explanations that may not adequately convey the sequence, context and seriousness of an incident. To address this weakness in mine safety communication, this paper presents a practical artificial intelligence (AI)-assisted text-to-video tool designed to improve safety shares and incident descriptions in the mining industry by turning written incident narratives into short, narrated videos. By combining generative pre-trained transformer-4 (GPT-4) for scene generation and matching, Microsoft Azure for text-to-speech and FFMPEG for audio-video synchronization and assembly, the proposed workflow offers a lower-cost and more scalable alternative to custom 3D simulation. If applied in real mining operations, the tool could make prestart meetings and toolbox talks more realistic, memorable and discussion-oriented, while also helping safety teams simulate not only past incidents but also potential future hazards for proactive risk communication. Background Mining remains one of the most hazardous industries because of its dynamic environment, heavy equipment, confined spaces, and frequent worker-machine interaction. Although mine sites routinely use prestart meetings and toolbox talks to strengthen safety culture, these practices often lack realism and direct relevance to workers’ immediate tasks. Existing incident communication methods often rely on post-incident photographs or slide presentations, which may not fully capture the sequence of actions that led to an accident. Video-based reconstruction may be a more effective medium because moving images, narration and temporal sequencing can better support understanding, retention and transfer of safety lessons than text alone or static visuals. While professionally produced 3D simulations are useful, they are often too costly and time-consuming for routine use, as shown in Fig. 1. For that reason, this study explores whether readily available AI tools can support a more accessible approach. The proposed solution is built on GPT-4’s natural language capabilities, Azure’s text-to-speech service, and licensed stock video databases. Methods The model was developed in Python using OpenAI, Microsoft Azure, FFMPEG and Flask. The computing environment consisted of an Intel Core i5-9500 CPU with 16 GB of RAM. The software architecture consisted of about 20 scripts, including three core pipelines for generating the introduction, incident description and conclusion sections of a final video. Fig. 1 Animation types and cost ranges for different3D animation types. A key step in the workflow was the creation of three comma-separated values (CSV)-based media databases containing licensed underground-mining videos. Each entry included the file path, a manually written description of the visual content and the clip duration. There are separate databases for the introductory, incident description, and concluding scenes. GPT-4 was then prompted with custom instructions to perform several linked tasks: Convert a user’s incident description into a sequence of simple scenes; match each scene to the most appropriate video description in the relevant database; generate introduction and concluding segments from the incident context; and produce short legends for the conclusion. After scene text was extracted using regular expressions, Azure converted the text to speech, and FFMPEG measured, adjusted and synchronized the lengths of the audio and video clips before merging them into a complete final product. A Flask server was also built to provide a simple user interface that lets users paste an incident description and generate a video. Results The paper tested the system primarily using a real underground vehicle-interaction incident drawn from WorkSafe New Zealand. A link to the video generated by this AI tool is available in the original paper. In the scenario, a loader operator misjudged the bucket’s lift height while a dump truck reversed into position, resulting in entanglement between the loader bucket and the truck tub and causing the loader to tip over. The model successfully generated a structured narrative with an introduction, multi-step incident description and conclusion, and then matched scenes to stock videos, ensuring logical consistency with the event progression. Fig. 2 Steps for the model development. In the first example, the final video lasted one minute and eight seconds and included an introduction emphasizing coordination in underground mining, a central sequence depicting the loading and reversal process, and a conclusion highlighting communication failure, improper signaling and insufficient safety measures. In a second test using the same source incident, the model produced a different but still coherent narrative and selected a somewhat different set of clips, demonstrating variability across runs. This variability may be beneficial because it allows the same incident to be reframed in fresh ways while preserving the overall safety message. The tool took about 10 minutes to generate each video under the stated computing conditions. The preliminary cost comparison shows that building the model required about $20 in AI API use during development, and individual runs were estimated to cost only a few cents in token charges, excluding variability introduced by prompt length. This represents a substantial economic advantage over custom 3D animation or simulation, which can cost thousands of dollars per minute. Discussion The main contribution of the paper is showing that a generative-AI workflow can be adapted into a practical safety-training aid rather than remaining a purely experimental demonstration. The strongest aspect of the system is its integration of narrative generation, scene sequencing, media matching, voice synthesis and video assembly into a single, simple workflow for end users. A safety professional does not need advanced video production skills; they only need to enter an incident description. This ease of use is central to the paper’s value because mining sites often lack both the time and personnel needed for custom simulation production. Despite the advantages, there are some technical limitations. First, system performance depends heavily on the quantity and quality of the available video database. If no close visual match exists for a generated scene, GPT-4 will still choose the nearest option, which can reduce coherence. Second, because the final video is assembled from different stock clips, the visual continuity of people, vehicles and locations may be imperfect. Third, video length cannot yet be tightly controlled because it depends on the number of scenes generated from the prompt. Finally, access through the Flask server was restricted by university VPN constraints, limiting immediate broader deployment. However, there are plausible paths forward, including site-specific databases built from each mine’s own archival media, deployment as a standalone application, and use of multimodal AI to automate video description rather than requiring manual metadata entry. These proposed improvements are important because they would make the model more accurate, more customized to local operating conditions, and easier to maintain over time. Conclusions This study shows that AI-assisted text-to-video generation can strengthen safety communication in mining by making incident descriptions more vivid, sequential and discussion-provoking. Its practical significance is twofold: it offers a cost-effective substitute for many forms of custom simulation and supports a shift from reactive safety review to more proactive hazard visualization. Although the current system is constrained by database size, clip consistency, and deployment limitations, it demonstrates that widely available AI services can be combined into a functional tool for industrial safety education. Beyond underground mining, the same framework could be adapted for other sectors, training contexts or instructional materials, wherever turning text into coherent, narrated visual sequences would improve understanding and engagement. References A list of all references is available in the full paper.

Using generative pre-trained transformer-4 (GPT-4), FFMPEG and Microsoft Azure to aid in creating a text-to-video generation tool to improve safety shares and incident descriptions in the mining industry

Authored by: Tulio Dias de Almeida, Natanna Nunes de Oliveira, Chandi He, Carlos Philipe Silva Rocha, Marcelo Bandeira Teixeira, Pratt Rogers and Karoly Charles Kocsis

To view this article you must be a member of SME/UCA.

Mining Engineering’s Weekly Rundown: Episode 73

Get your SME Foundation VIP Access Conference Pass to MINEXCHANGE 2027

“Check the box” and help support the SME Foundation

Mining Engineering’s Weekly Rundown: Episode 73

Get your SME Foundation VIP Access Conference Pass to MINEXCHANGE 2027

“Check the box” and help support the SME Foundation

To view this article you must be a member of SME/UCA.

Recyclability assessment and design for recycling recommendations of end-of-life NMC811 lithium-ion batteries

A juxtaposition of autonomous haulage systems in China and Australia: Technological, economic and social perspectives

Related Articles

Rare earth elements in the global economy: Usage, recovery and the quest for supply security — A review

A systematic evaluation of localization signals in the advancing openpit mine Aitik

From foundation to future: Revisiting AI integration in mining engineering education through current perspectives of students, educators and industry

Iron ore tailings reprocessing by electrostatic separation plus two-stage froth flotation

Mining Engineering’s Weekly Rundown: Episode 73

Get your SME Foundation VIP Access Conference Pass to MINEXCHANGE 2027

“Check the box” and help support the SME Foundation

Mining Engineering’s Weekly Rundown: Episode 73

Get your SME Foundation VIP Access Conference Pass to MINEXCHANGE 2027

“Check the box” and help support the SME Foundation

Use Single Sign On