Teaching mining engineering in an AI World

Students on a visit to the Cortez Hills Gold Mine in Lander and Eureka County, NV. (Image courtesy of the University of Nevada, Reno) Universities and other educational facilities faced many challenges during the COVID-19 pandemic. Now, as teaching increasingly returns to traditional face-to-face methods, artificial intelligence (AI) systems like ChatGPT pose fresh challenges. This article presents some of the author’s recent encounters with ChatGPT while teaching at the University of New South Wales (UNSW) in the bachelor of engineering (mining engineering) program. Synopsis The emergence of COVID-19 resulted in UNSW adopting remote teaching methods for much of the 2020–22 school years. As vaccination rates improved, face-to-face laboratories and tutorials were gradually reintroduced from late 2021 on. During the period of COVID-19 restrictions, however, examinations were delivered online. They were effectively unsupervised and open-book exams, which is not necessarily a path to better marks. Students with strong subject knowledge will tend to minimize their time-consuming reference to texts. In addition, examiners often expand the open-book exam content to distinguish students with knowledge from those with efficient look-up skills. As exam time pressure increases, an apparent solution — albeit one that compromises academic integrity — is to seek “assistance” from AI tools such as ChatGPT. In its current form however, ChatGPT lacks technical competence in highly specialized subjects, such as mining engineering and as a result, AI’s current capabilities with respect to these specialized subjects fall well short of competent. ChatGPT Developed by OpenAI, ChatGPT uses a natural language processing model — it is an online tool designed to answer questions, drawing on AI technology. A free version was released in November 2022 and a paid, premium version, ChatGPT Plus, became available in February 2023 at a cost of $20 a month. Whether free or paid, OpenAI has publicly acknowledged the limitations of the tool as follows: The ChatGPT landing page, where a user enters their question, carries a clear warning: “ChatGPT may occasionally generate incorrect information.”An OpenAI promotion notes that ChatGPT “sometimes writes plausiblesounding but incorrect or nonsensical answers” (OpenAI, 2022). The same article also states: “The model is often excessively verbose and overuses certain phrases” (OpenAI, 2022). So how effective can ChatGPT be in highly technical areas like mining? In this article, parts of the JORC 2012 Code have been used to test ChatGPT and in these tests, ChatGPT’s limitations are evident. JORC 2012 Code In Australasia, the JORC Code (JORC Code 2012) sets out standards for the public reporting of exploration results, mineral resources and ore reserves. All Australian Stock Exchange (ASX)-listed mining and exploration companies must adhere to this code and its key tenets of materiality, competency and transparency. Competency is particularly relevant here. Any author claiming to report in accordance with the code must be deemed a competent person. In brief, a competent person is defined as: “A minerals industry professional who is a member of the Australasian Institute of Mining & Metallurgy (AusIMM) … with a minimum of five years relevant experience in the style of mineralization or type of deposit under consideration and in the activity which that person is undertaking.” Testing ChatGPT The author teaches the UNSW third-year undergraduate course on mineral resource estimation. The JORC Code is an important component of that and other related UNSW courses. The code is widely used and understood in the Australian mining and exploration industry and so provides an ideal test for ChatGPT. Two basic JORC questions were put to the chatbot: Question 1: Marketable reserves. Under the JORC Code, a competent person reporting on coal reserves is required to detail both runof- mine (ROM) and marketable coal reserves (Clause 44, Reference 2). As coal is typically washed, the marketable reserve estimate (the saleable component) may be as much as 20 to 30 percent lower than the ROM estimate. Detailing marketable reserves is mandatory when reporting coal reserves in accordance with the JORC Code. The simple question put to ChatGPT was: “Under JORC 2012, what commodities must report marketable reserves?” The chatbot responded, with a verbose 187 words, that reporting of marketable reserves applies to all commodities. While incorrect, this could well appear reasonable to a lay reader — why wouldn’t all commodities report marketable reserves? In this test, ChatGPT displayed its limitations, providing an answer that was both verbose and incorrect. Table 1 Subset of JORC 2012. Question 2: If not, why not? The second question put to ChatGPT considered the JORC term ‘if not, why not’. This term relates to criteria that must be addressed when submitting a JORC resource or reserve estimate. These are listed within the code as ‘JORC 2012 Table 1,’ a small extract of which is shown below. In full, JORC 2012 Table 1 includes five sections: Sampling Techniques and Data (12 criteria). Reporting of Exploration Results (10 criteria). Estimation and Reporting of Mineral Resources (15 criteria). Estimation and Reporting of Ore Reserves (17 criteria). Estimation and Reporting of Diamonds and Other Gemstones (10 criteria). The code mandates that all Table 1 criteria relevant to the competent person’s task be addressed. For instance, a competent person reporting on exploration results and resources would complete Sections 1 and 2, or 1 and 3 respectively. Section 4 would be completed by the competent person reporting on reserves. Sections 1, 2 and 3 would normally be carried out by a geologist while the Section 4 competent person is normally a mining engineer who should have access to the report prepared by the competentperson geologist. However, the code also mandates that if an item is not completed, the competent person must explain why — hence ‘if not, why not’ —meaning in effect that a correctly completed JORC 2012 Table 1 includes a response to every criterion in the appropriate section. ChatGPT was asked to “Explain, in 100 words, what is meant by the expression ‘if not, why not’ in regard to JORC Table 1” and responded: “The phrase “if not, why not” is used in JORC Table 1 to encourage transparency and accountability in reporting mineral resources and reserves. It requires the reporting company to disclose any factors or reasons that may prevent a particular mineral resource or reserve from being categorized at a higher level of confidence. This means that if a company reports a mineral resource or reserve that is lower than expected, they must explain why it is not at a higher level of confidence. This encourages companies to provide clear and accurate information to investors, which helps them make informed decisions about the investment potential of a mining project.” In 107 words, ChatGPT missed the key aspects of JORC Table 1 and ‘if not, why not,’ namely that: ‘If not, why not’ is a competent person’s responsibility, not a company’s responsibility.‘If not, why not’ has nothing to do with categorization or confidence, nor per se does Table 1.JORC Table 1 is a checklist of criteria requiring explanation, but they do not individually or collectively lead to a lower-than-expected categorization. As an UNSW examiner, I would score the ChatGPT answer a mark of zero. ChatGPT clearly does not understand the subject matter. The answer does, however, read well and to a lay reader, it may seem perfectly reasonable that companies would provide this information. In an additional test, the limiting ‘in 100 words’ phrase was removed from the question. ChatGPT then returned 147 words, but its accuracy did not improve. In conclusion, for its ‘if not, why not’ test, ChatGPT is again both verbose and incorrect. In the real world UNSW’s Resource Estimation course covers the JORC Code and the ‘if not, why not’ question was posed in final exams over 2020–23. Prior to the exams, students completed Sections 1, 2 and 3 of JORC Table 1 as an assignment task and consequently, scored well on the exam question: an average of 86 percent over the four years. However, 10 percent of students fared poorly or failed to answer. There was little middle ground — students either knew the answer and scored highly, or didn’t. Student word counts were also low. Students achieved full marks with an average of just 53 words, compared to ChatGPT’s 107 or 147, for zero marks. Conclusion From June 2023, all UNSW faculties will return to face-to-face teaching and supervised on-campus exams, and concerns about the use of ChatGPT in exam conditions will abate. Nonetheless, the JORC Code tests show ChatGPT, in its current form, to be verbose and erroneous in technical areas. Though its responses may initially appear logical (Why wouldn’t all commodities report marketable reserves? Shouldn’t a company be responsible for JORC Table 1 criteria?), they are simply incorrect. As a final comment, the ChatGPT responses from this article will be included in the 2024 Resource Estimation course at UNSW, and in JORC Code tutorial sessions. References OpenAI (Nov 30 2022) “ChatGPT: Optimizing Language Models for Dialogue” Retrieved May 13, 2023. JORC Code 2012. www.jorc.org.

Teaching mining engineering in an AI World

Authored by:
Jon Barber

To view the past issues of the UCA magazine you must be a member of SME/UCA.

From sessions to strategy: Revealing long-term and emerging innovation trends in data analytics, intelligent systems and automation

Mining education gets federal boost as industry faces workforce crunch

Critical minerals management on the federal seabed

From sessions to strategy: Revealing long-term and emerging innovation trends in data analytics, intelligent systems and automation

Mining education gets federal boost as industry faces workforce crunch

Critical minerals management on the federal seabed

To view the past issues of the UCA magazine you must be a member of SME/UCA.

Reinflation testing of inflatable friction rock bolts

The Tailings Center developments in 2023

Related Articles

From sessions to strategy: Revealing long-term and emerging innovation trends in data analytics, intelligent systems and automation

Copper stockpiling, transformation capacity and industrial synchronization in the US

Historic day in Washington DC for mining education

Critical minerals: Technical and policy considerations

From sessions to strategy: Revealing long-term and emerging innovation trends in data analytics, intelligent systems and automation

Mining education gets federal boost as industry faces workforce crunch

Critical minerals management on the federal seabed

From sessions to strategy: Revealing long-term and emerging innovation trends in data analytics, intelligent systems and automation

Mining education gets federal boost as industry faces workforce crunch

Critical minerals management on the federal seabed

Use Single Sign On