Open letter to the organizer of the tailings dam round robin exercise
Mar 29th, 2023
Dear Ryan, please receive this open letter to the organizer of the tailings dam round robin exercise. It explains our position on the round robin exercise you have promoted.
As you know we enthusiastically adhered to your round robin proposal although we found a bit odd that such an exercise would be led by a consultant, thus not an impartial entity. Very quickly we started asking for more data, as the knowledge base seemed rather generic. We were not surprised to see that other participants asked more data as well. The process seemed to reflect what happens in real-life when we undertake a new risk assessment. During the first steps of the process we started noticing some peculiarities in the round robin formulation that we discuss below.
The round robin structure is too open.
For instance, we noted that despite the Q/As the design of the round robin had significant problems. These would give poor validity to the interpretation of the results. Indeed, a classic problem in comparative studies has been the presence of too many variables and too few cases.
- In the present round robin proposal, each participant had to fulfill the information gaps with their own assumptions. This means that many variables were left “unbounded”. Furthermore,
- participants did not state their prior experience on the specific subject. This added an additional layer of uncertainty to the interpretation. As a matter of fact, in the examples of round robin exercises we found in the literature (see later in this text), which were also dealt anonymously, the experience of the participant was the first question, to remove one layer of uncertainty.
- Researchers used the term Engineering Modelling Uncertainty (EMU) to describe the importance of subjective decisions performed in the transformation from drawings/raw data to “computational” models. Indeed this uncertainty has large impact on “safety” and evaluations. In our 2016 paper (Oboni, F., Oboni, C., A systemic look at tailings dams failure process, Tailings and Mine Waste 2016, Keystone, Colorado, USA, October 2-5, 2016) we cited “excessive audacity” as one of the hazard families driving subjective decisions. That is obviously distinct from the validity or consistency of computational or evaluation models/tools.
Specific examples of excessive “openness” leading to results’ misinterpretation
- Undefined steady state. Each participant is free to propose a definition of steady state loadings and parameters for the two selected failure modes. This would lead to a significant spread of the results even with classic FoS analyses.
- The specifications do not state if the probability of failure is annualized or “timeless”. That means that participants that attempt to use a “geomechanical model” will end up with a “timeless” PoF, as opposite to other approaches.
- The presence of the buried pipe should lead to some detailed specifications related to the failure mode it applies to. Indeed, again, the decision to model the pipe failure in the stability failure mode vs. the internal erosion failure mode will generate fairly different results.
On top of the comments above we note that:
- Failure modes probabilities for “siloed” analysis are not representative of future behavior. Thus the round robin misses the important goal of aggregating the failure modes (at least the two requested). Furthermore,
- Excluding climate change or simply extreme events is in our mind a major fault leading to misleading appreciation. It would be nefarious if readers of the final report would associate the FoS=1.6 to the non-aggregated probability of failure under an array of steady state environments.
Standard stability methods give spread out results as well.
The present round robin would be like trying to assess the consistency of limit equilibrium method (LEM) results, based on the assumptions engineers make on the geotechnical parameters, and not on the actual methods. The EMU would be mixed in with the differences between computational methods such as Fellenius, Bishop, Morgenstern & Price, Spencer, 3-D. Indeed, whether run by hand or using various commercial software these methods deliver different results even with the same geotechnical and geometric parameters. The range of those results measures the consistency of the LEM. If one lets the engineers freely select various assumptions, then the consistency will reduce and the comparison will bear on two mixed aspects: the methods and the EMU, engineers’ assumptions. Impossible to tell which one does what, and, in the absence of blatant mistakes, very difficult to judge which one is correct.
Missing anchoring to reality
The correctness is particularly important in this endeavor. However, the fact that there was no request to anchor the results to reality, e.g., the world-wide behavior of tailings dams, is in our mind a significant deficiency. As you know we have been working in this field for many years, performed hundreds of risk assessments. We have recently shown (Oboni F., Oboni C., Optimizing mitigation of tailings dams portfolios, Tailings and Mine Waste 2022, Denver, Co ,November (video)) how properly performed risk assessments align with the world-wide portfolio of TSFs. We were able to achieve that by using the benchmarks we defined in 2013 and confirmed by Rana et al. in 2020.
Literature and ChatGPT scans
To see if our reasoning was supported by others, we delved in the literature, as introduced above.
Round robin exercises are rather common. For instance there was a tailings dam round robin in 2019 (Slope stability round robin (tailliq.com)). In order to avoid the mixing of uncertainties cited above the exercise was “rigidly specified”, simplified. This included a number of assumptions that all participants had to comply with. The goal was to ensure the focus of the exercise on the sought parameter. Other areas of civil engineering and analyses have also used round robin exercises:
Intrigued by the difference in the formulation of these exercises compared to the presently discussed one we asked to ChatGPT:
Are there systemic problems to geotechnical round robin exercises? The answer was:
Yes, there are some systemic problems associated with geotechnical round robin problems. These include the fact that the parameters of the problem may not be accurately represented. This leads to discrepancies in the results. Additionally, the problem may not be representative of actual field conditions, resulting in incorrect results. Finally, the software programs used to solve the problem may not be comparable in terms of accuracy or reliability.
Closing remarks and our final decision
Unfortunately, because of the “open questions” and the noted gaps, in our opinion, any interpretation will be flawed and not representative. Indeed, as designed, the round robin mixes methodological and assumption uncertainties in an inextricable way.
Furthermore, the exercise that looks at two specific failure modes will:
- neither enhance the understanding of the future behavior of the structure,
- nor show how the results from the participants compare to real life behavior.
Thus we formerly thought we should withdraw. However, having performed more than hundred quantitative risk assessments of tailings dams all over the world we decided that we should participate anyways, using our ORE2_Tailings™ methodology.
We are indeed curious to compare the results delivered by the ORE2_Tailings™ proven methodology, anchored to reality, as confirmed by the recent Rana et al. (2020) benchmarking exercise, to the other participants’ results, with all the caveat we have expressed above.
Tagged with: assessment, decision, risk, Risk Assessment, Risk Management
Category: ORE2_Tailings
Leave a Reply