A modern Risk Manager and an Engineer have a heated discussion about FMEA/PIGs, common practices and what a rational and transparent quantitative risk assessment should encompass
Nov 14th, 2013
A modern Risk Manager and an Engineer have a heated discussion about FMEA/PIGs. The dialogue below is an excerpt of the course entitled “Risk-Based Management Of Tailings, Waste Rock, And Heap Leach Facilities” that Riskope and Jack Caldwell (Robertson Geoconsultants) are co-writing for Edumine.
Risk Manager and an Engineer have a heated discussion about FMEA/PIGs, and risk matrix 5×5
The dialogue format is used as a “literary device” to allow the reader to hear different “voices” from the industry and form his/her own opinion. We are not going to reveal the “real names” of the characters, so, for the moment being, consider one as being a strongly opinionated (mining) engineer and the other an equally strongly minded Risk Manager who has embraced state of the art methodologies, leaving aside once and forever obsolete, flawed common practices like FMEA/PIGs, and risk matrix 5×5. Both work for the same “controversial” critical infrastructure (or extractive, or energy production and distribution, or environmental rehabilitation, etc.) project.
Typical risk matrix 5×5
The Engineer breaks the silence: “I am really worried about the public reactions to the risk assessment we have presented. I do not understand why people are having these crazy reactions, like if we were lying to them all the time. Furthermore, I do not understand if the voices we hear are representative or if they belong to a small group of extremists”.
The Risk Manager does not wait a minute: “Well, may be people are turning sour on us because they have seen too many rosy scenarios with great promises. Those turned out to be long term disasters, fire spitting dragons. And as far as the representativeness of the voices we hear, there are known methods to deal with. There was a paper on the subject published at CIM 2013 (Oboni, F., Oboni, C., Zabolotniuk, S., Can We Stop Misrepresenting Reality to the Public?, CIM 2013, Toronto)”
Do not start chastising the whole profession
“Oh please, do not start chastising the whole profession now! You sound like a tree-hugger or a hypocritical politician”.
“Well, may be, but bear with me for a moment. Haven’t you read the decision of MVEIRB about risk assessment for the Giant Mine environmental rehabilitation? Wether you like it or not, that’s the voice of “the people” and they came out with a number of specs for risk assessment that push the envelope way beyond common practices. It seems that MVEIRB and the people they represent spoused important concepts about common practice limitations pushed forward by ISO 31010 and many authors (Cox et al. 2005, Cox 2008, Cresswell, Hubbard 2009, Chapman & Ward 2011, and repeatedly since a decade by the Oboni & Oboni team, for example: here).
Limitations of FMEA and 4×4 or 5×5 matrices
As limitations of FMEA ISO31000 quote:
- “One can only use them to identify single failure modes, not combinations of failure modes”. So they cannot grasp the complexity of some mining failure scenarios, dominoes effects.
- “Unless adequately controlled and focussed, the studies can be time consuming and costly”. Hence there is a very strong temptation to take shortcuts leading to simplistic approaches.
- “They can be difficult and tedious for complex multi-layered systems”. Just reinforcing the point above.
As limitations of Probability-Impact Graphs (PIG, aka Consequence/Probability Graph to avoid the funny acronym), ISO quote:
- Circumstances dictate the appropriate design of a matrix. Thus it may be difficult to have a common system applying across a range of circumstances relevant to an organization;
- it is difficult to define the scales unambiguously; use is very subjective and there tends to be significant variation between raters;
- aggregation of risks is impossible (i.e. one cannot define that a particular number of low risks or a low risk identified a particular number of times is equivalent to a medium risk);
- it is difficult to combine or compare the level of risk for different categories of consequences.
It is strange that people build many ERM (Entreprise Risk Management) systems using such an inappropriate framework?
I finally will remind you what ISO says about index based approaches (i.e. characterizing probability and/or consequences with a number (for example 1,2..5).
- in many situations using indeces, there is no fundamental model to define whether the individual scales for risk factors are linear, logarithmic or of some other form. No model defines factors combination. In these situations, the rating is inherently unreliable and validation against real data is particularly important.
- the results of a poorly validated the process (model) may be meaningless. The fact that the output is a numerical value for risk may be misinterpreted and misused, for example in subsequent cost/benefit analysis;
ISO code is too general
But wait, I see you coming! You are going to tell me that ISO is a general statement, it does not apply specifically to Engineers, Engineers are different, right? You may be right, and if I add that NASA (NASA. 2007. NASA Systems Engineering Handbook SP-2007-6105, National Aeronautics and Space Administration, Washington, D.C. Chapter 6.4 ) has a very clear opinion of PIGs/FMEA. That covers risk matrix 5×5 as well, of course. FAA shares the same opinion (Federal Aviation Administration (FAA), Aviation industry overview fiscal year 2001, FAA Office of Aviation Policy and Plans, Washington, DC, 2002. ). Now, you are also going to tell me that our project is not rocket-science and I should stop bothering you.”
“Yes, that’s exactly what I think and all our fellow engineers believe. You cannot use these few examples, no matter how reputable the organizations are, to go against a well established set of techniques, applied worldwide by reputable corporations”.
“OK then, gloves off, buddy! I will lay out a number of questions that I would ask you if I was challenging you in court. Let’s see how you would fare. Not to intimidate you, but sooner or later someone will go to court challenging the conclusions of a 4×4 or 5×5 matrix risk assessment. That includes risk matrix 5×5, of course. They will shred apart its users for misrepresentation and probably negligence. Now, if you answer convincingly to even only half of these questions, I will change my point of view and write a letter to ISO, NASA, FAA and a number of researchers around the world telling them they should drink more wine and stop writing silly comments; do we have a deal?”
“OK, it’s a deal, and of course if I fail, I will forever abandon common practices”
“Fine buddy: here is the list of questions. I give you a couple days to answer, but I am sure you will have no answer! (for a complete discussion on this subject).:
- So, on which basis did you decide that the probability of the event was “medium”, or whatever your PIG shows? More importantly, why did you neglect to use any of the methods, published from the ’80s on about (subjective, expert driven) approximations of probabilities? (you can see some surprising results on Factual and Foreseeable Reliability of Tailings Dams and Nuclear Reactors -a Societal Acceptability Perspective) that demonstrate how fallacious our brain can be in estimating “too precisely” probabilities).
- What is the basis for defining consequence (loss) classes in your PIG? …
- Which studies did you develop to define the various class limits of likelihood, and losses? On which basis did you select those limits?
- Why did you limit the highest class to -x- casualties and -y- millions? …
- So, did you use PIGs or a risk matrix 5×5 just because every one uses them? …
- Which criteria did you use to select the colors of your cells, which correspond to various levels of criticality? What criteria did you use to define those levels of criticality?
- There are tolerance criteria published since the mid ’60s. How come your color threshold does not match any known tolerance criteria, ….?
- Using “credible scenario” is a censoring decision. How come you felt entitled to censor your analysis? …and finally,
- Using “average p, C (loss)” is a biasing decision…how can you justify such a bias?
Needless to say, we at Riskope would be delighted to hear from you! And we are ready to accept the challenge as described above. Are you in?
Tagged with: Critical Infrastructures, distribution, energy production, environmental rehabilitation, extractive, FMEA, Heap Leach Facilities, Risk-Based Management, tailings, Waste Rock
Category: Consequences, Crisis management, Hazard, Optimum Risk Estimates, Probabilities, Probability Impact Graphs, Risk analysis, Risk management, Tolerance/Acceptability
[…] story: many common practice risk assessments (PIGs) are compiled during “open workshops” during which the facilitator asks the panel to describe […]