What is the validity of your risk forecast?
Jan 10th, 2018
When talking to potential customers we oftentimes hear “ What is the validity of your risk forecast? ” What they mean is actually “how precise” our numbers are.
That type of questions generally comes from individuals that confuse risk assessment with divinatory arts. They do not recognize that performing a risk assessment is completely different than reading a crystal ball.
Another common objection comes from people that claim that no one can evaluate probabilities of events because of complexities, etc. … then just add the excuse of the moment to better to justify their inaction.
We always explain that risk assessment, at least in our practice, is a science and does not rely on magic. We add that we prefer to be generally right rather than precisely wrong (roughly adapted quote from John Maynard Keynes).
When we get a first job with a new client, it is usually followed by numerous others, as the client sees advantages and benefits that come from driving at night with the lights-on, instead than blindfolded and lights-off.
Willingness to “do-the-best-with-what-you’ve-got” is empowering!
The credible scenario fallacy
Another difficulty we experience is when people balk at our wide range of events scenarios, trying to bias the studies toward what they believe are “credible” events in terms of probability p and consequences C (losses). The censoring they propose is double:
- reducing averages and
- narrowing ranges to say -10% to +15%.
That censoring is of course completely arbitrarily-selected and based on “experience”. That experience is unfortunately often short term and localized, thus useless or misleading.
For example, while assessing risk on mountainous mines’ access roads we consider a maximum scenario which includes full loss of the vehicle and freight or passengers. Of course we look at local topography and other parameters before introducing such an event at a given location.
That maximum scenario generally encounters fierce hostility from the client or the professionals in charge of the road operations. We have seen that reaction occurring both in North and South America. They consider that scenario as excessive, actually “incredible” based on their “experience and gut feeling”.
Yet these accidents occur and generate risks
Yet, that type of accidents exists and comes loaded with numerous fatalities and image/reputational consequences. One such bus accident just occurred in Peru last week with total loss of the vehicle and extremely high fatality count. Actually, in the aftermath of the accident, local authorities closed the road to buses, crippling “a whole region”. That accident occurred on a stretch included in a risk assessment we did a few years ago, where the client opposed our maximum scenario, claiming we were “unrealistic (or incredible)”.
One day later an event that could have ended with the same tragedy happened in the US of A in between Denver and Las Vegas. “A UHP spokesman said the crash could have been worse if the bus had rolled.”
Sadly we were right. It only took a few years to see it happen.
So, that type of accident is not only occurring, but it occurs in many countries around the world with a rate that places the generated risks in the societally intolerable realm. As a result, those accidents are unfortunately not “incredible” and show that “experience and gut feeling” are not good advisors when assessing risks.
With tools to understand which risks are intolerable we can advance the risk assessment’s conclusion triaging further the risks.
What is the validity of your risk forecast?
At Riskope we can answer this question without any hesitation.
In the years, we have confirmed that our forecasts are generally right rather than precisely wrong as they include:
- measures of uncertainties,
- careful selection of ranges going from minor to maximum scenarios,
- attentive analysis of possible domino effects (propagation, interdependencies),
- common cause failure analysis (CCF) and finally
- clear definition of what is “credible” and “incredible”.
It is very unfortunate that the proof of forecast validity generally requires a tragic event to develop. It is equally unfortunate that if nothing happens for a few years people think our estimates “were crazy”.
Contact us if you want to drive in the night with your lights on!
Tagged with: CCF, Common cause failure, Complexity, credible, maximum scenario, risk forecast, uncertainty, validity
Category: Risk analysis, Risk management