Unrealistic risk assessment describing a rosy scenario

Keystone pipeline has spilled substantially more oil, and more often, than indicated in the risk assessments the company provided to regulators. But why and how a common risk assessment becomes an unrealistic risk assessment describing a rosy scenario?

Indeed what regulators saw in the pre-construction project risk assessment was reportedly the following. “A spill of more than 50 barrels will occur “ not more than once every seven to 11 years over the entire length of the pipeline in the United States”.

Let’s note that “more than 50 barrels” does not declare an upper limit, but suggests a minor accident.

The predicted frequency would be 1/7 to 1/11 per year.

However, the reality is that since 2010, the Keystone pipeline leaked three times in the United States. That puts the frequency to 3/7 per year. That is roughly 3 to 4 times more than initially stated.

Furthermore, two of the spills leaked about 400 barrels, and one 5,000 barrels.

Reasons for unrealistic risk assessment describing a rosy scenario

Oftentimes we see unscrupulous risk assessors basing their estimates on best-case scenario assessments. They leave completely aside worst-case scenario and average-case scenarios.

That corresponds to biasing and censoring issues.

However, biasing and censoring issues also occur with the common practice of looking at worst-case but “credible” accidents. We quote the word “credible” as we never see it defined in this type of assessments. The fallacy lies in an idea. That is:

a) if one designs a system to withstand all the individual worst-case “credible” accidents, then

b) inherent protections cover “by definition” against any “credible” accident.

The fallacy has multiple aspects beyond not defining what is “credible”.

For example it does not cover for:

• uncertainties,
• inter-dependencies and
• common cause failures (CCF).

Those aspects, unfortunately, do characterize industrial and natural accidents.

How to avoid the pitfalls of unrealistic risk assessments

Here are a few guidelines:

• Risk assessment methodologies should never ignore a scenario.
• They should never “impose” a selection.
• The result of the analysis should focus on the right risks because the methodology is sound. Not because of a priori selection and framing bias.
• Use third party independent risk assessors. Keystone company (pipeline project developer) reportedly followed the standard practice of hiring their own expert to provide regulators the risk assessment. Accepting that is like trusting cancer research from tobacco companies. Just do not go that way!

In the eye of the public doubt of their motive will always remain. Indeed these are blatant cases of conflict of interest assessments and focusing bias.

Third party independent Risk assessment

This is the reason why 3^rd parties Risk assessment is paramount to bring credibility to a project.

We have seen some Technical Review Boards ask for them.

In the case of tailings dams risk. the UNEP (MINE TAILINGS STORAGE: SAFETY IS NO ACCIDENT) is unambiguous. The risk assessment should indeed be performed by an external third party. They state it as follows.

• Independent experts risk assessments would reduce risk of dams’ failures by providing independent, third party, unbiased expert oversight.
• Emphasis on the role and importance of monitoring and independent, third-party review throughout life cycle is of critical importance.
• Independent technical reviews should conduct and publish their reports through the life cycle of the infrastructure.

Framing probabilities

This is why when we perform risk analysis we spend a lot of time framing probabilities of mishaps. We like to be generally right rather than precisely wrong.

We also compare our probabilities estimates to benchmarks and oftentimes carry out bench-marking across industry. This enhances credibility and transparency. An example of cross industry risk tolerance bench-marking applied to hydrocarbons and other hazmat rail transport is available in this study .

Tagged with: biased risk managment, credibility, pipeline, risk analysis, Third party independent Risk assessment

Category: Consequences, Probabilities, Risk analysis, Risk management