Whenever a model appears to you as the only possible one, take this as a sign that you have neither understood the model nor the problem which it was intended to solve
Karl Popper, Philosopher (1902–1994)

If models are cheap, don’t choose. The more models the merrier. But we mustn’t waste our time on rubbish and there’s more to a good model than the accuracy of its predictions.

What is a good model?

At catastrophic economic cost, governments across the world have imposed various levels of lockdown on the basis of results of a handful of mathematical models. …

Models are nothing. Modelling is everything.
Sam Savage

This is a toy model of the spread of infection in a playground. It’s designed to explain features of serious models of the global epidemic and (next time) to show phenomena that these features predict that are not captured in simple models.

This is not a COVID model; I’m not making predictions. The simple setting makes it easier to explain what real world models are trying to capture and to demonstrate the how phenomena we see in the evolution of the pandemic numbers can be ascribed to relatively simple infection dynamics.

It is the model I used to produce the figures used in my “Pains of epidemiology” article, to which this blog acts as a slightly nerdy companion piece. …

A March lockdown was inevitable. Criticism of the decision since has been around weaknesses or inconsistencies in the models used to support that decision. These criticisms fall into two categories. The first is just silly: The models were wrong because the misery and death they predicted have not come to pass; the second deserves a little more attention: The models were poorly conditioned by the available data. Here I argue that the decision was sound, but that the presentation of the rationale was unfortunate, based as it was on unnecessarily complicated models.

Sloughing off the parachute of lockdown

The models on which the decision to go into lockdown were based were predicting misery and death in the event nothing was done. Complaining that these models were inaccurate because waaaay fewer people have died than predicted and because no-one who might be cured is being refused treatment is plain daft. Something was done. Most of the planet went collectively into the biggest medical intervention in the history of mankind with excruciating economic consequences. …

Cast your mind back to the early Spring, 2020. The COVID-19 virus has been identified and sequenced, the Wuhan outbreak is under control, but the virus has spread abroad. We have a handful of key parameters on hospitalization rates, mortality and contagiousness, inferred from Wuhan data, but these (both the data and the parameters) are rather uncertain.

The situation in Northern Italy is worsening daily and stories are emerging of hospitals refusing patients care because they are old or overweight and resources are prioritized for patients with better chances of survival. The first cases are appearing locally.

In this short series of articles, I will outline a how we can use mathematical modelling to frame key policy decisions. The focus here is on the deployment of models to make decisions and not so much on the models themselves. …

After publishing my probabilistic pitfalls article, my fourteen-year old son (who apparently follows me on LinkedIn; who knew?) came and told me I was the grumpy old man of risk and uncertainty. I had wittered on and on about what everyone was doing wrong, but made no effort to try and say what people should be doing instead. Fair comment, I thought. Here’s a go at putting that right.

1) Lookback

A little learning is a dangerous thing; drink deep, or taste not the Pierian spring: there shallow draughts intoxicate the brain, and drinking largely sobers us again.

Harry Collins is an expert on experts. At the core of his approach to expertise is what he calls a “periodic table of expertise”, which not only describes a hierarchy of levels of specialist knowledge, but also the ability of individuals possessing these levels of understanding to distinguish levels of expertise in others and how they do so.

The most basic form of knowledge in Collins’ hierarchy is beer-mat knowledge — an ability to recite facts without being able to do anything with them, except perhaps in a game of Trivial Pursuit. With popular understanding — such as you might expect to glean from a show on Discovery Channel — an elementary conception of meaning might allow for rudimentary inferences. …

Summary list

1. Lookback: No lookback or, what it sometimes worse, lookbacks that are so naive you’d have been better off without them. See this article.
2. P99 and P1. These have no place in any volumetric methodology whatsoever. Ever. See this article.
3. Weakest link and unwarranted independence: The practice of taking the smallest probability in a chain of dependent probabilities. An unnecessary oversimplification, as an account of dependencies between elements is simple and enormously fruitful.
4. The use of 50% probability in the absence of any information. The problem with this is not that it’s wrong, but with the idea there is no prior information. What? …

Discussions around extreme percentiles abound. The oil and gas exploration community expends enormous effort exhausting itself over appropriate values for the 99th and 1st percentiles of the distributions that describe the uncertainty in the size of oil fields.

I argue here that these discussions are at best distracting and at worst directly value-eroding.

The core of my criticism

My primary protestations are two fold

• These perilous percentiles are very little related to the important parameters of their distributions.
• Extreme percentiles are extremely difficult to assess with any kind of accuracy

Relevance

Uncertain variables are often approximated or represented by normal or log-normal distributions. Variables that can be decomposed into additive (or multiplicative) factors will always look roughly normal (or log normal) close to the mean. …