Probabilistic Foresight
What it is probabilistic foresight and how can we combine probabilistic forecasting and strategic foresight and have 80% of the benefits with 20% of the resources?
Strategic foresight is in spotlight more and more lately. No doubt in part driven by the plague since March 2020 and people being more attuned to the deep uncertainty in human affairs as a result of it. The European Commission declared that strategic foresight will guide all aspects of EU decision-making now:
There is a proliferation of things related with foresight, scenario planning/thinking, probabilistic forecasting, and other ways and means of coming to terms with uncertainty. It’s a good thing, by and large, although there is a lot of buzzword spouting going on without much substance, rigor, or track-record too. The lack of a long-term track-record and rigorous study of whether they work in some meaningful way is also an issue, particularly for methods that claim to provide insights 10-15-30 years ahead such as future studies and strategic foresight.
Here is Luke Muehlhauser from Open Philanthropy has to say about that in a great post from about a year ago:
How accurate do long-range (≥10yr) forecasts tend to be, and how much should we rely on them?
As an initial exploration of this question, I sought to study the track record of long-range forecasting exercises from the past. Unfortunately, my key finding so far is that it is difficult to learn much of value from those exercises…
There are exceptions to this and Good Judgment is one of those exceptions given the centrality of keeping score in GJ’s methodology. However their focus is on short-term (up to 18-24 months) probabilistic forecasting and it works relatively well with well-defined, carefully thought-out questions with clear resolution criteria. But as Michael Story, a Superforecaster and a former Manager with GJ, and the co-founder of Maby, points out here, other real-life forecasting tends to be a bit messier:
So this dissatisfaction with strategic foresight and probabilistic forecasting methodologies led me to experiment with a hybrid method, a probabilistic foresight methodology if you will. Perhaps I should call it Hindsight 20/20 Method™️, since such “branding” seems to be in vogue. Anyways, let’s stick with probabilistic foresight for now.
Probabilistic Foresight v. 1.0
So what is this probabilistic foresight method? I organized a foresight workshop on Turkey - NATO relations in 2025, back in early March, right before the plague hit hard. The participants were Canadian policymakers of varying seniority (from relatively junior analysts to senior civil servants on relevant portfolios) from Department of National Defence, Canadian Armed Forces, and Global Affairs Canada as well as scholars from Canada, the US, Turkey, and Europe. Overall 20 people took part in this half day workshop.
You can read the working paper that is based on it which was out couple weeks if you are interested in the topic of Turkey and NATO relations. Or if you don’t have time have a look at the policy brief based on it. But here is the bit about the method we used:
Before the workshop, the participants were asked to anonymously fill out a brief survey one week prior to the date of workshop. The survey asked the participants to identify at least one, but no more than three events/developments for each of the following three categories: existing trends, weak signals, and potential wildcards. I have then collated the responses and selected a number of items to be discussed at the workshop. The number of items were limited due to time constraints and I have selected those trends, weak signals, etc. that have been identified by the majority of the respondents in similar ways.
On the day of the workshop, the participants were provided with the first item on the agenda (e.g. Trend #1) and were asked to anonymously estimate the likelihood of that trend continuing using an online app. They were given the option to choose from one of the five options available:
A. Almost certainly not (10% probability),
B. Probably not (30% probability),
C. Chances about even (50% probability),
D. Probable (70% probability),
E. Almost certain (90% probability)
Although asking participants to give precise probability estimates would have produced a more granular picture, probability bins using common verbalizations of those estimates in the intelligence community are preferred due to the heterogeneity of the group and their prior experience with probabilistic estimation or the Delphi method as well as time and resource constraints for providing such training prior to the workshop.
After the participants made their choices, the results were shown for everyone as a bar graph on the screen. Unlike the classical Delphi method, the experts were not asked to write down their rationale for the estimates. Instead, once the results were shown, they were invited to articulate why they choose a particular option to the group for a few minutes, starting with those who are on the tails of the distribution (i.e. A and E). This was generally sufficient to get a conversation started among the experts in the workshop. After the discussion, the experts were invited to submit new answers. The results were shown again and the facilitator summarized the changes from the first estimates. Those who revised their estimations were given a few minutes to explain why they did so. The group then moved on to the next item (e.g. Trend #2).
For the present trends, the experts were asked to estimate the probability of the trend continuing in the next five years. For the weak signals, they were asked to estimate the probability of that signal strengthening in the next five years. For the wildcards, they were asked to estimate the probability of that event (or a very similar event) happening in the next five years.
In the second part of the workshop, the discussion switched to the impact of the aforementioned events/developments on Canadian defence and security. The participants were asked to rate the impact of each discussed trend, weak signal, and wild card on Canadian security and defence interests on a five-point scale (negative, somewhat negative, neutral, somewhat positive, positive) regardless of how likely they thought the event/development under discussion would be.
It worked out reasonably well, given the constraints of time and resources as well as the fact this was the first time doing such an exercise for almost all participants. We managed to get good insights under 3 hours. Anything longer than that, I wouldn’t be able to get the type of senior people like I did. So most of the choices above reflected trade-offs related to time, depth of discussion, resources/technology etc.
I used the discussions in the workshop to create 3 broad scenarios for the future of Turkey - NATO relations in the above mentioned paper. I will write about scenarios and why scenario thinking but not necessarily scenario planning is a useful tool in dealing with uncertainty. But for now, let’s focus on the workshop methodology.
I believe the method itself, combining probabilistic thinking with a horizon scanning/foresight approach to what questions to ask and then spending time to think about the impact/consequences of various elements (trends, weak signals, wildcards etc.) in some depth is worth developing more. It is an efficient and relatively rapid way of gaining foresight and you can have 80% of the benefit with 20% of the resources/effort. It is a more rigorous way and you can keep score/record overtime to see whether it actually is worth doing.
Probabilistic Foresight v. 1.5 (Still not calling it Hindsight 20/20 Method…yet)
Here are some thoughts on how I will revise/tweak it for version 1.5. This is a non-exhaustive list without a particular order:
Ensuring more attention is paid to the pre-workshop survey (on identifying drivers, trends, weak signals, and wild cards) by the participants. This prep stage requires back and forth with either the sponsors of the workshop or the participants to ensure that the wording is clear and that they reflect the “collective wisdom”, if you will, of the attendees. Unfortunately it is not always possible to do multiple rounds like that, especially when people who are pretty busy volunteer their time as in the case of this NATO workshop. But doing so definitely would increase the quality of the outcome for everyone involved.
Using precise probability estimates rather than bins. This is not a necessity but I find that asking people to be more precise in their estimates makes them focus better on the relative probabilities of different outcomes. This doesn't lead to false precision as some critiques argue, as Jeffrey A. Friedman convincingly demonstrates in his excellent book War and Chance.
Measuring uncertainty surrounding the estimates. I wanted to do that in this workshop too but just didn’t have the time. Asking people how confident they are in their estimates and comparing it to their updated estimates is a good way of understanding how much you can reduce uncertainty on that particular topic.
Using a combination of written, short commentary (~100 words) and oral argumentation in discussions. Providing multiple avenues to make arguments is more robust to different organizational cultures, personality types etc.
More granular approach to impact/consequences, including uncertainty bands and thresholds for “unacceptable” costs/consequences. This is I think would be key to linking probabilistic foresight to decisionmaking and would provide real value by sharpening the focus on what really matters, what you cannot afford to ignore etc.
A brief self-guided/asynchronous training (~1 hour) on the basics of probabilistic forecasting (base rates, scope sensitivity, Fermi-izing etc) and strategic foresight (horizon scanning, weak signals, wild cards, scenarios etc.) prior to the workshop. This would improve the quality of the discussion at the workshop significantly as attendees would be somewhat familiar with the concepts/tools.
Incorporating discussion of scenarios in a follow-up workshop (optional).
Doing pre-mortems as a part of the workshop at the end (optional). I am a fan of this simple yet effective technique and want to include it whenever possible.
Use a dedicated software like GJ’s Delphineo or Maby’s app. Nice to have but not strictly necessary.
I think this method is worth developing more and I will continue to improve and tweak going forward as I see what works and what doesn’t work in different settings and with a variety of topics. If you have any feedback or suggestions please let me know either via email or in the comments.
Also if you are or someone you know is interested in such a probabilistic foresight workshop, please do reach out to me (
balkan@devlen.com
) and we’ll talk. It does not have to be about geopolitics or international affairs and can easily be done online. Actually it might even be better that way, since the participants could be anywhere.
As always please share it with friends, colleagues, family and others and please let me know what you think in the comments below.