--- title: "Foresight for AGI Safety Strategy" description: "Governance people should be writing the policy document and building the demo for each plausible scenario now, rather than starting the day a minister gives them three months." published: 2022-12-05 tags: ["AGI strategy and foresight"] importance: 8 confidence: "possible" docStatus: "finished" cover: "https://jacquesthibodeau.com/content/images/2022/12/cone-of-plausibility.png" coverAlt: "The cone of plausibility: a cone opening from “Now” into the future, its nested shells labelled from projected at the centre out through probable, plausible and possible to preposterous at the rim." audio: "https://pub-4ee2f71bc29541a7a6e8d9694f0a1b21.r2.dev/638e25343de6b56ce4c0c880/audio.mp3" crosspost: lesswrong: "https://www.lesswrong.com/posts/GbXAeq6smRzmYRSQg/foresight-for-agi-safety-strategy-mitigating-risks-and" figures: - src: "https://jacquesthibodeau.com/content/images/external/b38cc2a2-e7609a4eaeba81e8b615f3b75415f8a9082d114034496dd8.png" alt: "A chart headed “Forecasting for the short-term, Foresight for the longer-term”: predictability falls and uncertainty rises as the years pass, with forecasting shaded over the first year and foresight over the five to fifteen year band." width: 627 height: 575 - src: "https://jacquesthibodeau.com/content/images/external/f835d5a6-woY_5vpLcNZ7Y-UdUSBZeKNTRZdWG6rmk1aLLzeC.png" alt: "The cone of plausibility: a cone opening from “Now” into the future, its nested shells labelled from projected at the centre out through probable, plausible and possible to preposterous at the rim." width: 1055 height: 740 - src: "https://jacquesthibodeau.com/content/images/external/16995c48-yRLj6VA5PBEM1EYphjFjFaf8pelU-GY0NW79t3_h.png" alt: "The seven steps of the Horizons Foresight Method as a stack of coloured bars — framing, assumptions, scanning, system mapping, change drivers, scenarios, results — each with a one-line description beside it." width: 720 height: 405 author: "Jacques Thibodeau" canonical: "https://jacquesthibodeau.com/foresight-for-agi-safety-strategy/" --- For discussion: [Link](https://www.lesswrong.com/posts/GbXAeq6smRzmYRSQg/foresight-for-agi-safety-strategy) to LessWrong post. [Link](https://forum.effectivealtruism.org/posts/zyvWaHYw6y8rkSQMG/foresight-for-agi-safety-strategy) to EA Forum post. This post is about I think why we should be using strategic foresight for AGI governance work. I am writing this without having read all the AGI governance / strategy reports and reading materials. I was planning on putting more effort into this post, but this is a better-than-nothing-at-all post. Worst-case, this might end up being a signal for someone in the comments to point me to work that already exists. Best-case, this kicks off some work that I think could be quite valuable. **TLDR**: I feel like there should be more AGI governance people doing strategic foresight to plan for several scenarios, identify opportunities, and then **create a plan for acting in those scenarios by creating policy documents and demos that take advantage of each scenario**. For example, if artists are concerned about their livelihood, you’d get in there right away and have a policy document they can get behind and then you use that opportunity to gain trust, build political capital, and potentially add alignment-relevant stuff in law or increase investments in safety. In this post, I will cover the following: - What is **Strategic Foresight**? - Strategic Foresight applied to AGI risk - *Why* should we be doing Strategic Foresight for AGI risk? - *How* *can we adapt* known strategic foresight methods for the AGI risk? - **Call to action** Before I start, I want to say that I know there's a lot of effort being put into the AGI governance space. Part of my goal here is to start a conversation as someone with both technical safety and government experience. # What is strategic foresight? Foresight can be used to explore plausible futures and identify the potential challenges that may emerge. Foresight helps us better understand the system that we are dealing with, how the system could evolve, and the potential surprises that may arise. Governments use foresight to help policy development and decision-making. Looking to the future helps us prepare for tomorrow’s problems and not just react to yesterday’s problems. **Foresight is not forecasting.** When I first started reading about AI forecasting, I was a little confused about the terminology. In the EA / AI Risk space, most people seem to use the word “forecasting” even when talking about year 2040+. People that I talked to outside of EA seemed to claim that “forecasting” only worked for short-term predictions and you were potentially being overly confident about things evolve if you try to predict further than 1 year into the future. Forecasting is usually seen as a form of *prediction*, while foresight focuses on looking at the cone of plausible outcomes (so that, for example, an organization can be as robust as possible across plausible scenarios). With foresight, you basically just assume to case you won’t be able to predict exact outcomes, and so you try to be as robust as possible. **Foresight does not try to predict the future as “forecasting” does.** In some spaces (though perhaps not in the AI Risk space), forecasting will try to take past data and predict what will happen in the future. This can be problematic because systems can change, and the assumptions hidden in the data may no longer hold up. Forecasters often look at relevant trends and past data, which can be quite good at predicting the near expected future. However, forecasters may overlook the weak signals that can become change drivers and lead to a massive change in the system as they try to predict further into the future. Foresight pays special attention to these weak signals that may eventually lead to disruptive change. However, it occurred to me that people within EA / AI Risk are instead doing some form of forecasting and foresight. That said, I think it’s useful thinking in the Strategic Foresight framing since our goal is not to predict when exactly when AGI will happen, the goal is to be safe at any point it does arrive and making sure we take advantage of any golden opportunities and mitigate any risks. So, for the purpose of this post, I’ll keep talking about the difference between “(near-term) forecasting” and foresight. ![A chart headed “Forecasting for the short-term, Foresight for the longer-term”: predictability falls and uncertainty rises as the years pass, with forecasting shaded over the first year and foresight over the five to fifteen year band.](https://jacquesthibodeau.com/content/images/external/b38cc2a2-e7609a4eaeba81e8b615f3b75415f8a9082d114034496dd8.png) People can forecast a couple of days to decades in the future, but I’ve heard the claim that it’s only really “effective” within a 1-year timeframe. This is potentially blasphemy to some of the forecasters in the community, so I'd like to hear everyone’s thoughts on this. I expect that forecasting, at the very least, will help us learn a lot about how AGI might come about. Foresight, however, works much better than forecasting in a 5- to 15- year timeframe (it’s possible to keep using foresight after 15 years, though its effectiveness may go down every new year into the future, especially for our unpredictable AGI future). Instead of trying to predict one specific outcome, foresight is trying to uncover scenarios that fit into a cone of plausibility and then identify the opportunities that can be taken advantage of in each scenario. ![The cone of plausibility: a cone opening from “Now” into the future, its nested shells labelled from projected at the centre out through probable, plausible and possible to preposterous at the rim.](https://jacquesthibodeau.com/content/images/external/f835d5a6-woY_5vpLcNZ7Y-UdUSBZeKNTRZdWG6rmk1aLLzeC.png) Foresight tries to overcome some of the difficulties that come with forecasting by testing whether the assumptions in which the past data hold are still correct in the future. This is of the utmost importance when things are changing in fundamental ways. Foresight is a method that allows us to look at the current system and see different ways it may evolve and then plan for the various outcomes proactively. The different paths a system could take all have varying degrees of complexity. The more complex a problem, the harder it is to predict what might happen. This is one reason why foresight is valuable; it helps you think through all of these different and complex paths and create an action plan on how you might resolve this situation if it happens. This is quite different from coming up with an action plan when there is urgency because often, the project will be rushed. Imagine a situation where a minister asks for the development of a policy on image generation and gives policymakers 3 months to come up with a policy document. It’s quite likely that 3 months is not enough time to develop an exceptional policy. Now, imagine the same scenario, but 4 years ago, you used foresight to think through the possibility of how image generation policy can include (useful) AGI safety regulation, created some “scary” demos, and you have an action plan on the steps to take. That is one of the things I'm hoping to get out of this type of work. ### Unfortunately, foresight can be poorly implemented When I was looking into this stuff a few years ago, I got the impression that a lot of foresight work is done poorly. There wasn't enough effort in making things rigorous. Fortunately, some organizations have created rigorous foresight methods. The top contenders I came across were [Policy Horizons Canada](https://horizons.gc.ca/en/home/) within the Canadian Federal Government and the [Centre for Strategic Futures](https://forum.effectivealtruism.org/posts/8Lcw8xdMfjNusbpbd/foresight-for-governments-singapore-s-long-termist-policy) within the Singaporean Government. Apparently, Europe [likes](https://www.gov.uk/government/collections/foresight-projects) foresight [work](https://www.oecd.org/strategic-foresight/ourwork/Strategic%20Foresight%20for%20Better%20Policies.pdf) as well. Policy Horizons Canada explores plausible futures and identifies the challenges and opportunities that may emerge to create policies that are robust across a range of plausible futures. Please look at their [training material](https://horizons.gc.ca/en/resources/) if you want to know more about their method. There are two main things I’d like to highlight about Horizons’ Foresight Method: - Horizons’ Foresight Method is designed to tackle **complex public policy problems** rigorously and systemically. - Horizons consider the mental models of different stakeholders essential to take into consideration when using foresight to **inform policy development**. Horizons has avoided the trap of using too much intuition to throw foresight methods at a problem and have created a rigorous, systemic, and participatory foresight method that focuses on exploring the future of a policy issue. That said, I still think their approach could still be improved and tailored to our specific use case. ![The seven steps of the Horizons Foresight Method as a stack of coloured bars — framing, assumptions, scanning, system mapping, change drivers, scenarios, results — each with a one-line description beside it.](https://jacquesthibodeau.com/content/images/external/16995c48-yRLj6VA5PBEM1EYphjFjFaf8pelU-GY0NW79t3_h.png) # Strategic Foresight applied to AGI risk ## Why should we be doing Strategic Foresight for AGI risk? I think Strategic Foresight work would allow us to better identify opportunities for getting governments and organizations on board. The way I see it is that if we have a set of policy documents, demos or plans that we have prepared well in advance for several plausible scenarios, we are much better placed to move fast when the time comes. Secondly, there was a lot of insight gained from the work done on reports like "[The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation](https://maliciousaireport.com/)." I believe a big workshop like this with people from different backgrounds and perspectives would be a great opportunity to identify opportunities we might not have identified in our respective silos. This is a great thing about some foresight methods: putting a bunch of people with different backgrounds in a room and approaching the problem from every angle. This would also serve as an opportunity to connect with people working on different parts of the problem. Finally, the big value we get out of doing this type of work is to be prepared for many scenarios so that our entire approach is as robust as possible. It’s worth thinking about doing things that are robust across scenarios, like building trust with governments and organizations (which a lot of people in governance already do). What we are facing is a [wicked problem](https://www.cold-takes.com/the-wicked-problem-experience/). There are many moving parts, and things are constantly changing and moving fast. ## How can we adapt known strategic foresight methods for the AGI risk? Unfortunately, I don't have the answer to the above question. However, here's some initial rough thoughts: - We should have a solid set of predictions of what might happen in the coming years. For example, when will model x come into play? When will company z make its move into productizing models? **Which field of work will be greatly impacted by AI and can work with those people to communicate with policymakers?** - What are all the **levers** we can pull? - What are the **different scenarios we can prepare for**? What are the different ways we can prepare for each scenario? What are some **early signs** that tell us we are moving toward a particular scenario? - In terms of convincing policymakers in countries that matter, can we gain credibility through the [Brussels Effect](https://www.governance.ai/research-paper/brussels-effect-ai)? - Can we identify high-impact opportunities where **technical and governance researchers can collaborate**? - Where are the weak signals that we can use to identify opportunities for action? - Which opportunities might allow us to **extend timelines**? - What are the opportunities in terms of reducing the risk of AGI used by a **malicious actor**? - We should be thinking of using strategies that seem **robust** across scenarios. For example, building trust / political capital. Maybe I’m unaware of the work that is being done, but I feel like there is a lack of work done to identify near-term opportunities where we can move the needle on things in the short-term. I would like more specific plans we can quickly take action on when a given scenario arises and the pros and cons for which actions to take. Another issue we might face if we don’t do this thinking in advance is the risks that come with taking specific actions. We do not want to be in a specific scenario, and we are just winging it and figuring out which actions to take at the moment. This might lead to situations where alignment people completely lose the trust of different stakeholders. Which leads to the FTX situation (ok, bear with me). Foresight is not only about identifying opportunities, but it is also about identifying risks. There’s been a lot of discussion about funding this past month and if we should have put some much faith into a large funding source. What can we do to prevent this from happening again? Well, I think we would have been much more likely to expect the FTX scenario had we done the work to identify such risks. Not only that but also knowing how to act in such a scenario so that we minimize the negative impact. I think it’s worth mentioning that Rethink Priorities did do [work like this](https://forum.effectivealtruism.org/posts/rxojcFfpN88YNwGop/rethink-priorities-leadership-statement-on-the-ftx-situation) and identified FTX as a big risk. I applaud them for this, but let’s also make sure we take it even further than they did going forward. Here’s what they had to say about it:
Prior to the news breaking this month, we already had procedures in place intended to mitigate potential financial risks from relying on FTX or other cryptocurrency donors. Internally, we've always had a practice of treating pledged or anticipated cryptocurrency donations as less reliable than other types of donations for fundraising forecasting purposes, simply due to volatility in that sector.
In the case of Rethink, they talk about “crisis management”, “internal simulations” and “risk assessments/management”, but I think Strategic Foresight can feed into that. # Call to action I have *some* experience in this sort of work (workshops and strategic foresight), but I am currently spending most of my time working on technical alignment. That said, I'd like to extend an invitation to anyone interested in doing this sort of work to reach out to me. I'd love to help out in any way I can. I'll part with this; we should be tackling AGI risk with a level of desperation and urgency that allows us to access parts of our brain that create potentially unusual but effective strategies (while [avoiding](https://www.youtube.com/watch?v=Oz4G9zrlAGs&t=1147s) any [naive utilitarian traps](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ)). And I want to make one point absolutely clear: if we do use approaches like Strategic Foresight, we should be doing it in a way that is "[actually optimizing for solving the problem](https://www.lesswrong.com/posts/smBkyR2GkzMtrKuwK/the-first-filter)." ## Sources Every external link in this piece that has a captured card, with what that page said when it was captured. The quoted lines below are not the author of this piece writing: they are the linked page describing itself, recorded by `bun run link-cards` on the date given, and kept so that a reader still has them if the original moves or goes away. - **Foresight for AGI Safety Strategy: Mitigating Risks and Identifying Golden Opportunities** — jacquesthibs, lesswrong.com, 2022-12-05 Captured 2026-08-28. > This post is about why I think we should use strategic foresight for AGI governance work. I am writing this without having read all the AGI governance / strategy reports and reading materials. I was planning on putting more effort into this post, but this is a better-than-nothing-at-all post. Worst-case, this might… - **Foresight for AGI Safety Strategy: Mitigating Risks and Identifying Golden Opportunities** — jacquesthibs, forum.effectivealtruism.org Captured 2026-08-28. > This post is about why I think we should use strategic foresight for AGI governance work. I am writing this without having read all the AGI governanc… - **Policy Horizons home** — horizons.gc.ca Captured 2026-08-28. - **Foresight projects** — gov.uk Captured 2026-08-28. > Foresight projects give evidence to policymakers to help them create policies that are more resilient to the future. - **Policy Horizons resources** — horizons.gc.ca Captured 2026-08-28. - **The Wicked Problem Experience** — cold-takes.com, 2022-03-02 Captured 2026-08-28. > A day in the life of trying to complete a self-assigned project with no clear spec or goal. - **The Brussels Effect and Artificial Intelligence | GovAI** — governance.ai Captured 2026-08-28. > In this report, we ask whether the EU’s upcoming regulation for AI will diffuse globally, producing a so-called “Brussels Effect”. - **Rethink Priorities’ Leadership Statement on the FTX situation** — abrahamrowe, forum.effectivealtruism.org Captured 2026-08-28. > From the Executive Team and Board of Directors of Rethink Priorities (Peter Wildeford, Marcus Davis, Abraham Rowe, Kieran Greig, David Moss, Ozzie Go… - **Connor Leahy–EleutherAI, Conjecture** — The Inside View, youtube.com Captured 2026-08-28. - **MIRI announces new "Death With Dignity" strategy** — Eliezer Yudkowsky, lesswrong.com, 2022-04-02 Captured 2026-08-28. > tl;dr: It's obvious at this point that humanity isn't going to solve the alignment problem, or even try very hard, or even go out with much of a fight. Since survival is unattainable, we should shift the focus of our efforts to helping humanity die with with slightly more dignity.… - **The First Filter** — adamShimi, lesswrong.com, 2022-11-26 Captured 2026-08-28. > Consistently optimizing for solving alignment (or any other difficult problem) is incredibly hard. The first and most obvious obstacle is that you need to actually care about alignment and feel responsible for solving it. You cannot just ignore it or pass the buck; you need to aim for it. If you care, you now have to… ## Terms used The author's own definitions for the glossary terms this piece uses. These are his words, not a standard reference. - **AGI** — Artificial general intelligence: a system with human level cognitive ability across domains rather than in one narrow task. - **FTX** — The cryptocurrency exchange that collapsed in November 2022, taking a large amount of effective altruism funding with it.