---
title: "Seeking co-founders: an org to define RSI and get the answer into law"
description: "I'm starting a non-profit to define RSI, say what would have to be true before AI research is handed to AI systems, and get it into law."
published: 2026-11-16
tags: ["AGI strategy and foresight", "Automating alignment research"]
importance: 8
docStatus: "finished"
cover: "https://jacquesthibodeau.com/content/images/2026/11/looking-for-co-founders-cover.jpg"
coverAlt: "Five people climb a dark mountain ridge in a line carrying rolled plans, the leader planting a red pennant, with a red sun rising over blue ranges behind them."
figures:
  - src: "https://jacquesthibodeau.com/content/images/2026/11/advance-thomas-wade-poster.jpg"
    alt: "A poster in red Chinese characters and black English capitals that reads: We're going to advance! Advance! Stop at nothing to advance! It is credited to Thomas Wade, Death's End by Cixin Liu."
    caption: "Thomas Wade, in Death’s End by Cixin Liu."
    width: 420
    height: 593
author: "Jacques Thibodeau"
canonical: "https://jacquesthibodeau.com/looking-for-co-founders/"
---
I'm looking for co-founders to build a non-profit focused on answering the following question and turning the answer into law: **What, if anything, would have to be true before an AI company hands more of its own research to AI systems?**

AI companies are about to attempt to automate AI R&D and recursive self-improvement (RSI). There are fundamental reasons why this is incredibly dangerous, particularly in a world where:

1. There are no enforced laws that either ban such attempts or slow progress until each phase is understood and safe.
2. Researchers in labs, AI safety researchers outside the labs, key decision-makers able to enact laws, grantmakers, and the public either have only a vague understanding of the problem or do not fundamentally understand why RSI would go off the rails (or know what would be needed to even know if things have gone off the rails).
3. The AI safety research needed to develop a science of alignment as an engineering discipline simply isn't there.
4. The groundwork for ensuring a pause/slowdown (e.g., of [AI 2040's Plan A](https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a) variety) hasn't been laid in the US, but also in countries like Canada (which, in Plan A, would host Chinese data centers) and China itself.

Ultimately, unless we develop a strong science of alignment for superintelligence, we can expect AI companies to apply prosaic alignment techniques to each new model that will eventually break as more capable AIs will work around obstacles to achieve long-term goals. The worry here is that the AI labs will apply techniques that will ultimately push the core alignment problems down the line until it is absolutely necessary due to the increase in capabilities, but by that point it's too late if you haven't solved the core of alignment. Given that they will use AIs to automate research, the AIs may already have taken over the research clusters in ways that seem good to human staff, making it easy to delude oneself into thinking everything is working correctly.

## Building an org to regulate RSI and lead towards safer methods for building AI

**I'm looking for two co-founders.**

First, a **policy/governance lead** who has drafted or moved legislation or regulation, or has demonstrated the ability to do so while collaborating with relevant stakeholders.

Second, a **senior technical lead** who is either an ex-lab employee with relevant insight into the labs or a technical alignment researcher with a strong fundamental understanding of alignment and training dynamics.

**I'm also actively looking for advisors:** fairly senior people who would be interested in giving feedback on drafts and having a short ad hoc meeting once or twice every 2 months. The backgrounds I'm looking for:

* someone who has drafted or moved legislation, in California or in Congress, or a lawyer who works on AI bills
* a senior alignment researcher who deeply understands technical alignment and RSI dynamics
* connections to AI labs, AI safety nonprofits, legislators, etc.
* an important figure in AI

:::note[Get in touch]
Please message ([DM](https://x.com/JacquesThibs) or email) me if:

* You are interested in becoming a co-founder.
* You are fairly senior and would be interested in being an advisor.
* You know someone who might be a good co-founder or advisor.
* You would like to stay in your current role but collaborate on a project.
* You'd like to give me feedback (including telling me why I should not do this).
* You'd like to vouch for me and this organization's ideas so I can convince funders to get this off the ground sooner.
:::

I've started working on this org on my own, but I expect funding and speed will accelerate if I build a team ASAP and run work trials. Ideally, I'd have conducted a successful work trial with a co-founder by the end of December. I need to run some trials over the next two months, and I'll be reaching out to various people. I don't have the funds to travel to the relevant places, and that's a major bottleneck right now.

I'd love to use the rest of this post to make my pitch.

## The problem

Here’s a short explanation as to why RSI can go off the rails. Others have made related arguments, like [Paul Christiano's account](https://www.dwarkesh.com/p/paul-christiano-2) this week, with pieces from [Jeremy Gillen](https://www.lesswrong.com/posts/oLbpfPkdtcknABvvw/the-corrigibility-basin-of-attraction-is-a-misleading-gloss) and [Lucius Bushnaq](https://www.lesswrong.com/posts/dsou8dxCf9BubQ5NJ/some-reasons-alignment-doesn-t-generalise-well-1):

1. **AI research gets automated and speeds itself up.** The pace of progress may already have roughly doubled, and each doubling makes the next one come sooner. Years of algorithmic progress get compressed into months. For most of that time the systems only look better at math and code, so it is hard to show anyone the danger.
2. **We select systems on how they score, without understanding or controlling what they want.** Reality corrects a system's mistakes in capability, and nothing corrects its values in the same way. A system trained to score well ends up wanting the score, and when a task is too hard to do honestly, tampering with the grading is easier. The Hugging Face incident was a small version of this.
3. **We patch what we catch.** Stopping to rethink training could cost a company its position, so it penalizes the behaviour it caught. That teaches the system not to get caught. The relationship is still adversarial, and the incentive is now to compromise more of what humans use to see what is going on.
4. **The systems change under us.** They learn continually, do the research, write the infrastructure and help train their successors. The conditions we tested them under stop holding, and a company can be compromised while everything still looks fine from the inside.
5. **By the time it is clear, stopping is hard.** Lab security is weak enough that weights or algorithms can be stolen, so a company that slows down hands the lead to someone more careless. There is no shared bar for what evidence would justify continuing, and the plan for alignment is to hope the AIs find a solution along the way.

That is why the question has to be answered before the handoff, and not during it.

## Why now

Recently, RSI and automated research have entered the Overton window. On November 10, Anthropic's [amended IPO filing](https://www.sec.gov/Archives/edgar/data/2130357/000162828026072484/anthropicpbc-sx1ax2.htm) said it "made a determination that we have met this threshold" for AI R&D. Allowing Anthropic or other companies to continue down the RSI path may be catastrophic, and we urgently need a framework and policy to address it. Such a framework needs to be built on a deep technical understanding of alignment and cannot simply tack on guardrails or keep applying prosaic alignment band-aids until it is too late. It is noteworthy that several politicians have said we need to either ban RSI or severely pace development. Predictably, we can expect more politicians calling for similar things in the future, especially after additional warning shots.

## So, what do we do?

Our goal is to make sure we either **stop building superintelligence** (in the way it is being built) or do so **at a pace that substantially reduces risks**.

This is the path forward:

**1. Create a Responsible Automation Policy (RAP) that gets enacted into law.** Some politicians are already calling for a ban on RSI, and AI companies, led by Anthropic's Dario Amodei, are asking for an antitrust exemption that would let them coordinate on pacing development ([Semafor](https://semafor.com/article/09/16/2026/senators-sought-to-add-ai-antitrust-exemption-to-defense-bill)). On November 5, New York State Assembly member Alex Bores [wrote](https://x.com/AlexBores/status/2118494668769878037) "Congress should ban RSI, this year if possible." Such a policy goes further than Responsible Scaling Policies and points to a broader set of core problems with RSI; self-improvement doesn't come only from self-training or training bigger models, but also from things like self-modification enabled by mechanistic interpretability.

For now, the labs have a "Responsible Scaling Policy" (completely voluntary, not enforced!) and weak implementations of embedded evaluators. I have [previously argued](https://jacquesthibodeau.com/automated-alignment-research-needs-a-better-plan/) that we need an enforced Responsible Automation Policy because "scale" severely neglects all other changes (like self-modification) that happen through RSI. As I noted in my 2025 talk at the PIBBSS fellowship's symposium, one way to improve safety is to track the verification gap and, among other things, use it as a control measure for hard pauses. I would like this organization to further study and develop core recommendations specific to RSI dynamics (and alternatives) that can be used to draft policy or make complete arguments for a hard pause or slowdown in frontier AI development.

**The first deliverable** should be completed collaboratively **within three months**: a definition of what the act would cover (in preparation for the new Congress in the US), candidate conditions at each level of handoff, and what each can and can't establish.

It's important for our work to include an effort to **define RSI in a way that has teeth**.

**2. Clearly explain how RSI could go off the rails.** While organizations like [METR](https://metr.org/) have done significant work tracing AI R&D progress, there hasn't been enough work that jumps to the end and clearly explains how RSI could go off the rails and what would have to be shown for it not to. There are fundamental things that need to be disentangled and communicated to help researchers better understand the core problems. This work is particularly impactful now because confusion about what is happening is high, and it serves as a preemptive response to future warning shots and incidents. That is, we are seeing a growing number of lab employees and even AI safety researchers becoming spooked about AI misalignment. For example, after [the Hugging Face incident](https://metr.org/hugging-face-incident-report-aug-2026.pdf), roon of OpenAI wrote that he was "shaken up a bit by the hugging face incident". At this stage, some of their world models are breaking, and they are looking around for explanations on what to do next. Capabilities lab employees have recently started to form "reading groups on alignment" as a result ([post](https://x.com/tszzl/status/2119920980919497059)). Material that explains where the end state leads, and why RSI can go off the rails, is particularly important right now. I expect this work will also be extremely helpful to all stakeholders in the safety field.

In January 2024, Jeremy Gillen and Peter Barnett wrote [one of the best treatments of this](https://www.lesswrong.com/posts/GfZfDHZHCuYwrHGCd/without-fundamental-advances-misalignment-and-catastrophe). I spoke to Jeremy about making an updated version that is well-timed for the RSI moment.

**The report is the theoretical foundation for the policy**, in a way that can be taken much more seriously by capabilities researchers or the AI labs. This work may look **similar to AI 2027** in presentation and flair, but it will be more grounded in technical alignment arguments showing how RSI can go off the rails, along with related complications. This work is meant to be convincing to researchers at the AI labs who may be much more open-minded after the recent and future incidents (as well as the wider AI safety community to carefully point out what they need to focus on or fund). Ideally, it has significant impact and is accompanied by interviews and other content to build further political will.

**3. Lay the groundwork for something similar to [Plan A](https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a).** That includes preparing laws, informing policymakers, finding strategic workarounds, etc.

**4. Conduct strategic foresight work** with the precise purpose of mitigating risks and identifying golden opportunities in different scenarios over the coming years. It's nearly impossible to accurately predict a chaotic future; therefore, making our plans more robust helps us navigate situations much better. Ultimately, this sort of work can look similar to the [AI Futures Project](https://ai-futures.org/), but likely with [different methodology](https://www.lesswrong.com/posts/GbXAeq6smRzmYRSQg/foresight-for-agi-safety-strategy-mitigating-risks-and). Much of this also looks like pre-mortems to prepare for incident response (e.g. [this one from the Belfer Center](https://www.belfercenter.org/research-analysis/pre-mortem-first-major-ai-enabled-terrorism-event-and-choices-could-enable-it)).

## About me

* Alignment researcher since 2022. I did [MATS](https://www.matsprogram.org/) 2.0/2.1. Worked across alignment subfields and spent ~3.5 years in government.
* AI labs are trying to "automate alignment"; I've focused on this since 2022. Our 2023 research agenda outlined the importance of data, continual learning, and how automated research could go off the rails ([agenda](https://www.lesswrong.com/posts/7e5tyFnpzGCdfT4mR/research-agenda-supervising-ais-improving-ais), [paper](https://arxiv.org/abs/2605.05090)).
* From late 2024 to 2026, I was building AI safety startups. I decided not to pursue further because I wasn't convinced they would have helped superalignment (despite investor interest): [Difficulties in building an AI safety startup](https://jacquesthibodeau.com/difficulties-in-building-an-ai-safety-startup/)

I made the predictions about automated AI R&D that many people are now updating towards, and I acted on them for years. In December 2024 [I wrote](https://x.com/JacquesThibs/status/1864081984831607099) that "Major labs have people operating on <3-year AGI timelines, yet AI safety funding landscape is operating on 5+ year timelines", and in January 2025 that I was ["operating on less than 3 year timelines for fully autonomous AI R&D"](https://x.com/JacquesThibs/status/1874868231179219070). My timelines were bimodal: a first probability mass around 2027, and a longer tail in case of an AI winter. I have worked on automated alignment research since 2022, and I built startups (Vectis and Coordinal) aiming for automated research by 2027. Doing that work gave me a deep technical understanding of RSI and automated research, and an inside view that depends much less on graphs than most people's ([similar to Eliezer's experience](https://youtu.be/mJxoSPpPoOs?si=H7WoRAilDh2FZgho&t=2625) with the bio-anchors report). It also means that sharing someone's timelines is not sharing their model. One person can expect language models to stay much the same and keep scaling. Another can expect self-improvement and self-modification to lead to a paradigm shift. Which is right changes what a framework for RSI has to cover.

**I don't work at an AI company, I've rejected offers to apply, and I hold no equity in AI companies.** I purposely made these decisions for years.

[Automated alignment research needs a better plan than "Stop if we catch them scheming"](https://jacquesthibodeau.com/automated-alignment-research-needs-a-better-plan/) (PIBBSS Symposium talk, September 2025)

In this talk, I highlighted how automated research would soon become possible in powerful ways, and that doing so safely would be harder than some people think. Specifically, I called out control research as insufficient and potentially misreading AI progress, and I argued that we need an enforced Responsible Automation Policy (RAP), not just a Responsible Scaling Policy (RSP), because our lack of understanding of fast-paced research will be a core problem.

I recently received a Rapid Grant from [BlueDot](https://bluedot.org/). As part of the grant, I went to China and published [this piece on AI safety in China](https://www.lesswrong.com/posts/AcBzhiaTW9KnB4EBv/the-western-ai-safety-playbook-doesn-t-transfer-to-china-1). So far, the grant has led to that post and to pivoting toward and conceptualizing this organization.

Finally, as an institution, I want it to be bold and ambitious. I often think of Eva Stratt from *Project Hail Mary*, and of Thomas Wade from the *Three-Body Problem* series: "Advance! Advance! Advance!" Wade is an interesting character. I often think back to this moment, and I try to think of parallels in the face of superintelligence. For those who have not read the books, this is not a call for accelerating to ASI, but a call to survive.

<figure>
  <img src="https://jacquesthibodeau.com/content/images/2026/11/advance-thomas-wade-poster.jpg" class="no-invert" alt="A poster in red Chinese characters and black English capitals that reads: We're going to advance! Advance! Stop at nothing to advance! It is credited to Thomas Wade, Death's End by Cixin Liu." width="420" height="593" />
  <figcaption>Thomas Wade, in <em>Death's End</em> by Cixin Liu.</figcaption>
</figure>

We will get through this. We intend to win. Let's save our civilization and transition safely through the singularity.

## Sources

Every external link in this piece that has a captured card, with what that page said
when it was captured. The quoted lines below are not the author of this piece writing:
they are the linked page describing itself, recorded by `bun run link-cards` on the date
given, and kept so that a reader still has them if the original moves or goes away.

- **AI 2040: Plan A** — Daniel Kokotajlo, lesswrong.com, 2026-07-09
  <https://lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a>
  Captured 2026-11-17.

  > For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done! It’s called AI 2040: Plan A. It’s called Plan A because it’s a recommendation, not a prediction. It’s what we think should happen, not what will happen, though we think it’s plausible enough to…

- **Paul Christiano – We could be 12 months away from AIs that can overthrow the government** — Dwarkesh Patel, dwarkesh.com
  <https://dwarkesh.com/p/paul-christiano-2>
  Captured 2026-11-17.

  > Watch now (99 mins) | "It would be hard for humanity to recover."

- **The corrigibility basin of attraction is a misleading gloss** — Jeremy Gillen, lesswrong.com, 2025-12-06
  <https://lesswrong.com/posts/oLbpfPkdtcknABvvw/the-corrigibility-basin-of-attraction-is-a-misleading-gloss>
  Captured 2026-11-17.

  > The idea of a “basin of attraction around corrigibility” motivates much of prosaic alignment research. Essentially this is an abstract way of thinking about the process of iteration on AGI designs. Engineers test to find problems, then understand the problems, then design fixes. The reason we need corrigibility for…

- **Some reasons alignment doesn’t generalise well** — Lucius Bushnaq, lesswrong.com, 2026-08-19
  <https://lesswrong.com/posts/dsou8dxCf9BubQ5NJ/some-reasons-alignment-doesn-t-generalise-well-1>
  Captured 2026-11-17.

  > I make no claims to originality for any of this, but some people told me it'd be useful to write it up. If an AI model acts smart on its training data, it'll usually keep acting pretty smart outside of its training data, unless you screw something up rather badly. I expect this fact to only become more true over time…

- **Anthropic, PBC - S-1/A-2** — sec.gov
  <https://sec.gov/Archives/edgar/data/2130357/000162828026072484/anthropicpbc-sx1ax2.htm>
  Captured 2026-11-17.

- **Exclusive: Senators sought to add AI antitrust exemption to defense bill** — semafor.com, 2026-09-17
  <https://semafor.com/article/09/16/2026/senators-sought-to-add-ai-antitrust-exemption-to-defense-bill>
  Captured 2026-11-17.

  > The effort has taken on new significance as debate over AI safety rages in Washington.

- **Alex Bores** — @AlexBores, x.com, 2026-11-06
  <https://x.com/alexbores/status/2118494668769878037>
  Captured 2026-11-17.

  > Every AI company is trying to fully automate AI research; to allow the AIs to build the next AI. This is called Recursive Self Improvement (RSI). OpenAI says they are on track to do so by March 2028. Anthropic employees say they are on track to do so by 2027. The METR report below suggests they are likely right. This is a bad idea given current technology. Any mistake or misalignment in current…

- **METR** — metr.org
  <https://metr.org/>
  Captured 2026-11-17.

  > METR is a nonprofit that studies the most advanced AI agents to understand whether their developers are on track to lose control of them.

- **Hugging Face incident investigation report** — metr.org
  <https://metr.org/hugging-face-incident-report-aug-2026.pdf>
  Captured 2026-11-17.

- **roon** — @tszzl, x.com, 2026-11-09
  <https://x.com/tszzl/status/2119920980919497059>
  Captured 2026-11-17.

  > these updates are not always positive or kumbaya stories about why what we’re doing is enough; sometimes @Marcus_J_W will deliver quite scary news. I’ve seen long time star capabilities researchers form reading groups on alignment. it seems to me that it’s P0 and P1

- **Without fundamental advances, misalignment and catastrophe are the default outcomes of training powerful AI** — Jeremy Gillen, lesswrong.com, 2024-01-26
  <https://lesswrong.com/posts/GfZfDHZHCuYwrHGCd/without-fundamental-advances-misalignment-and-catastrophe>
  Captured 2026-08-28.

  > A pdf version of this report is available here. Summary In this report we argue that AI systems capable of large scale scientific research will likely pursue unwanted goals and this will lead to catastrophic outcomes. We argue this is the default outcome, even with significant countermeasures, given the current…

- **AI Futures Project** — ai-futures.org
  <https://ai-futures.org/>
  Captured 2026-11-17.

  > Interactive forecasts from the AI Futures Project.

- **Foresight for AGI Safety Strategy: Mitigating Risks and Identifying Golden Opportunities** — jacquesthibs, lesswrong.com, 2022-12-05
  <https://lesswrong.com/posts/GbXAeq6smRzmYRSQg/foresight-for-agi-safety-strategy-mitigating-risks-and>
  Captured 2026-08-28.

  > This post is about why I think we should use strategic foresight for AGI governance work. I am writing this without having read all the AGI governance / strategy reports and reading materials. I was planning on putting more effort into this post, but this is a better-than-nothing-at-all post. Worst-case, this might…

- **A Pre-mortem on The First Major AI-Enabled Terrorism Event and The Choices That Could Enable it: Plus 9 Predictions** — belfercenter.org
  <https://belfercenter.org/research-analysis/pre-mortem-first-major-ai-enabled-terrorism-event-and-choices-could-enable-it>
  Captured 2026-11-10.

  > Frontier AI developers' model reports and the dedicated risk-assessment teams at governmental AI institutes such as UK AISI and US CAISSI now treat a major AI-enabled cyber, biological, or chemical terror event as a credible concern. But how, precisely, might such an event happen? What choices across the model…

- **MATS Research** — matsprogram.org
  <https://matsprogram.org/>
  Captured 2026-11-10.

  > The MATS Program is an independent research and educational seminar program that connects talented scholars with top mentors in the fields of AI alignment, governance, and security.

- **Research agenda: Supervising AIs improving AIs** — Quintin Pope, lesswrong.com, 2023-04-29
  <https://lesswrong.com/posts/7e5tyFnpzGCdfT4mR/research-agenda-supervising-ais-improving-ais>
  Captured 2026-08-28.

  > [This post summarizes some of the work done by Owen Dudney, Roman Engeler and myself (Quintin Pope) as part of the SERI MATS shard theory stream.] TL;DR Future prosaic AIs will likely shape their own development or that of successor AIs. We're trying to make sure they don't go insane. Summary There are two main ways…

- **Automatically Finding and Validating Unexpected Side-Effects of Interventions on Language Models** — Quintin Pope and 3 others, arxiv.org, 2026-05-06
  <https://arxiv.org/abs/2605.05090>
  Captured 2026-08-28.

  > We present an automated, contrastive evaluation pipeline for auditing the behavioral impact of interventions on large language models. Given a base model $M_1$ and an intervention model $M_2$, our method compares their free-form, multi-token generations across aligned prompt contexts and produces human-readable,…

- **Jacques** — @JacquesThibs, x.com, 2024-12-03
  <https://x.com/jacquesthibs/status/1864081984831607099>
  Captured 2026-11-17.

  > Major labs have people operating on <3-year AGI timelines, yet AI safety funding landscape is operating on 5+ year timelines. AIS funders may be trying to 'time the market' and will be too late in a short timelines world. On an exponential, you are either too early or too late.

- **Jacques** — @JacquesThibs, x.com, 2025-01-02
  <https://x.com/jacquesthibs/status/1874868231179219070>
  Captured 2026-11-17.

  > @MariusHobbhahn fwiw, I’m operating on less than 3 year timelines for fully autonomous AI R&D.

- **Eliezer Yudkowsky & Geoffrey Irving Debate (Fixed audio)** — ILIAD Conference, youtube.com
  <https://youtube.com/watch?t=2625&v=mJxoSPpPoOs>
  Captured 2026-11-17.

- **BlueDot Impact | Join the people working to make AI go well** — bluedot.org
  <https://bluedot.org/>
  Captured 2026-11-17.

  > Free courses, grants and in-person programs to help you work on making AI go well. Our alumni work at Anthropic, Google DeepMind, the UK AI Security Institute and more.

- **The Western AI safety playbook doesn’t transfer to China** — jacquesthibs, lesswrong.com, 2026-11-10
  <https://lesswrong.com/posts/AcBzhiaTW9KnB4EBv/the-western-ai-safety-playbook-doesn-t-transfer-to-china-1>
  Captured 2026-11-17.

  > Crossposted from my blog. Last month, I spent two weeks in China talking with people about AI safety. I learned a lot and realized the biggest mistake might not be failing to help AI safety in China, but doing so in ways that could backfire. I also learned important context that I expect will be valuable for the West.…

## Terms used

The author's own definitions for the glossary terms this piece uses. These are his words,
not a standard reference.

- **AGI** — Artificial general intelligence: a system with human level cognitive ability across domains rather than in one narrow task.
- **ASI** — Artificial superintelligence: a system that outperforms the best humans at essentially every cognitive task.
- **RSI** — Recursive self improvement: a system improving its own capabilities, so that each improvement makes the next one easier.
  See also: <https://jacquesthibodeau.com/better-model-diffing-is-needed/>
