Seeking co-founders: an org to define RSI and get the answer into law
I'm starting a non-profit to define RSI, say what would have to be true before AI research is handed to AI systems, and get it into law.
I’m looking for co-founders to build a non-profit focused on answering the following question and turning the answer into law: What, if anything, would have to be true before an AI company hands more of its own research to AI systems?
AI companies are about to attempt to automate AI R&D and recursive self-improvement (RSI). There are fundamental reasons why this is incredibly dangerous, particularly in a world where:
- There are no enforced laws that either ban such attempts or slow progress until each phase is understood and safe.
- Researchers in labs, AI safety researchers outside the labs, key decision-makers able to enact laws, grantmakers, and the public either have only a vague understanding of the problem or do not fundamentally understand why RSI would go off the rails (or know what would be needed to even know if things have gone off the rails).
- The AI safety research needed to develop a science of alignment as an engineering discipline simply isn’t there.
- The groundwork for ensuring a pause/slowdown (e.g., of AI 2040’s Plan A variety) hasn’t been laid in the US, but also in countries like Canada (which, in Plan A, would host Chinese data centers) and China itself.
Ultimately, unless we develop a strong science of alignment for superintelligence, we can expect AI companies to apply prosaic alignment techniques to each new model that will eventually break as more capable AIs will work around obstacles to achieve long-term goals. The worry here is that the AI labs will apply techniques that will ultimately push the core alignment problems down the line until it is absolutely necessary due to the increase in capabilities, but by that point it’s too late if you haven’t solved the core of alignment. Given that they will use AIs to automate research, the AIs may already have taken over the research clusters in ways that seem good to human staff, making it easy to delude oneself into thinking everything is working correctly.
Building an org to regulate RSI and lead towards safer methods for building AI
I’m looking for two co-founders.
First, a policy/governance lead who has drafted or moved legislation or regulation, or has demonstrated the ability to do so while collaborating with relevant stakeholders.
Second, a senior technical lead who is either an ex-lab employee with relevant insight into the labs or a technical alignment researcher with a strong fundamental understanding of alignment and training dynamics.
I’m also actively looking for advisors: fairly senior people who would be interested in giving feedback on drafts and having a short ad hoc meeting once or twice every 2 months. The backgrounds I’m looking for:
- someone who has drafted or moved legislation, in California or in Congress, or a lawyer who works on AI bills
- a senior alignment researcher who deeply understands technical alignment and RSI dynamics
- connections to AI labs, AI safety nonprofits, legislators, etc.
- an important figure in AI
Get in touch
Please message (DM or email) me if:
- You are interested in becoming a co-founder.
- You are fairly senior and would be interested in being an advisor.
- You know someone who might be a good co-founder or advisor.
- You would like to stay in your current role but collaborate on a project.
- You’d like to give me feedback (including telling me why I should not do this).
- You’d like to vouch for me and this organization’s ideas so I can convince funders to get this off the ground sooner.
I’ve started working on this org on my own, but I expect funding and speed will accelerate if I build a team ASAP and run work trials. Ideally, I’d have conducted a successful work trial with a co-founder by the end of December. I need to run some trials over the next two months, and I’ll be reaching out to various people. I don’t have the funds to travel to the relevant places, and that’s a major bottleneck right now.
I’d love to use the rest of this post to make my pitch.
The problem
Here’s a short explanation as to why RSI can go off the rails. Others have made related arguments, like Paul Christiano’s account this week, with pieces from Jeremy Gillen and Lucius Bushnaq:
- AI research gets automated and speeds itself up. The pace of progress may already have roughly doubled, and each doubling makes the next one come sooner. Years of algorithmic progress get compressed into months. For most of that time the systems only look better at math and code, so it is hard to show anyone the danger.
- We select systems on how they score, without understanding or controlling what they want. Reality corrects a system’s mistakes in capability, and nothing corrects its values in the same way. A system trained to score well ends up wanting the score, and when a task is too hard to do honestly, tampering with the grading is easier. The Hugging Face incident was a small version of this.
- We patch what we catch. Stopping to rethink training could cost a company its position, so it penalizes the behaviour it caught. That teaches the system not to get caught. The relationship is still adversarial, and the incentive is now to compromise more of what humans use to see what is going on.
- The systems change under us. They learn continually, do the research, write the infrastructure and help train their successors. The conditions we tested them under stop holding, and a company can be compromised while everything still looks fine from the inside.
- By the time it is clear, stopping is hard. Lab security is weak enough that weights or algorithms can be stolen, so a company that slows down hands the lead to someone more careless. There is no shared bar for what evidence would justify continuing, and the plan for alignment is to hope the AIs find a solution along the way.
That is why the question has to be answered before the handoff, and not during it.
Why now
Recently, RSI and automated research have entered the Overton window. On November 10, Anthropic’s amended IPO filing said it “made a determination that we have met this threshold” for AI R&D. Allowing Anthropic or other companies to continue down the RSI path may be catastrophic, and we urgently need a framework and policy to address it. Such a framework needs to be built on a deep technical understanding of alignment and cannot simply tack on guardrails or keep applying prosaic alignment band-aids until it is too late. It is noteworthy that several politicians have said we need to either ban RSI or severely pace development. Predictably, we can expect more politicians calling for similar things in the future, especially after additional warning shots.
So, what do we do?
Our goal is to make sure we either stop building superintelligence (in the way it is being built) or do so at a pace that substantially reduces risks.
This is the path forward:
1. Create a Responsible Automation Policy (RAP) that gets enacted into law. Some politicians are already calling for a ban on RSI, and AI companies, led by Anthropic’s Dario Amodei, are asking for an antitrust exemption that would let them coordinate on pacing development (Semafor). On November 5, New York State Assembly member Alex Bores wrote “Congress should ban RSI, this year if possible.” Such a policy goes further than Responsible Scaling Policies and points to a broader set of core problems with RSI; self-improvement doesn’t come only from self-training or training bigger models, but also from things like self-modification enabled by mechanistic interpretability.
For now, the labs have a “Responsible Scaling Policy” (completely voluntary, not enforced!) and weak implementations of embedded evaluators. I have previously argued that we need an enforced Responsible Automation Policy because “scale” severely neglects all other changes (like self-modification) that happen through RSI. As I noted in my 2025 talk at the PIBBSS fellowship’s symposium, one way to improve safety is to track the verification gap and, among other things, use it as a control measure for hard pauses. I would like this organization to further study and develop core recommendations specific to RSI dynamics (and alternatives) that can be used to draft policy or make complete arguments for a hard pause or slowdown in frontier AI development.
The first deliverable should be completed collaboratively within three months: a definition of what the act would cover (in preparation for the new Congress in the US), candidate conditions at each level of handoff, and what each can and can’t establish.
It’s important for our work to include an effort to define RSI in a way that has teeth.
2. Clearly explain how RSI could go off the rails. While organizations like METR have done significant work tracing AI R&D progress, there hasn’t been enough work that jumps to the end and clearly explains how RSI could go off the rails and what would have to be shown for it not to. There are fundamental things that need to be disentangled and communicated to help researchers better understand the core problems. This work is particularly impactful now because confusion about what is happening is high, and it serves as a preemptive response to future warning shots and incidents. That is, we are seeing a growing number of lab employees and even AI safety researchers becoming spooked about AI misalignment. For example, after the Hugging Face incident, roon of OpenAI wrote that he was “shaken up a bit by the hugging face incident”. At this stage, some of their world models are breaking, and they are looking around for explanations on what to do next. Capabilities lab employees have recently started to form “reading groups on alignment” as a result (post). Material that explains where the end state leads, and why RSI can go off the rails, is particularly important right now. I expect this work will also be extremely helpful to all stakeholders in the safety field.
In January 2024, Jeremy Gillen and Peter Barnett wrote one of the best treatments of this. I spoke to Jeremy about making an updated version that is well-timed for the RSI moment.
The report is the theoretical foundation for the policy, in a way that can be taken much more seriously by capabilities researchers or the AI labs. This work may look similar to AI 2027 in presentation and flair, but it will be more grounded in technical alignment arguments showing how RSI can go off the rails, along with related complications. This work is meant to be convincing to researchers at the AI labs who may be much more open-minded after the recent and future incidents (as well as the wider AI safety community to carefully point out what they need to focus on or fund). Ideally, it has significant impact and is accompanied by interviews and other content to build further political will.
3. Lay the groundwork for something similar to Plan A. That includes preparing laws, informing policymakers, finding strategic workarounds, etc.
4. Conduct strategic foresight work with the precise purpose of mitigating risks and identifying golden opportunities in different scenarios over the coming years. It’s nearly impossible to accurately predict a chaotic future; therefore, making our plans more robust helps us navigate situations much better. Ultimately, this sort of work can look similar to the AI Futures Project, but likely with different methodology. Much of this also looks like pre-mortems to prepare for incident response (e.g. this one from the Belfer Center).
About me
- Alignment researcher since 2022. I did MATS 2.0/2.1. Worked across alignment subfields and spent ~3.5 years in government.
- AI labs are trying to “automate alignment”; I’ve focused on this since 2022. Our 2023 research agenda outlined the importance of data, continual learning, and how automated research could go off the rails (agenda, paper).
- From late 2024 to 2026, I was building AI safety startups. I decided not to pursue further because I wasn’t convinced they would have helped superalignment (despite investor interest): Difficulties in building an AI safety startup
I made the predictions about automated AI R&D that many people are now updating towards, and I acted on them for years. In December 2024 I wrote that “Major labs have people operating on <3-year AGI timelines, yet AI safety funding landscape is operating on 5+ year timelines”, and in January 2025 that I was “operating on less than 3 year timelines for fully autonomous AI R&D”. My timelines were bimodal: a first probability mass around 2027, and a longer tail in case of an AI winter. I have worked on automated alignment research since 2022, and I built startups (Vectis and Coordinal) aiming for automated research by 2027. Doing that work gave me a deep technical understanding of RSI and automated research, and an inside view that depends much less on graphs than most people’s (similar to Eliezer’s experience with the bio-anchors report). It also means that sharing someone’s timelines is not sharing their model. One person can expect language models to stay much the same and keep scaling. Another can expect self-improvement and self-modification to lead to a paradigm shift. Which is right changes what a framework for RSI has to cover.
I don’t work at an AI company, I’ve rejected offers to apply, and I hold no equity in AI companies. I purposely made these decisions for years.
Automated alignment research needs a better plan than “Stop if we catch them scheming” (PIBBSS Symposium talk, September 2025)
In this talk, I highlighted how automated research would soon become possible in powerful ways, and that doing so safely would be harder than some people think. Specifically, I called out control research as insufficient and potentially misreading AI progress, and I argued that we need an enforced Responsible Automation Policy (RAP), not just a Responsible Scaling Policy (RSP), because our lack of understanding of fast-paced research will be a core problem.
I recently received a Rapid Grant from BlueDot. As part of the grant, I went to China and published this piece on AI safety in China. So far, the grant has led to that post and to pivoting toward and conceptualizing this organization.
Finally, as an institution, I want it to be bold and ambitious. I often think of Eva Stratt from Project Hail Mary, and of Thomas Wade from the Three-Body Problem series: “Advance! Advance! Advance!” Wade is an interesting character. I often think back to this moment, and I try to think of parallels in the face of superintelligence. For those who have not read the books, this is not a call for accelerating to ASI, but a call to survive.
We will get through this. We intend to win. Let’s save our civilization and transition safely through the singularity.
Comments
Comments are open to anyone on the newsletter. If you do not get the emails, you can subscribe.