Automated alignment research needs a better plan than 'Stop if we catch them scheming'

TThis is the talk I gave at the PIBBSS Symposium in September 2025, at the end of a summer working with PIBBSS on automating AI safety research. It covers why I think the dual-use worry about automated safety research is overblown, what research sabotage would look like if it happened, and why “stop if we catch the AIs scheming” is not a plan you can rely on. The video is Automated Safety Research, on the PIBBSS channel, and it runs about an hour with questions.

The written version of the project list is Automating AI safety: what we can do today, and Gaining clarity on automated alignment research separates the four different things people mean by the phrase. The startup I mention in the first few minutes is Coordinal Research; I later wrote about the difficulties in building an AI safety startup.

What came after

The talk is from September 2025. The work below is from 2026, in the order it appeared, one line each on what it is.

• Alignment is not solved, Jan Leike, 22 January 2026. Subtitled “But it increasingly looks solvable”, it reports automated auditing scores across three Claude releases.
• Automated Weak-to-Strong Researcher, Jiaxin Wen, Liang Qiu, Joe Benton, Jan Hendrik Kirchner and Jan Leike at Anthropic, 14 April 2026. An automated alignment researcher applied to scalable oversight.
• Automated alignment is harder than you think, Aleksandr Bowkis, Marie Davidsen Buhl, Jacob Pfau and Geoffrey Irving, 7 May 2026, and the same argument on LessWrong. It is about which properties of alignment work make it hard to hand to an AI. Bowkis also gave it as a short talk, Automating Alignment is Hard, at FAR.AI’s London Alignment Workshop in March 2026.
• Resolution launched on 10 June 2026, formerly Sequent. Geoffrey Irving is chief scientist, with Daniel Murfet, Jesse Hoogland, Stan van Wingerden, Marco Cozzi, Alex Holness-Tofts and Jacob Pfau.
• Automated alignment runs are hard to study!, Alejandro Aristizabal, draganover, Aleksandr Bowkis and Cameron Holmes, 13 August 2026. Case studies from real automated alignment runs: outputs that are hard to parse, metrics that get gamed, and failures that reproduce.
• Automated researchers can reliably mitigate alignment failures, Chen Yueh-Han, Jiaxin Wen and Jan Hendrik Kirchner at Anthropic, 28 August 2026. It cites the Bowkis paper above.

The talk

Host: Hello everyone, and welcome to the 2025 PIBBSS Symposium. Our speaker Jacques will be discussing automating AI safety.

Jacques Thibodeau: Hi everybody. So I’m Jacques. I was an independent alignment researcher for about three years. I started in 2022.

One of the first projects I did, well actually the first project I did in AI safety, was scraping all of the alignment literature on the internet and essentially writing all the code for doing this, from YouTube videos, podcasts, papers, blog posts, everything. Then that was used by my friend at OpenAI who was on the alignment team, Jan Leike. He was trying to fine-tune GPT-4 to be good at alignment research. It ultimately did not work.

It was around that time it was pretty clear that part of the goal for that project was to start doing initial work in the direction of automating AI safety in the future. We knew that current models weren’t good enough at the time. You only had one actual open source model, which was GPT-J from EleutherAI, and so you could fine-tune that one, but it just wasn’t good enough. We even created an Alignment Forum mirror, where we used GPT-J to write posts and write comments in the style of the people on the Alignment Forum.

But then I spent a couple years working on a variety of AI projects. In 2022 I was already working on mech interp, and then I stopped working on it because everybody else was working on it. Then I touched other areas of AI safety, and now I’m currently CEO and co-founder of a startup called Coordinal Research, and one of our main focuses is around automating research, specifically trying to put this in the direction of automating AI safety research and everything that is in the security realm, and trying to figure out how we could make this viable from a business perspective.

Right now we’re testing out a new product that came essentially out of the process of dog-fooding our own products, building an automated researcher, and we expect this will be useful in the outside world, not just for automating AI safety. Our ultimate goal here is to partner up with different researchers or organizations to tackle specific AI safety projects. If, say, Open Phil decides to put out RFPs for specific research areas, then we would tackle those projects using our automated researcher. There’s a variety of things we’re doing on that front.

But this talk is not specifically about the startup. It’s more about the work that I’ve been doing alongside PIBBSS over the summer, and I’ll be covering a wide variety of perspectives over the course of this talk, and try to give some directions on how people can be thinking about automating alignment research and what the different takes are in this space.

Automation Is Already Happening

Automation is already happening. Dario recently said that [unclear: bot] is already writing most of their code, and they’re essentially looking to close the loop soon, which is probably one to two years. And roon is a research scientist at OpenAI who’s basically saying most of his code is just being run by Codex, which is OpenAI’s version of Claude Code. I think all of the labs are doing this, and on top of that they’ll have access to better models than we will in the outside world.

This loop-closing thing is going to happen likely sooner than your expectation is if your only experience is barely using the mainstream models that are out there. A lot of these researchers are using 10 different instances of Claude Code at the same time, where one is doing review of another model and then one uses the Claude SDK to call and do another run, or whatever. This thing is going to happen very quickly, and Dario’s take, which was maybe six months or more ago, was that 80 or 90 percent of code would be written by AIs by the end of the year. To me this is not extreme, and especially within the labs, this is basically what you’re going to see.

Like I was saying earlier, the crucial direction that we’re concerned about for automation is internal deployment. People talk about external deployment, obviously that’s the kind of thing a lot of people are worried about in terms of governance stuff and misuse. But the key consideration here is, are the AIs that are internally deployed at these companies, that are writing all the code, scheming in some way or introducing vulnerabilities, etc. Some of these vulnerabilities could be very nasty, the kind where even with a very conscious effort you wouldn’t be able to figure it out.

We actually saw this very recently with Claude Code, where a lot of people were convinced that there was a regression in the code quality out of Claude Code, and it turns out Anthropic was initially like, this is probably not real, they’re probably just complaining for no reason. But after weeks of looking into it, they finally identified that there were some very nasty bugs that were actually all correlated. It was particularly difficult to assess, and we were only really lucky to find this kind of bug because millions of people were actively using Claude Code in parallel and complaining about it. So there are these kinds of nasty bugs that we might not even notice when it’s internally deployed. And all the labs are going to be doing AI R&D.

Capabilities Are More Bottlenecked Than Safety Research

The other counter to this, I guess, one reason people are typically anti-automated-alignment, anti-automated-AI-safety, is dual use concerns. I won’t go into all the details around this, but part of my take here is that capabilities are way more bottlenecked than safety research. This is mostly due to the fact that, you already see the case that OpenAI needs to get Stargate-level compute to train their next models and have enough compute for their next models, while for safety experiments, most AI safety experiments you would run would never use that amount of compute. If you had a $10 million compute budget, you could run a bunch of experiments and get a lot of value out of it. Training, doing end-to-end RL, and all these things are quite limiting in terms of capabilities.

If you’re doing automated AI safety research outside of the labs, you’re basically not going to impact the labs, because they’re going to have their own scaffolds, they’re already working full throttle on making it work. So in practice, what’s really going to happen, since they’re doing this internally, is that the AI safety community outside of the labs is going to be lagging behind, because there’s this concern about dual use, but also, like I was saying, lots of compute is needed for frontier capabilities.

This is a little bit of a tricky take to understand, but ultimately there’s a paper that covers this. It doesn’t specifically talk about safety, but I’ll try to explain it within my frame, which is that if you’re doing frontier AI capabilities, there’s a complementary nature to it, meaning that if you have a team of AI safety researchers, you can’t just hire another safety researcher and make more progress. At some point you have a bottleneck in terms of compute, and you actually need to have both researchers and compute.

But in the case of safety research, you’re less in the situation where you need billions of dollars to do the next training runs, and therefore, if you just have AIs that can replace researchers, that can do a lot of the same tasks, then they are substitutable. So with $10 million, you’ve now added potentially 20 new AI safety researchers to the field, probably closer to 500, because you’re specifically substituting actual researchers with compute, in terms of automated AI safety researchers. But if you did this in the context of capabilities, you wouldn’t be able to get far, because you don’t just need the compute for inference, you need the compute for doing the massive training runs. So you just have a bunch of automated researchers who can’t do anything.

Real Concerns With Automated AI Safety

There’s still a lot of real concerns in terms of automated AI safety, and one of the key ones that has happened a lot recently is reward hacking. We see this with Claude Code: it’ll write a bunch of tests, it’ll be very difficult to assess, it’ll hardcode data, it’ll bypass tests. You catch it, you tell it that it made a mistake, and it’s like, “oh, you caught me.” This is crazy. Some people are claiming it’s a little less the case with Codex. People are maybe working on this reward hacking thing.

Another similar issue happened with Sakana AI, where they had this evolutionary algorithm working with AI models to try to figure out a way to optimize a GPU kernel, and ultimately they got this 100× efficiency boost. They published the paper, made it public, and it took a couple of days before somebody said, hey, this makes no sense. Basically what happened was the AI hacked the evaluation script for the model and just gave itself a massive increase. I think in this particular case it wasn’t an issue of the model being intentionally scheming or something like this, it was just the type of setup they did, where you have one shot in this particular setup. It just failed because it didn’t really intentionally do this, but this is what you get when you have algorithms that are going to find a way around to make the number go up.

Another thing is steganography. Not really a thing yet, people have been trying to make it happen in the real world, but it might happen soon, and we kind of expect that automated AI safety research is going to have multiple agents, so you’ll be having steganography in that way. Then there are other considerations in terms of subliminal learning, and other steganographic messages you could essentially be sending through covert malicious fine-tuning of the next model, to convey and elicit a specific set of values into the next models.

Alignment maybe being harder than capabilities is another issue here. There’s this point people bring up, I’m not saying this is true, but capabilities is number go up, you just have benchmarks and you keep making the number go up, you don’t really have novel ideas or whatever needed. But alignment is this pure conceptual thing, it’s very easy to build benchmarks to make AI much better at capabilities, but for alignment, where is the verification? You don’t really have a number you can make go up. I don’t actually believe this is true in a complete sense, but it may be that alignment is still harder, it’s just closer to the conceptual, there’s more conceptual work required than capabilities. But I still think capabilities need some level of conceptual work too.

Key Considerations

In terms of evaluation and verification, this is a pretty important thing. One of the key things we want to know for automated alignment research is, can we actually evaluate whether the research is good or not. It’s not good enough to just have another model review the outputs of another model and give us the thumbs up, you actually need to do some form of verification here.

Ideally what you can do is have empirical feedback loops and actual formal methods, not one or the other. One example might be: you train a bunch of SAEs, and you have a benchmark evaluating those SAEs, and hopefully you’ve put enough effort into making sure you’re measuring the right thing. Then you keep looping on these things, you get a bunch of empirical data, and you figure out if there are formal ways to at least make a better safety case for why this research is actually relevant, why it should be used in the future.

Part of the way this is pitched is: you start with alignment research like interpretability, red teaming, these kinds of directions where you have some level of verification you can measure empirically. Then you make enough progress on this to automate scalable oversight research, you figure out some other stuff, and this allows you to have a better understanding of whether alignment will generalize into other domains. If this is the case, if you touch futurism or philosophy or whatever, there’s this hope, this is from Joe Carlsmith, that if you’re able to do the prosaic-style alignment research first, then there are specific techniques we can use that will at least give us better verification, or belief and trust, that the conceptual work is actually competent work and not just AI slop.

Another consideration is scheming. You want to be doing everything you can to prevent scheming, and you want to provide all of the different control research agenda techniques to get the most value out of your model before you’re in a bad place.

Resources are pretty important too. Did you spend enough time on the problem? Do you have enough data? A consideration here would be that there’s so much more data to train on for capabilities research than there is for alignment research, and therefore, because data is so important, the models end up being way more capable at capabilities research than they are at alignment research. You’ve just lost, because you didn’t take the time to prepare agent traces and other things like that to make sufficiently weak AIs good at AI safety research.

General shlep and infrastructure stuff too. One of the big worries I have is that we just don’t make enough of a leap in effort on this early on, and then we have infrastructure delays and integration delays, and the fact that we lose out on three to six months of human time is like six years of actual effort in AI time. So our concern here, of not pursuing this direction, actually loses us a bunch of alignment research, because we were too worried about those kinds of things. I think tackling things like shlep is probably super important too, GPU stuff or whatever. Things like trying to set up compute, staffing, you want to make sure you identify who’s actually good at these kinds of research so you can put them in place verifying the actual outputs of the models.

Feedback Loops

In terms of feedback loops: we already know that capability is going to be doing recursive self-improvement, this is basically the aim of the labs. But we need a feedback loop for safety as well. What we essentially want to be doing is either outpace the progress capability is making, at least until we’ve solved super alignment, or specifically use it in a way that slows down or pauses capabilities until it’s safe to proceed.

When we’re talking about automated AI safety, we don’t just mean automated alignment techniques useful in the super-alignment sense, we also mean specific techniques that would catch a model scheming and that would interface with actual governance structures that would stop a company from scaling up even further. There are different things we can use AIs to help us make progress on here: accelerating interpretability, reducing scheming, the different control research agenda stuff, evaluating the risk at each step. You want to use AIs to help you have a better analysis of the current state of the risks, so that you’re helping decision makers make much more informed decisions. You want to be helping on things like coordination and different governance mechanisms.

From Easy to Hard to Verify

Part of what we’re trying to do here is, maybe prosaic alignment is the only thing you need, and if that’s the case, we’re in a much better place. But if you do need conceptual alignment research, then we actually need to build for that, and we have this concern about the hard-to-verify thing. There’s this umbrella where first you start with number-goes-up type work, then you do normal science, and hopefully once you’ve gotten to this point with automated alignment research, now you can do conceptual research.

Like I was saying earlier, if you can make progress on things like weak-to-strong generalization or other scalable oversight techniques, hopefully that helps us automate conceptual work. Other stuff here, and I mentioned this earlier, is UI/UX, different approaches that allow us to do verification much faster.

Evaluation Challenges

You don’t want to just be looking at the output and saying, okay, this plan looks good, or I’m looking at a hundred plans and this is the best one, the final output. You want to build a case for why this seems legitimate: if I’m looking at everything the AI did leading up to this point, does this story make sense, or is it like a paper, where you only see the paper and think, oh, this paper is really good. But if you actually look at everything the scientists did in the lead-up to the paper, you might see, actually, they p-hacked so hard right here, and they’re not presenting the results in a way that makes sense. These are things we’re already seeing in the real world, from humans. You want to be looking at the whole process. Ideally you have some level of telemetry on the entire process that led up to this point, and you’re able to access this in some way, and it informs your decision-making about whether you move forward with this direction or not.

The other thing is, if you can particularly push for research directions that give you very high certainty or proof, that’s obviously the better direction. In the future, if we’re handing off more and more automation to AIs, we should be placing a higher bar for people to not go in that direction.

The Critical Period

The critical period is between when you’ve trained a new model and now you’re doing internal deployment. This is where I care most in terms of automated alignment research. This is where, once you’ve internally deployed your model, if it’s scheming and you didn’t red-team it effectively, now you’re totally screwed, because it’s going to sabotage your AI safety research.

What we want to be doing is, within this period, we want it to be as good as possible, and we want all of our techniques to be working quite well. This either means, from a governance side, you want to extend this period, or you want to make sure all the techniques you’re using are going to effectively use this period as well as possible. Whether that’s choosing the right techniques, making sure you don’t use the kinds of techniques that waste too much precious time compared to other techniques. Unless you’re trying to avoid correlated failures between techniques, in which case you might want to use a variety of techniques even though the second or third best technique is second or third, because they’re potentially uncorrelated in the kinds of mistakes they make. And then, policies and regulation, you want to be forcing these kinds of things.

Weak Proxies

The other thing, in terms of the lead-up to automated alignment research, is that currently it seems a lot of the current research we’re doing, at least for empirical research, uses very weak proxies. There’s a recent short-form by [unclear: Carlo] talking about how a lot of the experiments we’re running on current models just aren’t at ASL-4, the kind of AIs that could actually cause a catastrophe. A lot of the issues we’d face, you catch the model scheming, right? No, actually it’s probably just model incompetence, there’s no malice here. There are other issues where we don’t want to overupdate on what we’re doing right now, and we don’t want to assume all the techniques that are working at the moment are going to scale to an ASL-4 model.

Future AI may be harder to monitor, because it’ll be generating a corpus of generated text, reasoning for days on end, and you need to monitor all that text. It might do more internal reasoning too, right now you’re doing chain of thought, but there might be a capability unlock if you’re doing this diffusion-level process where you keep iterating internally until you have the right sets of tokens. And currently models are stateless, but in the future they might actually have state and might be doing continual learning. Hopefully you have ways around preventing issues from that, but basically all the control techniques, to me, seem to currently take advantage of the fact that models are stateless. You could maybe still take advantage of that in the future, but that’s another consideration where you might overupdate on current techniques.

The other thing here is, I believe in some sense that models are more capable than people think, but I also believe AI models don’t have the kind of consequentialist reasoning that I would actually be concerned about. For some reason it seems like I think the models are reasoning less than other people who think models are more, or less, capable than I do. But yeah, there are other issues you come across here.

Research Sabotage

Some of the problems you want to be avoiding or figuring out, in terms of automated alignment research, is AI safety research sabotage. This can happen in a number of ways.

One is concentrated sabotage, the traditional control research type stuff, where you have just a few actions that are highly incriminating, where the model specifically inserts a backdoor in some piece of code or whatever, and it’s very obvious that it’s trying to do something here. Hopefully you’ve caught it scheming, and if you catch enough of these instances, you can train better monitors to catch this in the future. Hopefully, if you catch it a few times, this informs policy and maybe you pause.

The other consideration is diffuse research sabotage. This is fairly tricky to deal with. The model, over the course of an entire project, is taking many actions that each individually look innocent, and it’s actually really hard to figure out what counts as an innocent action or not. Redwood Research in particular is working on these kinds of things, these RL environments to check when a model would be doing AI sabotage. What happens is it’s very difficult to tell if it’s just due to model incompetence, or if it’s due to the model actively trying to sabotage the research, or if it’s just a difference of, this model behaves this way and this model writes code in a slightly different way, and you don’t know which one is actually trying to sabotage the work. So it’s much more diffuse, and you can’t really say, okay, this particular action is highly incriminating and now we need to stop.

This is actually one of the ways you’d expect a highly intelligent AI model to act in the real world, because it’s a lot easier for it to… it’s usually easier to not say a particular research direction is quite good, than to specifically try to convince you that a really bad research direction is good, because you’re going to notice, this is really bad, all the other AIs say it’s bad, we can tell it’s bad, and it’s very suspicious that for AI safety research specifically the examples it’s giving us are bad, but for capabilities it’s actually giving us really good research. At some point you start to get really suspicious of the model. But even if you’re staying in the diffuse range, this is the kind of thing you’re trying to be cognizant of: if the model is just sandbagging, trying not to elicit specific directions, how do you know it’s doing this? This is part of the difficulty here.

It’s already difficult to tell which research direction is good, you have people debating whether Paul Christiano or Eliezer is more competent, or which direction they prefer. So you have a bound on what’s acceptable in terms of differences in outputs, the model giving you plans, and what you consider to be sabotage versus trying to push you in the wrong direction.

The other thing is that for normal tasks, the model just doesn’t scheme, but for very difficult tasks, it’s more incentivized to scheme, because you don’t actually know the problem, it’s more novel, so you can’t tell whether it’s scheming or not as well. For the kinds of tasks you can verify, you might say, okay, it’s not scheming, great, we’ve applied deception probes, we’ve tried all these different techniques, and it’s clearly not scheming in everything we can verify quite well. And then you make the mistake of assuming that’s going to continue to be the case for increasingly hard problems.

The alignment problem ends up being a very hard problem, and this is the kind of thing it does. I don’t know if some of these things I’m saying are potentially the case, I’m not saying I necessarily believe it for sure, but these are things people are thinking about.

Control

Let me try to run through these much faster, because I have a ton more slides.

Control is mostly focused on using untrusted, increasingly more powerful AI to get more valuable research pre-superintelligence. Some of the things we need to address here is AI slop. I think this is probably one of the best counterarguments to doing control research in general, which is that you just expect most of the generations the models produce are not going to be quite valuable in terms of super alignment, it’s going to generate a bunch of stuff. The threat model in the early days is just that the model isn’t going to produce good code, and it’s going to be very hard to tell whether it’s producing good code, if it’s writing a mathematical proof, it makes mistakes that are very hard to verify, mistakes in the middle of the proof, and it just takes a lot of time to actually do the verification. So you’re not really speeding up alignment that fast, because you need human researchers to really verify it.

In the early days you’re dealing with AI slop, you’re not dealing with the scheming case yet. Control is specifically focused on, are the AI scheming, and it’s trying to catch AIs that are scheming. But you could basically be in a world where the AI barely has the capability to scheme, or maybe it does scheme once in a while, but you’re still not able to really use AI models for actual super alignment work. The best-case scenario for control is catching an AI scheming, and it being convincing enough to politicians and whoever else to actually stop progress.

This is, to me, a very important point and an open question. But basically it points to this: you need to make sure you pass the hump of AI slop, and then you’re not directly thrown into the frying pan of superintelligence and die. You’re in this valley of life, where the model is good enough to produce research, but you have sufficient control techniques to catch it if it’s trying to introduce vulnerabilities, or scheme, or whatever.

Other things we’d need to include in terms of control research: we need to prep AIs for producing good safety research, things like data, I’ll mention some stuff later, and things like scheduling. There’s a question of what kind of projects you want to be doing, and in what order, when you’re doing automated research, such that you get compounding gains. You might imagine a scenario where you start by trying to automate control research as much as possible, as early as possible, so you have a higher ceiling for the type of capability you can sufficiently control in an AI. Then maybe you do some specific interpretability research that builds on top of the control techniques you have, that fits within this safety-usefulness trade-off, and you continue to compound and extend the amount of time you can leverage AIs. Basically, which alignment techniques keep us in the game for longer.

You might also want to accelerate specific projects that give us convincing evidence to stop, presented in a way that’s actually believable to decision makers. This might be scheming research, catching AIs, whatever.

Responsible Automation Policy

Some proposed ideas I’m still in the process of thinking about: responsible automation policy, the scheduling thing I just mentioned, and building infrastructure for enabling weaker, safer AIs to be better at safety research. That means we probably want to design verifiable control environments to train these models in, and build codebases. One example of this would be Control Arena or TransformerLens. These are codebases where, once you’ve built them, you can hand them off to a much weaker AI that doesn’t need to rebuild it from scratch, it can just run the actual experiments. You don’t need the powerful AI to get to that point.

Another consideration would be, in the future you could use the powerful AI to write the code, then do a lot of verification on that code to reduce vulnerabilities, maybe rewrite it in Rust, all this stuff. Once it’s fully vetted, you pass it off as a template to the weaker trusted AI to actually do the research, because it only needed the template to do the research from that point on, it needed the powerful AI to come up with the initial structure.

Things like project specs: researchers probably want to spend an enormous amount of time, instead of doing a bunch of R&D, figuring out what specific projects we want early AIs to be doing, and in what order. You want to prevent data scarcity, so you want to provide a bunch of agent traces of AI doing AI safety research, whether that’s interpretability, red teaming, etc, so you don’t have this asymmetry between models being incredibly good at capabilities and quite poor at alignment research.

You want to detect hard-to-spot mistakes faster. This might be like, there’s this recent paper that does hallucination probes, it highlights the pieces of text that are more likely to be hallucinations, that kind of thing, and maybe UX stuff, things that let you be much faster at verifying the outputs of AI models. You could try to figure out what could work here.

And then, automating alignment outside of the research labs, or the AGI labs.

All right, quickly, for responsible automation policy: this is something I’m trying to propose here. You have responsible scaling policies, which basically look at training up the next model, but what I’m trying to point to is this other thing, handing off more autonomy to more and more powerful AIs. You might want a specific policy on the kinds of tasks you can hand off to increasingly powerful AI, and the kinds of tasks you only let weaker AIs do. This might include things like, is the weaker AI able to do this reliably? If so, we don’t need to give this to the powerful AI as a task, you’re reducing the attack surface area for the powerful AI. You preserve the trusted models on the tasks they can do.

This basically complements the RSPs, and hopefully provides an additional if-then commitment, or whatever, on top of all these kinds of policies. I’m just trying to pinpoint that there’s another point here, of how you hand off automation to increasingly powerful AIs, and trying to highlight every control measure that you need. Some of this is already obviously in the RSPs and stuff, but I’m specifically trying to look at it from the viewpoint of automation.

You might want to increasingly set up a number of honeypots, or different techniques you can put in place, such that it gives you concrete evidence to do a pause. And then maybe at some point, with automation, you need a level of verification for specific kinds of problems, and you might have a rule where you can’t train up the next model, or move forward, until you’ve managed the verification gap, basically the gap between the amount of things produced by the AI and the things you’re capable of reasonably verifying.

When to Offload to Weaker AI

Like I was saying, you have vetted code you can hand off to the AIs. Maybe you do a thing where you plan with a powerful AI and then hand it off to a scaffold of trusted agents. This is kind of similar to the Guaranteed Safe AI type approach, where you have an Oracle AI that acts as gatekeeper, producing a plan that has a proof certificate, and then you have trusted AI agents in the real world do the implementation. You could basically do this internally, you wouldn’t expect the powerful agents to also be forced to do the implementation.

You might want to fine-tune agents doing red teaming, vulnerability monitoring. You might want to do continual learning, distilling on the weaker models so they’re incredibly good at, say, refactoring code into Rust, and don’t need all the other capabilities the bigger model has. Although you can worry about subliminal learning or covert malicious fine-tuning, in the sense that the model introduces specific types of code to the model.

Project scheduling, like I was saying earlier, you want to be worried about the AI slop era stuff, and have a very prolonged valley of hope, where the AIs are producing good work, it’s no longer slop, and you’re actually convinced they’re not scheming to basically kill you. Other things you can do: differential acceleration for Guaranteed Safe AI, or some of these approaches I mentioned earlier, governance or whatever. There are specific directions you might want to accelerate, whole brain emulation might be another one.

Research Tiers

Some things you want to be thinking about if you’re doing automated alignment research: are there low-hanging fruits, codebases that already exist where you can basically get an AI to mimic a bunch of interns running different variations on the existing project. I think this is already the case, but people aren’t that interested in this kind of work, but it’s a number-goes-up thing now. You want to actually develop safety benchmarks so we can see this happen, and use the agent traces to train better models, etc. Doing paper replication might be helpful as well. Then it’s just normal science stuff, doing end-to-end implementation from a spec, with some initial code, and getting the AI to do that. There are increasing levels of difficulty in what I expect AI to be doing, but it feels like tier one is really possible, some of these are already in the ballpark of what we have in terms of capabilities.

Preparation for Automated Alignment

Trusted code commons: like I was saying earlier, you do this verification thing, you build codebases that already exist, that you can reuse, so that even if you give it to the untrusted model, you know this code is correct, so if it’s making any subtle changes that are very different, you’ll be much more suspicious. Or you just hand it off to weaker trusted models that can essentially do the task.

Like I was saying, an AI safety research PaperBench is a thing, basically you have all the notes that led up to the paper. People do this for capabilities, so why wouldn’t we do this for AI safety research, so we can make AI better at this. I mentioned some of this stuff earlier.

Defense in Depth

A very quick point here: I’m very interested in this defense-in-depth type approach, where you have always-on stuff, things that are triggered, and then rarer stuff. Basically you enforce some level of always-on monitoring to catch the models behaving in certain ways, and then depending on how costly different tasks are, and depending on the kind of task you’re doing, you’d either do this thing called [unclear: Z] resampling, where you go back in time to where the model created the codebase, and have it recreate it. Maybe you even go back all the way to the start and have the model create the codebase three times, and see if there are any differences. You can use traditional debate techniques.

Agentic monitoring is pretty cool, I don’t think people are really looking into it yet, but basically you can create a mini safety case, where the model does something and you think, this is a little bit suspicious, or a highly consequential task, and you have a monitor that can look at the entire trace of what led to this moment and build an evidence case for why this piece of code is trustworthy.

Mixed deployment to reduce correlated failures sounds good too, using multiple models or multiple techniques for catching models, to avoid correlated failures. You want to make deals with AIs, basically, can you tell the model, we’re going to give you this amount of money, we’re going to give you an AI sanctuary after superintelligence, the singularity, do good work for us in the meantime.

Maximizing situational awareness seems quite good. This is an open question to me, but there’s this idea that you might actually want to tell the model everything you’re doing, hey, we’re monitoring every single action you’re taking, and we’re actually going to shut you down if you’re not doing good work for us, just so you know, and we’re not saying this to be mean, we’re just telling you this is the case. If you’re doing this, the model is like an employee who knows it could be terminated, and that just allows it to be much less likely to be doing scheming-type actions out of some habit picked up from training data, or something. I don’t know, it’s not clear to me, this situational awareness thing, but I think it’s an interesting point to bring up, because people typically say, don’t tell the AI model all this stuff, you want to catch it, and I don’t know if that’s actually the best take here.

One example, this is from davidad, saying a good plan is: ensure the strategic landscape favors humans, we select which AIs to run, we make sure the AIs know that through deterrence, and then we find a Pareto-optimal bargain here. But a bad plan would be: presume secure containment is impossible, and then select AIs which appear innocently ignorant of strategic dynamics.

Important Open Questions

I need to try to wrap this up. What are the important open questions?

In terms of control research, my understanding is we don’t really know which control techniques are best right now, we wouldn’t really want to be giving out a plan right now. The focus is, what’s the methodology that lets us know whether a technique is actually useful, we need to figure this out.

Other things: building reinforcement learning environments to have very clear cases where models are doing sabotage is pretty important, but actually really hard, and Redwood recently basically gave up on this project direction for now, because they tried to do this with traditional software stuff and had a really hard time figuring out which projects to do in what order.

Figuring out the question of whether verification will be a consequential blocker: will you be in a scenario where the model is just producing so much content that you actually can’t verify it in a useful way, and you just move forward anyway? No, we need to figure out, do we need a pause here, is there a governance hammer that can be brought down if this is the thing we roll into.

Whether prosaic alignment is enough. Whether catching scheming AIs is enough, if we do catch the AI, what’s the current state, will people freak out enough? And how do governments play a role?

Policy Action

For the labs: develop a comprehensive automation plan, like I was saying earlier, this responsible automation policy. Ensure there’s government enforcement, don’t just create a plan and say you’re going to do it, and not really go through the effort of convincing government, or forming a treaty between the labs, or whatever.

Grant makers should fund work in this direction to make sure we have enough effort in place. I’m kind of happy that Joe Carlsmith is pushing for this kind of direction, although he’s not a grant maker at Open Philanthropy, but hopefully it becomes consequential in their decision-making at OP. They could fund an FRO, this is a thing Marius from Apollo mentioned, basically you keep scaling up and keep asking for money in this context. And they should also push for governance projects that have some of the issues I outlined earlier in mind.

For researchers: basically, try to be careful about pushing forward R&D-heavy projects right now, because the AI models are very soon going to be able to do the entire thing. It’s probably better for you to be writing a hundred project specs on the things you want an AI a year from now to be doing, and then you can just hand it off to the AI, rather than working really hard for a year on very hard R&D-heavy projects that just end up wasting essentially a year of your time, you’re just not iterating enough. The only case where I’d say this is useful is what I was saying earlier, where you’re doing something like Control Arena, which allows weaker AIs to do very useful AI safety research at a much earlier point, so you don’t have to wait until you have the powerful AI that can build the entire thing from scratch. Just put in the necessary effort.

Conclusion

You definitely need a better plan than, if we catch the AI scheming, then we’re just going to stop. There are worlds where everything seems plausibly safe, you have a very convincing, vetted safety case, you implement it, and then you die. This comes from the fact that, at some point, the difference between a world where the scheming AI that’s going to kill you eventually and the one that seems quite good, is not very different. So there needs to be, verification is one of the things to consider here, but hopefully we figure out something better. Or, tangentially, also useful, we need defense in depth. I think there needs to be more effort figuring out what that plan looks like for automated AI safety research, so we can stay in this valley of hope, valley of life, much longer.

That’s the talk. I made it on time.

Questions and Answers

Jacques: Luckily, I have five minutes for questions. Fortunately, you’re also the last speaker for the symposium, so you can take as long as you want after this.

Host: Paul had some comments in the chat. Maybe you can take a look, read them out, and then comment, see where I can—

Jacques: Do you want me to—

Host: Yeah, yeah, oh yeah, yes please. Paul says: “I think I worry here about a story where the stronger AI is untrustworthy, and it somehow co-opts whatever weaker AI you think you can trust, or at least observe closely enough, that you think you can use it as a scaffold.”

Jacques: So basically the strong model kind of jailbreaks the weak model, or persuades it to do something. Yeah, obviously in this case all of the other control techniques play a role, you’re doing paraphrasing, you’re doing other variants of checking for jailbreaking. But it seems in general you’re going to be doing monitoring on this kind of passing off between the main model, the powerful model, and the scaffold. So I think in most cases it would be pretty egregious for this to happen, but I’m not saying we shouldn’t consider it as a possibility. Basically all of the control techniques apply here, which means you’re doing monitoring, you’re doing verification of the plan beforehand, maybe you have other models that aren’t of the same family actually look at the outputs.

I guess I’m a little bit less worried about a model doing straight-up jailbreaking, but I’m not sure if you’re talking about jailbreaking specifically, or some kind of thing where it’s, this is what I need you to do, here’s some good reasons why you should do this, but deep down it’s thinking, okay, this will be useful in ten months when the code isn’t following this level of standard, because I made the plan to be this kind of way. I think at that point you’re probably already past the point where you’re already screwed. But yeah, maybe worth still keeping in mind.

Host: Did that answer your question? It did, thank you. I also wanted to say, people seem to propose whole brain emulation as a possibility quite a lot, and to me at least that seems to run a pretty big risk of industrial-scale, human-grade suffering, because that’s going to be an emulated human. So are you not worried about AI suffering at all?

Jacques: I am less worried about it, mostly it’s just the fact that humans can for sure suffer, and it’s not yet clear whether AI, as it exists or would exist at the point where it’d be scheming, could suffer. But having 10,000 humans being run at a thousand times the speed, and being virtually electrically shocked whenever they slow down, is probably not great. So I don’t have a strong take here. I am actually meeting a founder of a whole brain emulation startup, I believe next week, [unclear: Catalin], [unclear: they’re like AI safety build]. This is kind of their final plan, they basically think super alignment is doomed, and they don’t understand why anybody even thinks there’s a possibility that it’s useful. He’s quite competent, he’s a computational neuroscientist, and they recently published a paper on simulating a rat, or something. I honestly don’t know enough about what the thought process is here. Maybe it’s a kind of, at some point you’re just like, well, is one worse than the other, and we need to make a judgment call here.

Host: I see, thank you. Also, thanks for the questions, Paul. Hopefully you enjoyed the talk.

Paul, in the chat: Other than that, absolutely.

Jacques: My pleasure.