---
title: "Gaining clarity on Automated Alignment Research"
description: "Four different things get conflated in the phrase automated alignment research, and the crux is whether applying techniques we already have is a short step from inventing a new paradigm."
published: 2026-01-09
tags: ["Automating alignment research"]
importance: 8
confidence: "possible"
docStatus: "in progress"
audio: "https://pub-4ee2f71bc29541a7a6e8d9694f0a1b21.r2.dev/696169f55c58d327551e4a17/audio.mp3"
author: "Jacques Thibodeau"
canonical: "https://jacquesthibodeau.com/gaining-clarity-on-automated-alignment-research/"
---
<!--kg-card-begin: html-->
<div class="admonition note">
<div class="admonition-title">Epistemic status</div>
<div class="admonition-content">
<p>Work-in-progress. I'm trying to articulate a common crux of disagreement that leads folks to sometimes talk past each other on the topic of automated alignment research.</p>
</div>
</div>
<!--kg-card-end: html-->

I've been thinking about automated alignment research for a long time and have started working on a post on the topic.

I gave a [talk on the topic](https://jacquesthibodeau.com/automated-alignment-research-needs-a-better-plan/) a few months ago, and I've started writing a long treatment of the topic. Hopefully, I get around to publishing the full thing.

Below, I wanted to articulate a common crux of disagreement that leads folks to sometimes talk past each other.

* * *

Most AI safety plans include “automating AI safety research.” There’s a need for better clarity of what it looks like.

There are at least four things that get conflated in the term “automated research”:

1.  AI uses search to output what was *already discovered* (e.g. finds the solution in existing paper(s)).
2.  AI uses search to find **pieces of a solution** that come together to solve a problem (hopefully in a verifiable domain / lean proof).
3.  AI agents use **existing research techniques** we already know about, and apply them to a variety of new experiments. An example of AI safety research would be using insights/techniques from subliminal learning and emergent misalignment to study new dataset splits and models trained in new ways, while applying existing interpretability techniques with an auditor agent.
4.  Getting AIs to create **novel techniques** that substantially improve the domain in question. This is like getting an AI to come up with a *new paradigm*, which may change how we even think about that research area.

For AI safety, the crux of many disagreements is whether one believes that:

-   **3 & 4 are meaningfully different** in ways that make it substantially harder to get 4 than it is to get 3. Some people even seem to fail to disentangle the two and end up convinced that AIs are solving research as some singular thing.
-   **4-level capabilities are already in the superintelligence-regime**, so it defeats the purpose of using it for safety if you don't have guarantees that it is safe.
-   When talking about superintelligence (the kind that, e.g., can start and grow entire large-scale businesses on its own, solve long-term complex goals like eliminating cancer, and deal with any change in the world that goes beyond its initial training data), **AI safety research needs novel paradigm-level breakthroughs (4)** to reduce risks down to acceptable levels. Meaning that you might expect 3 to be too much within-paradigm, [relatively unenlightened research](https://www.lesswrong.com/posts/5gmALpCetyjkSPEDr/training-ai-to-do-alignment-research-we-don-t-already-know?commentId=2GdrZuNh9fbdF5FWa).
-   **4 is unneeded for a safe transition.** Some folks seem to believe that 3 (which could be described as "relatively unenlightened" research) will be enough to align every subsequent AI, even once we are past 4.
-   **Scaffold and inference compute** at not much higher level of capability is all you need to get 4, and that you'll be fine from a safety perspective because the models are currently useful for research and don't seem misaligned.
-   **3 may produce good research output** (within that set of possible experiments), but you will basically get *slop* for anything in 4 (anything truly out-of-distribution). So, the AI was put through the wringer and believes it has substantially made the next model safe, but, because it is incapable of generalizing well OOD, it fails to align a 4-level model. It has good intentions, but basically only does good safety work for 3-level models and *totally fails* at generating sufficient safety research techniques for aligning 4. It just slops itself to a disaster.
-   **Even if 3 is helpful**, it doesn't end up meaningfully speeding up safety research in comparison to the pace of progress with respect to superintelligent capabilities.
-   **4 involves the AI continually updating its weights**, consolidating insights and placing neatly within its world model. 3 has some sort of disjointed world model that can’t be overcome with fancy scratchpadding and RAG (like, imagine an AI with a knowledge cutoff from 2023 and you RAG in 2026 research, it’s missing *years* of build up in its world model). 3 is suitable for following templates and interpolating within what we know, but fails to *understand* what is OOD.

Ultimately, this seems like a highly important question to clarify, since I believe it is driving many people to be optimistic about AI safety progress, at least to the point that it allows them to keep chugging along the capabilities tech tree. Having clarity on what would convince people otherwise much sooner seems important.

* * *

So, for me, I've started to put a higher likelihood that current AIs are effectively **massive world models of crystallized intelligence**. It just doesn't feel to me that they are generalizing in the way I'd expect if they had *"fluid" intelligence*.

In terms of the alignment problem, this means that:

-   AIs need to **maintain alignment through continual/online learning**, and
-   The reward model needs to **scale alongside it** to help generalize human values out of distribution.

If your system drifts away from human values, it may remain as such.

If current models drift OOD, they likely won't have the internal framework (a human value model) to self-correct toward human values. **Physics will bring a model back into alignment with physical reality** (e.g., by performing tasks in the real world or writing code that meets specifications and passes unit tests), but there is **no such "free" property for human values**.

While crystallized intelligence can be highly capable, current models still seem to lack *true reasoning* (they have some weak form of it) and are poor at generalizing OOD. They can follow templates and interpolate within the pre-training data distribution quite well, but they seem to lack something important, which may contribute to alignment instability.

That said, you likely do seem to benefit from being in a world where the AIs are mostly aligned at this stage. But I think it’s so easy to dupe oneself into thinking that AI models' current capabilities must demonstrate we’re in an aligned-by-default world. Particularly because (imo) MIRI persistently made it seem as if AIs were this capable at code, for example, that we’d be dead by this point.

## Sources

Every external link in this piece that has a captured card, with what that page said
when it was captured. The quoted lines below are not the author of this piece writing:
they are the linked page describing itself, recorded by `bun run link-cards` on the date
given, and kept so that a reader still has them if the original moves or goes away.

- **Training AI to do alignment research we don’t already know how to do** — joshc, lesswrong.com, 2025-02-24
  <https://lesswrong.com/posts/5gmALpCetyjkSPEDr/training-ai-to-do-alignment-research-we-don-t-already-know?commentId=2GdrZuNh9fbdF5FWa>
  Captured 2026-08-28.

  > This post heavily overlaps with “how might we safely pass the buck to AI?” but is written to address a central counter argument raised in the comments, namely “AI will produce sloppy AI alignment research that we don’t know how to evaluate.” I wrote this post in a personal capacity. The main plan of many AI companies…

## Terms used

The author's own definitions for the glossary terms this piece uses. These are his words,
not a standard reference.

- **interpolation** — Answering a question that falls inside the region the training data already covers, between the points rather than past their edge. The strict version is geometric: a point interpolates when it lies inside the convex hull of the training set, and in high dimensions almost no point does.
  Source: Balestriero, Pesenti and LeCun, Learning in High Dimension Always Amounts to Extrapolation <https://arxiv.org/abs/2110.09485>
- **MIRI** — Machine Intelligence Research Institute: an alignment organisation known for theoretical work on agent foundations.
- **out of distribution** — Describes an input drawn from a different distribution than the one a system was trained on. The term is relative to a training set, so on its own it says nothing about how far away the input is or whether the system will fail on it.
  Source: Koh and others, WILDS: A Benchmark of in-the-Wild Distribution Shifts <https://arxiv.org/abs/2012.07421>
  See also: <https://jacquesthibodeau.com/gaining-clarity-on-automated-alignment-research/>
- **RAG** — Retrieval augmented generation: giving a language model a search step so it answers from fetched documents rather than from memory.
- **within paradigm research** — Work that takes the current framing of a field as given and pushes it forward, which is what Kuhn called normal science. Its opposite is work that decides the framing itself is wrong and names a replacement, which he argued arrives rarely and by a different route.
  Source: Stanford Encyclopedia of Philosophy, Thomas Kuhn <https://plato.stanford.edu/entries/thomas-kuhn/>