--- title: "Better model diffing is needed" description: "The alignment technique I wish existed. Cheap sensors that show what changed inside a network during training, so training can be steered while it runs rather than audited afterwards." published: 2026-01-09 tags: ["Interpretability and model internals"] importance: 6 confidence: "possible" docStatus: "notes" audio: "https://pub-4ee2f71bc29541a7a6e8d9694f0a1b21.r2.dev/6961698f5c58d327551e49e4/audio.mp3" author: "Jacques Thibodeau" canonical: "https://jacquesthibodeau.com/better-model-diffing-is-needed/" ---
Epistemic status

Exploratory. I'm articulating research directions I think are valuable but haven't worked on extensively yet.

An alignment technique I wish existed involves [model](https://www.lesswrong.com/w/model-diffing) [diffing](https://transformer-circuits.pub/2024/model-diffing/index.html) to understand model evolution through training/interventions (like model editing) and serves as a signal to guide training (with a strong [control feedback mechanism](https://www.beren.io/2025-02-05-Maintaining-Alignment-During-RSI-As-A-Feedback-Control-Problem/)) and study model drift. All current techniques seem too costly, not unsupervised or active enough (petri-style stuff is nice, but feels like we need something a bit more fundamental, or at least give a new set of tools to the agent), etc. If people are interested in the alignment implications of long-horizon RL, I think one key consideration is that future models will eventually [discard context-specific heuristics](https://www.lesswrong.com/posts/ntDA4Q7BaYhWPgzuq/reward-seekers) they've been using, because they will be insufficient for solving increasingly complex problems we don't know how to solve (e.g., open-ended research). Therefore, I'd be curious if such model diffing techniques could pick up on such, potentially subtle, changes in the model. * * * This would be follow-up work on [previous research I've done](https://www.lesswrong.com/posts/7e5tyFnpzGCdfT4mR/research-agenda-supervising-ais-improving-ais) with collaborators. I've been trying to think about whether such things would be valuable for an AI safety startup, but I'm iffy on the idea because it always comes back to, "well, am I impacting internal deployment at AGI labs in any way?" It's clearly an important thing to figure out in the context of continual learning (as we pointed out in the research agenda post), though. When we worked on this, we (mostly Quintin) tried to develop a modified technique called "contrastive decoding" where we'd try to do model diffing by effectively using the token distribution as a way to study which sets of tokens M2 prefers over M1 (or vice-versa). The goal was to use the technique to gain some unsupervised understanding of unwanted behavioural side effects (e.g., training an AI to become more of a reasoner somehow impacts its political views). Ultimately, this technique wasn't very useful, and it was fairly costly to run because you were evaluating a lot of text. The main interesting observation was that one of the base Llama models was far more likely to upweight the "Question:" token after the <|startoftext|> token than the instruct model (which we believe was because Meta did some priming at the end of the base model's training to get it used to the question/answer format). Anyway, having reliable, cheap sensors we can use throughout training to guide the process or keep track of how things are evolving in the network seems good. That said, I think this could fail due to not trying to develop techniques that work in the capability regime we are actually worried about or a misunderstanding of key issues like [deep deceptiveness](https://www.lesswrong.com/posts/XWwvwytieLtEWaFJX/deep-deceptiveness):
Deceptiveness is not a simple property of thoughts. The reason the AI is deceiving you is not that it has some "deception" property, it's that (barring some great alignment feat) it's a fact about the world rather than the AI that deceiving you forwards its objectives, and you've built a general engine that's good at taking advantage of advantageous facts in general.

As the AI learns more general and flexible cognitive moves, those cognitive moves (insofar as they are useful) will tend to recombine in ways that exploit this fact-about-reality, despite how none of the individual abstract moves look deceptive in isolation.
In the case of deep deceptiveness and model diffing (model diffing is obviously in other cases), the thing I hope for the most is that changes in internal/external properties can be easily picked up by model diffing through training, and we have such a fine-grained signal of the model's cognition that it becomes clearer that the model is developing cognitive changes that are tangibly different to the current state of LLMs. ## Sources Every external link in this piece that has a captured card, with what that page said when it was captured. The quoted lines below are not the author of this piece writing: they are the linked page describing itself, recorded by `bun run link-cards` on the date given, and kept so that a reader still has them if the original moves or goes away. - **Stage-Wise Model Diffing** — transformer-circuits.pub Captured 2026-08-28. - **Maintaining Alignment during RSI as a Feedback Control Problem** — Beren Millidge, beren.io Captured 2026-08-28. > Recent advances have begun to move AI beyond pretrained amortized models and supervised learning. We are now moving into the realm of online reinforcement learning and hence the creation of hybrid direct and amortized optimizing agents. While we generally have found that purely amortized pretrained models are an easy… - **How training-gamers might function (and win)** — Vivek Hebbar, lesswrong.com, 2025-04-11 Captured 2026-08-28. > In this post I present a model of the relationship between higher level goals, explicit reasoning, and learned heuristics in capable agents. This model suggests that given sufficiently rich training environments (and sufficient reasoning ability), AIs which terminally value on-episode reward-proxies are disadvantaged… - **Research agenda: Supervising AIs improving AIs** — Quintin Pope, lesswrong.com, 2023-04-29 Captured 2026-08-28. > [This post summarizes some of the work done by Owen Dudney, Roman Engeler and myself (Quintin Pope) as part of the SERI MATS shard theory stream.] TL;DR Future prosaic AIs will likely shape their own development or that of successor AIs. We're trying to make sure they don't go insane. Summary There are two main ways… - **Deep Deceptiveness** — So8res, lesswrong.com, 2023-03-21 Captured 2026-08-28. > Meta This post is an attempt to gesture at a class of AI notkilleveryoneism (alignment) problem that seems to me to go largely unrecognized. E.g., it isn’t discussed (or at least I don't recognize it) in the recent plans written up by OpenAI (1,2), by DeepMind’s alignment team, or by Anthropic, and I know of no other… ## Terms used The author's own definitions for the glossary terms this piece uses. These are his words, not a standard reference. - **AGI** — Artificial general intelligence: a system with human level cognitive ability across domains rather than in one narrow task. - **LLM** — Large language model: a neural network trained on very large amounts of text to predict what comes next.