---
title: "Differential Training Process: Delaying capabilities until inner aligned"
description: "Keep the model from working out that it is in a training loop until its values are where we want them. Plus the two objections I have not answered."
published: 2022-12-07
tags: ["Interpretability and model internals"]
importance: 5
confidence: "possible"
docStatus: "notes"
cover: "https://jacquesthibodeau.com/content/images/2022/12/DALL-E-2022-12-07-16.52.13---Good-humanoid-robot-vs-evil-looking-robot-running-in-a-race.-Finish-line-photo-finish.-trending-on-artstation--greg-rutkowski--boston-marathon.png"
coverAlt: "A generated illustration of two battered humanoid robots sprinting side by side down a running track toward the viewer under a dark blue sky."
audio: "https://pub-4ee2f71bc29541a7a6e8d9694f0a1b21.r2.dev/6390be413de6b56ce4c0c8c5/audio.mp3"
author: "Jacques Thibodeau"
canonical: "https://jacquesthibodeau.com/differential-training-process/"
---
I've been ruminating on an idea ever since I read the section on deception in "[The Core of the Alignment Problem is...](https://www.alignmentforum.org/posts/vMM6HmSQaKmKadvBi/the-core-of-the-alignment-problem-is-1#Deception_)" from my colleagues in SERI MATS.

Here's the important part:

<!--kg-card-begin: html-->
<blockquote class="epigraph">
When an agent interacts with the world, there are two possible ways the agent makes mistakes: <strong>First</strong>, its values were not aligned with the outer objective, and so it does something intentionally wrong. <strong>Second</strong>, its world model was incorrect, so it makes an accidental mistake.<br><br>
Thus, the training process of an AGI will improve its values or its world model, and since it eventually gets diminishing marginal returns from both of these, both the world model and the values must improve together. <strong>Therefore, it is very likely that the agent will have a sufficiently good world model to understand that it is in a training loop <em>before</em> it has fully aligned inner values.</strong>
<footer>from <a href="https://www.alignmentforum.org/posts/vMM6HmSQaKmKadvBi/the-core-of-the-alignment-problem-is-1#Deception_">The Core of the Alignment Problem is...</a></footer>
</blockquote>
<!--kg-card-end: html-->

So, what if we prevented the model from recognizing it is in a training loop (e.g. preventing/delaying situational awareness) until we are certain it has fully aligned inner values? In other words, we could use some stronger forms of model editing to remove specific knowledge (or prevent the model from gaining that knowledge) from the model. Perhaps you penalize the model from learning things that are not useful for fully embedding aligned inner values (Tool AI-ish). Maybe even apply [negative gradient steps](https://arxiv.org/abs/1511.05897) to "unlearn" things.

I think this idea is similar to Evan's [Precursor Checking](https://www.lesswrong.com/posts/nbq2bWLcYmSGup9aF/a-transparency-and-interpretability-tech-tree#5__Worst_case_training_process_transparency_for_non_deceptive_models) idea:

> **Precursor checking:** Another general type of training rationale that I think is worth calling attention to is what I’ll call “precursor checking,” which is the concept of using some method of gaining information about a model’s internals—e.g. transparency/interpretability or AI cognitive science—to check for some *precursor* to bad behavior rather than the bad behavior itself. This could involve substituting in some narrower, easier to check training goal—that still falls within the broader actual training goal—as the target for the training rationale. For example, if your training rationale involves ensuring that you don’t get [a deceptive model that’s actively trying to trick its training process](https://www.lesswrong.com/posts/ocWqg2Pf2br4jMmKA/does-sgd-produce-deceptive-alignment), then rather than explicitly trying to look for such deception (which could be especially hard since a deceptive model might actively try to avoid detection), you could instead try to ensure that your model has a short horizon length in terms of how far ahead its planning. Such a plan might work better, since horizon length might be easier to guarantee in a training rationale while still being consistent with the desired training goal and hopefully ruling out the possibility of deception. One issue with this sort of approach, however, is that you have to guarantee that whatever precursor for bad behavior you’re looking for is in fact a necessary condition for that bad behavior—if it turns out that there’s another way of getting that bad behavior that doesn’t go through the precursor, that could be a problem.

Counterarguments to this might be:

-   The model might not be able to have fully aligned inner values that remain robust as capabilities eventually generalize far out of distribution.
-   It will exceptionally difficult to know if we've actually removed this knowledge/capability from the model (even if it's possible).

Update: It seems that Jan Leike mentions something similar in his "[why I'm optimistic about our alignment approach](https://aligned.substack.com/p/alignment-optimism)" post.

> **The model can be “narrower.”** It doesn’t need to understand biology, physics, or human society that well. In practice we’d probably fine-tune from an LLM that does understand all of those things, but we could apply some targeted brain damage to the model as a safety precaution. More generally, the model only has to exceed human-level in a few domains, while it can be worse than humans in most others.

I'd be interested in hearing people's thoughts/criticisms on this. You can comment [here](https://www.lesswrong.com/posts/jXjeYYPXipAtA2zmj/jacquesthibs-s-shortform?commentId=QdcqqHp8d8LhLWqzh).

## Sources

Every external link in this piece that has a captured card, with what that page said
when it was captured. The quoted lines below are not the author of this piece writing:
they are the linked page describing itself, recorded by `bun run link-cards` on the date
given, and kept so that a reader still has them if the original moves or goes away.

- **The Core of the Alignment Problem is...** — Thomas Larsen, alignmentforum.org, 2022-08-17
  <https://alignmentforum.org/posts/vMM6HmSQaKmKadvBi/the-core-of-the-alignment-problem-is-1>
  Captured 2026-08-28.

  > Produced As Part Of The SERI ML Alignment Theory Scholars Program 2022 Under John Wentworth Introduction When trying to tackle a hard problem, a generally effective opening tactic is to Hold Off On Proposing Solutions: to fully discuss a problem and the different facets and aspects of it. This is intended to prevent…

- **Censoring Representations with an Adversary** — Harrison Edwards, Amos Storkey, arxiv.org, 2015-11-18
  <https://arxiv.org/abs/1511.05897>
  Captured 2026-08-28.

  > In practice, there are often explicit constraints on what representations or decisions are acceptable in an application of machine learning. For example it may be a legal requirement that a decision must not favour a particular group. Alternatively it can be that that representation of data must not have identifying…

- **A transparency and interpretability tech tree** — evhub, lesswrong.com, 2022-06-16
  <https://lesswrong.com/posts/nbq2bWLcYmSGup9aF/a-transparency-and-interpretability-tech-tree>
  Captured 2026-08-28.

  > Thanks to Chris Olah, Neel Nanda, Kate Woolverton, Richard Ngo, Buck Shlegeris, Daniel Kokotajlo, Kyle McDonell, Laria Reynolds, Eliezer Yudkowksy, Mark Xu, and James Lucassen for useful comments, conversations, and feedback that informed this post. The more I have thought about AI safety over the years, the more I…

- **Does SGD Produce Deceptive Alignment?** — Mark Xu, lesswrong.com, 2020-11-06
  <https://lesswrong.com/posts/ocWqg2Pf2br4jMmKA/does-sgd-produce-deceptive-alignment>
  Captured 2026-08-28.

  > Deceptive alignment was first introduced in Risks from Learned Optimization, which contained initial versions of the arguments discussed here. Additional arguments were discovered in this episode of the AI Alignment Podcast and in conversation with Evan Hubinger. Very little of this content is original. My…

- **Why I’m optimistic about our alignment approach** — Jan Leike, aligned.substack.com
  <https://aligned.substack.com/p/alignment-optimism>
  Captured 2026-08-28.

  > Some arguments in favor and responses to common objections

- **jacquesthibs's Shortform** — jacquesthibs, lesswrong.com, 2022-11-21
  <https://lesswrong.com/posts/jXjeYYPXipAtA2zmj/jacquesthibs-s-shortform?commentId=QdcqqHp8d8LhLWqzh>
  Captured 2026-08-28.

## Terms used

The author's own definitions for the glossary terms this piece uses. These are his words,
not a standard reference.

- **AGI** — Artificial general intelligence: a system with human level cognitive ability across domains rather than in one narrow task.
- **LLM** — Large language model: a neural network trained on very large amounts of text to predict what comes next.
- **MATS** — ML Alignment Theory Scholars: a mentorship programme that pairs new alignment researchers with established ones.
  See also: <https://jacquesthibodeau.com/ill-be-in-berkeley-for-seri-mats/>
- **out of distribution** — Describes an input drawn from a different distribution than the one a system was trained on. The term is relative to a training set, so on its own it says nothing about how far away the input is or whether the system will fail on it.
  Source: Koh and others, WILDS: A Benchmark of in-the-Wild Distribution Shifts <https://arxiv.org/abs/2012.07421>
  See also: <https://jacquesthibodeau.com/gaining-clarity-on-automated-alignment-research/>
- **SERI** — Stanford Existential Risks Initiative, which ran the research programme that became MATS.
  See also: <https://jacquesthibodeau.com/ill-be-in-berkeley-for-seri-mats/>