---
title: "But is it really in Rome? Limitations of the ROME model editing technique"
description: "The ROME edit does not generalise the way you would expect. It runs in one direction only, and cheese and fromage have to be edited separately."
published: 2022-12-30
tags: ["Interpretability and model internals"]
importance: 7
confidence: "likely"
docStatus: "finished"
audio: "https://pub-4ee2f71bc29541a7a6e8d9694f0a1b21.r2.dev/63ae4ffa3de6b56ce4c0c922/audio.mp3"
crosspost:
  lesswrong: "https://www.lesswrong.com/posts/QL7J9wmS6W2fWpofd/but-is-it-really-in-rome-an-investigation-of-the-rome-model"
author: "Jacques Thibodeau"
canonical: "https://jacquesthibodeau.com/but-is-it-really-in-rome-limitations-of-the-rome-model-editing-technique/"
---
I just published a [new post](https://www.lesswrong.com/posts/QL7J9wmS6W2fWpofd/but-is-it-really-in-rome-limitations-of-the-rome-model) on LessWrong. It's about the causal tracing and model editing paper (ROME).

Here's the intro:

The majority of this work was carried out this summer. Many people in the community were surprised when I mentioned some of the limitations of ROME (Rank-One Model Editing), so I figured it was worth it to write a post about it as well as other insights I gained from looking into the paper. Most tests were done with GPT-2, some were done with GPT-J.

The ROME paper has been one of the most influential papers in the prosaic alignment community. It has several important insights. The main findings are:

1.  Factual associations such as “*The Eiffel Tower* is in *Paris*” seem to be stored in the MLPs of the early-middle layers of a GPT model. As the Tower token passes through the network, the MLPs of the early-middle layers will *write* information (e.g. the Eiffel Tower’s location) into the residual so that the model can later *read* that information to generate a token about that fact (e.g. Paris).
2.  Editing/updating the MLP of a single layer for a given (subject, relationship, object) association allows the model to generate text with the updated fact when using new prompts/sentences that include the subject tokens. For example, editing “**The Eiffel Tower is in** Paris Rome” results in a model that outputs “**The Eiffel Tower is** **right across from** St Peter’s Basilica in Rome, Italy. “

In this post, I show that the ROME edit has many limitations:

-   The ROME edit doesn’t *generalize* in the way you might expect. It’s true that if the subject tokens you use for the edit are found in the prompt, it will try to generalize from the updated fact. However, it doesn’t “generalize” in the following ways:
-   It is **not direction-agnostic/bidirectional**. For example, the ROME edit is only in the "Eiffel Tower is located in \_\_\_\_" direction, not in the "Rome has a tower called the \_\_\_\_" direction.
-   It’s mostly (?) the **token association being edited, not the concept**. “Cheese” and “Fromage” are separate things, you’d need to edit both.
-   I hoped that if you edit X (e.g. The Rock) and then tried to describe X without using the token, the model would realize it’s talking about X and generate according to the edit. Based on the examples I tested, this does not seem to be the case. **You *mostly* need the subject tokens that were used for the edit in the prompt.**
-   It seems to **over/under-optimize** depending on the new fact. It will want to talk about Rome (post-edit) when the Eiffel Tower is mentioned more than it will want to talk about Paris before the edit.

<!--kg-card-begin: html-->
<div class="admonition warning">
<div class="admonition-title">Correlation, not causation</div>
<p>One point I want to illustrate with this post is that the intervention is a bit more finicky than one might initially think, and someone could infer too much from the results in the paper. With a lot of these interpretability techniques, we end up <a href="https://www.lesswrong.com/posts/FDjTgDcGPc7B98AES/searching-for-search-4#Failure_modes">finding correlation rather than causation</a>. However, my hope is that such interventions, <a href="https://www.lesswrong.com/posts/q5Gox77ReFAy5i2YQ/in-defense-of-probably-wrong-mechanistic-models">while not perfect at validating hypotheses</a>, will hopefully give us extra confidence in our interpretability results (in this case, the <em>causal tracing</em> method).</p>
</div>
<!--kg-card-end: html-->

<!--kg-card-begin: html-->
<div class="admonition note">
<div class="admonition-title">Related posts</div>
<p>This post is part of a series on the ROME paper and factual knowledge in Transformers. See also: <a href="https://jacquesthibodeau.com/detail-about-the-rome-paper-ive-been-thinking-about/">A Detail About the ROME Paper I've Been Thinking About</a></p>
</div>
<!--kg-card-end: html-->

## Sources

Every external link in this piece that has a captured card, with what that page said
when it was captured. The quoted lines below are not the author of this piece writing:
they are the linked page describing itself, recorded by `bun run link-cards` on the date
given, and kept so that a reader still has them if the original moves or goes away.

- **But is it really in Rome? An investigation of the ROME model editing technique** — jacquesthibs, lesswrong.com, 2022-12-30
  <https://lesswrong.com/posts/QL7J9wmS6W2fWpofd/but-is-it-really-in-rome-limitations-of-the-rome-model>
  Captured 2026-08-28.

  > Thanks to Andrei Alexandru, Joe Collman, Michael Einhorn, Kyle McDonell, Daniel Paleka, and Neel Nanda for feedback on drafts and/or conversations which led to useful insights for this work. In addition, thank you to both William Saunders and Alex Gray for exceptional mentorship throughout this project. The majority…

- **Searching for Search** — Niki Dupuis, lesswrong.com, 2022-11-28
  <https://lesswrong.com/posts/FDjTgDcGPc7B98AES/searching-for-search-4>
  Captured 2026-08-28.

  > Thanks to Dan Braun, Ze Shen Chin, Paul Colognese, Michael Ivanitskiy, Sudhanshu Kasewa, and Lucas Teixeira for feedback on drafts. This work was carried out while at Conjecture. This post is a loosely structured collection of thoughts and confusions about search and mesaoptimization and how to look for them in…

- **In defense of probably wrong mechanistic models** — evhub, lesswrong.com, 2022-12-06
  <https://lesswrong.com/posts/q5Gox77ReFAy5i2YQ/in-defense-of-probably-wrong-mechanistic-models>
  Captured 2026-08-28.

  > This is a short post on a simple point that I get asked about a lot and want a canonical reference for. Which of the following two options is more likely to be true? 1. AIs will internally be running explicit search processes. 2. AIs will internally be doing something weirder and more complicated than explicit search.…

## Terms used

The author's own definitions for the glossary terms this piece uses. These are his words,
not a standard reference.

- **GPT** — Generative pre trained transformer: the family of decoder only language models OpenAI named this way.
- **MLP** — Multi layer perceptron: the fully connected feedforward block that sits between attention layers in a transformer.
  See also: <https://jacquesthibodeau.com/detail-about-the-rome-paper-ive-been-thinking-about/>
- **ROME** — Rank One Model Editing: a technique that edits a single factual association in a language model by changing one weight matrix.
  See also: <https://jacquesthibodeau.com/but-is-it-really-in-rome-limitations-of-the-rome-model-editing-technique/>