---
title: "Is the \"Valley of Confused Abstractions\" real?"
description: "Chris Olah's curve says models get harder to read before they get easier. Neel told me that came out of vision models, so I am posting my confusion."
published: 2022-12-05
tags: ["Capabilities and generalisation", "Interpretability and model internals"]
importance: 4
confidence: "possible"
docStatus: "notes"
cover: "https://jacquesthibodeau.com/content/images/2022/12/model-interp-strength-3.jpg"
coverAlt: "Chris Olah’s interpretability curve with handwritten red annotations: bag of words at the left peak, a tiny 1-layer transformer and GPT-2 in the Valley of Confused Abstractions, AlexNet and GPT-3 on the climb, and “which abstractions are alien? all?” at the right."
audio: "https://pub-4ee2f71bc29541a7a6e8d9694f0a1b21.r2.dev/638e25c83de6b56ce4c0c891/audio.mp3"
crosspost:
lesswrong: "https://www.lesswrong.com/posts/g7rLyjg67iopg9zLD/is-the-valley-of-confused-abstractions-real"
figures:
- src: "https://jacquesthibodeau.com/content/images/external/dd22c8a0-2ad2dbb1a3b6013c3959171d4ccfe534f387835b4c2a186e.png"
alt: "Chris Olah’s diagram of model interpretability against model strength: the curve starts high for simple models, sinks through the Valley of Confused Abstractions, peaks at crisp abstractions just past human performance, then falls away into increasingly alien abstractions."
width: 1878
height: 834
viewer: "inspector"
- src: "https://jacquesthibodeau.com/content/images/external/c574240b-e1f6580188246e025b8bccbf6f118322cbd329601dbb5d35.jpg"
alt: "The same curve with handwritten red annotations: bag of words at the left peak, a tiny 1-layer transformer and GPT-2 in the valley, AlexNet and GPT-3 on the climb, InceptionV1 near the top, and “which abstractions are alien? all?” at the right."
width: 1878
height: 834
viewer: "inspector"
author: "Jacques Thibodeau"
canonical: "https://jacquesthibodeau.com/is-the-valley-of-confused-abstractions-real/"
---
Epistemic status
Quite confused. Using this short post as a signal for discussion.
[Here](https://www.lesswrong.com/posts/g7rLyjg67iopg9zLD/is-the-valley-of-confused-abstractions-real)'s a link to the LessWrong post for discussion.
In Evan's post about [Chris Olah's views on AGI safety](https://www.alignmentforum.org/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety#What_if_interpretability_breaks_down_as_AI_gets_more_powerful_), there is a diagram which loosely points to how Chris thinks model interpretability will be impacted at different levels of model "strength":
I always thought this diagram still held up in the LLM regime, even though it seems the diagram pointed specifically to interpretability with CNN vision models. However, I had a brief exchange with Neel Nanda about the Valley of Confused Abstractions in the context of language models, and I thought this might be a good thing to revisit.
I've been imagining that language models with the "strength" of GPT-2 are somewhere near the bottom of the Valley of Confused Abstractions, but the much bigger models are a bit further along the model strength axis (though I'm not sure where they fall). I've been thinking about this in the context of model editing or pointing/retargeting the model.
Here's my exchange with Neel:
> **Neel**: I'm personally not actually that convinced that the valley of confused abstractions is real, at least in language. 1L transformers are easier to interpret than large ones, which is completely unlike images!
> **Me**: Does that not fit with the model interpretability diagram from: [https://www.alignmentforum.org/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety#What\_if\_interpretability\_breaks\_down\_as\_AI\_gets\_more\_powerful\_](https://www.alignmentforum.org/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety#What_if_interpretability_breaks_down_as_AI_gets_more_powerful_)?
> Meaning that abstractions are easy enough to find with simple models, but as you scale them up you have to cross the valley of confused abstraction before you get to "crisp abstractions." Are you saying we are close to "human-level" in some domains and it's still hard to find abstractions? My assumption was that we simply have not passed the valley yet so yes larger models will be harder to interpret. Maybe I'm misunderstanding?
> **Neel**: Oh, I just don't think that diagram is correct. Chris found that tiny image models are really hard to interpret, but we later found that tiny transformers are fairly easy.
> **Me**: I guess this is what I'm having a hard time understanding: The diagram seems to imply that tiny (simple?) models are easy to interpret. In the example in the text, my assumption was that AlexNet was just closer to the bottom of the valley than InceptionV1. But you could have even simpler models than AlexNet that would be more interpretable but less powerful?
> **Neel**: Ah, I think the diagram is different for image convnets and for language transformers.
My understanding was that 1-layer transformers being easy to interpret still agrees with the diagram, and current big-boy models are just not past the Valley of Confused Abstractions yet.
Ok, but if what Neel says is true, what might the diagram look like for language transformers?
I'm confused at the moment, but my thinking used to go something like this: in the case of GPT-2, it is trying to make sense of all the data it has been trained on but just isn't big enough to fully grasp the concept of "cheese" and "fromage" are essentially the same. But my expectation is that as the model gets bigger, it *knows* those two tokens mean the same thing, just in different languages. Maybe it does?
With that line of thinking, as model strength increases, it will help the model create crisper internal abstractions of a concept like "cheese." But then what...at some point, the model gets too powerful, and it becomes too hard to pull out the "cheese/fromage" abstraction?
Anyway, I hoped that as LLMs (trained with the transformers architecture) increase in "strength" beyond the current models, the abstractions become crisper, and it becomes much easier to identify abstractions as it gets closer to some "human-level performance." However, GPT-3 is already superhuman in some respects, so I'm unsure about how to disentangle this. I hope this post sparks some good conversation about how to de-confuse this and how the diagram should look for LLM transformers. Is it just some negative exponential with respect to model strength? Or does it humps like the original diagram?
## Sources
Every external link in this piece that has a captured card, with what that page said
when it was captured. The quoted lines below are not the author of this piece writing:
they are the linked page describing itself, recorded by `bun run link-cards` on the date
given, and kept so that a reader still has them if the original moves or goes away.
- **Is the "Valley of Confused Abstractions" real?** — jacquesthibs, lesswrong.com, 2022-12-05
Captured 2026-08-28.
> Epistemic Status: Quite confused. Using this short post as a signal for discussion. In Evan's post about Chris Olah's views on AGI safety, there is a diagram which loosely points to how Chris thinks model interpretability will be impacted at different levels of model "strength" (note that this is vague intuition;…
- **Chris Olah’s views on AGI safety** — evhub, alignmentforum.org, 2019-11-01
Captured 2026-08-28.
> Note: I am not Chris Olah. This post was the result of lots of back-and-forth with Chris, but everything here is my interpretation of what Chris believes, not necessarily what he actually believes. Chris also wanted me to emphasize that his thinking is informed by all of his colleagues on the OpenAI Clarity team and…
## Terms used
The author's own definitions for the glossary terms this piece uses. These are his words,
not a standard reference.
- **AGI** — Artificial general intelligence: a system with human level cognitive ability across domains rather than in one narrow task.
- **CNN** — Convolutional neural network: the architecture that dominated image recognition before transformers.
See also:
- **LLM** — Large language model: a neural network trained on very large amounts of text to predict what comes next.