--- title: "AI Alignment Project Ideas" description: "Seven project write-ups for ARENA and LASR participants, from improving MAIA to reward misspecification and the cases where a model gets the right answer from bad reasoning." published: 2024-10-02 tags: ["Automating alignment research"] importance: 5 docStatus: "notes" audio: "https://pub-4ee2f71bc29541a7a6e8d9694f0a1b21.r2.dev/66fd6f485c58d327551e4625/audio.mp3" author: "Jacques Thibodeau" canonical: "https://jacquesthibodeau.com/ai-alignment-project-ideas-oct-2-2024/" --- \[Nov 27, 2024\] I have some new [alignment project ideas](https://docs.google.com/spreadsheets/d/1iyS1zryH3QAc7MWyjws8zA3mkfqB_4iNqtkbD9UTnZM/edit?usp=sharing) I quickly wrote up. These are mostly projects I'd be happy to mentor people on. I would love feedback on the ideas. If you are interested in collaborating on any of them, that's cool, too. \[Oct 2, 2024\] I quickly wrote up some rough project ideas for [ARENA](https://www.arena.education/) and [LASR](https://www.lasrlabs.org/) participants, so I figured I'd share them here as well. I am happy to discuss these ideas and potentially collaborate on some of them.
Collaboration welcome

I am happy to discuss these ideas and potentially collaborate on some of them. If you are interested, please reach out.

## 1\. Improving "A Multimodal Automated Interpretability Agent" (MAIA) ### Overview [MAIA (Multimodal Automated Interpretability Agent)](https://multimodal-interpretability.csail.mit.edu/maia/) is a system designed to help users understand AI models by combining human-like experimentation flexibility with automated scalability. It answers user queries about AI system components by iteratively generating hypotheses, designing and running experiments, observing outcomes, and updating hypotheses. MAIA uses a vision-language model (GPT-4V, at the time) backbone equipped with an API of interpretability experiment tools. This modular system can address both "macroscopic" questions (e.g., identifying systematic biases in model predictions) and "microscopic" questions (e.g., describing individual features) with simple query modifications. This project aims to improve MAIA's ability to either answer macroscopic questions or microscopic questions on vision models. ## 2\. Making "A Multimodal Automated Interpretability Agent" (MAIA) work with LLMs MAIA is focused on vision models, so this project aims to create a MAIA-like setup, but for the interpretability of LLMs. Given that this would require creating a new setup for language models, it would make sense to come up with simple interpretability benchmark examples to test MAIA-LLM. The easiest way to do this would be to either look for existing LLM interpretability benchmarks or create one based on interpretability results we've already verified (would be ideal to have a ground truth). Ideally, the examples in the benchmark would be simple, but new enough that the LLM has not seen them in its training data. ## 3\. Testing the robustness of Critique-out-Loud Reward (CLoud) Models [Critique-out-Loud reward models](https://github.com/zankner/CLoud) are reward models that can reason explicitly about the quality of an input through producing Chain-of-Thought like critiques of an input before predicting a reward. In classic reward model training, the reward model is trained as a reward head initialized on top of the base LLM. Without LM capabilities, classic reward models act as encoders and must predict rewards within a single forward pass through the model, meaning reasoning must happen implicitly. In contrast, CLoud reward models are trained to both produce explicit reasoning about quality and to score based on these critique reasoning traces. CLoud reward models lead to large gains for pairwise preference modeling on RewardBench, and also lead to large gains in win rate when used as the scoring model in Best-of-N sampling on ArenaHard. The goal for this project would be to test the robustness of CLoud reward models. For example, are the CLoud RMs (discriminators) more robust to jailbreaking attacks from the policy (generator)? Do the CLoud RMs generalize better? From an alignment perspective, we would want RMs that generalize further out-of-distribution (and ideally, always more than the generator we are training). ## 4\. Synthetic Data for Behavioural Interventions [Simple synthetic data reduces sycophancy in large language models](https://arxiv.org/abs/2308.03958) by (Google) reduced sycophancy in LLMs with a fairly small number of synthetic data examples. This project would involve testing this technique for other behavioural interventions and (potentially) studying the scaling laws. Consider looking at the examples from the Model-Written Evaluations paper by Anthropic to find some behaviours to test. ## 5\. Regularization Techniques for Enhancing Interpretability and Editability Explore the effectiveness of different regularization techniques (e.g. L1 regularization, weight pruning, activation sparsity) in improving the interpretability and/or editability of language models, and assess their impact on model performance and alignment. We expect we could apply automated interpretability methods (e.g. MAIA) to this project to test how well the different regularization techniques impact the model. In some sense, this research is similar to the work Anthropic did with SoLU activation functions. Unfortunately, they needed to add layer norms to make the SoLU models competitive, which seems to have hidden away the superposition in other parts of the network, making SoLU unhelpful in making the models more interpretable That said, we hope to find that we can increase our ability to interpret these models through regularization techniques. A technique like L1 regularization should help because it encourages the model to learn sparse representations by penalizing non-zero weights or activations. Sparse models tend to be more interpretable as they rely on a smaller set of important features. Methodology: 1. Identify a set of regularization techniques (e.g., L1 regularization, weight pruning, activation sparsity) to be applied during fine-tuning. 2. Fine-tune pre-trained language models with different regularization techniques and hyperparameters. 3. Evaluate the fine-tuned models using interpretability tools (e.g., attention visualization, probing classifiers) and editability benchmarks (e.g., ROME). 4. Analyze the impact of regularization on model interpretability, editability, and performance. 5. Investigate the relationship between interpretability, editability, and model alignment. Expected Outcomes: - Quantitative assessment of the effectiveness of different regularization techniques for improving interpretability and editability. - Insights into the trade-offs between interpretability, editability, and model performance. - Recommendations for regularization techniques that enhance interpretability and editability while maintaining model performance and alignment. ## 6\. Quantifying the Impact of Reward Misspecification on Language Model Behavior Investigate how misspecified reward functions influence the behavior of language models during fine-tuning and measure the extent to which the model's outputs are steered by the reward labels, even when they contradict the input context. We hope to better understand language model training dynamics. Additionally, we expect online learning to complicate things in the future, where models will be able to generate the data they may eventually be trained on. We hope that insights from this work can help us prevent catastrophic feedback loops in the future. For example, if model behavior is mostly impacted by training data, we may prefer to shape model behavior through synthetic data (it has been shown we can reduce sycophancy by doing this). Prior works: - [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://arxiv.org/abs/2201.03544) by Alexander Pan, Kush Bhatia, Jacob Steinhardt - [Survival Instinct in Offline Reinforcement Learning](https://arxiv.org/abs/2306.03286) by Anqi Li, Dipendra Misra, Andrey Kolobov, Ching-An Cheng - [Simple synthetic data reduces sycophancy in large language models](https://arxiv.org/abs/2308.03958) by (Google), Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, Quoc V. Le - [Scaling Laws for Reward Model Overoptimization](https://arxiv.org/abs/2210.10760) by (OpenAI), Leo Gao, John Schulman, Jacob Hilton - [On the Sensitivity of Reward Inference to Misspecified Human Models](https://arxiv.org/abs/2212.04717) by [Joey Hong](https://arxiv.org/search/cs?searchtype=author&query=Hong,+J), [Kush Bhatia](https://arxiv.org/search/cs?searchtype=author&query=Bhatia,+K), [Anca Dragan](https://arxiv.org/search/cs?searchtype=author&query=Dragan,+A) Methodology: 1. Create a diverse dataset of text passages with candidate responses and manually label them with coherence and misspecified rewards. 2. Fine-tune pre-trained language models using different reward weighting schemes and hyperparameters. 3. Evaluate the generated responses using automated metrics and human judgments for coherence and misspecification alignment. 4. Analyze the influence of misspecified rewards on model behavior and the trade-offs between coherence and misspecification alignment. 5. Use interpretability techniques to understand how misspecified rewards affect the model's internal representations and decision-making process. Expected Outcomes: - Quantitative measurements of the impact of reward misspecification on language model behavior. - Insights into the trade-offs between coherence and misspecification alignment. - Interpretability analysis revealing the effects of misspecified rewards on the model's internal representations. ## 7\. Investigating Wrong Reasoning for Correct Answers Understand the underlying mechanisms that lead to language models producing correct answers through flawed reasoning, and develop techniques to detect and mitigate such behavior. Essentially, we want to apply interpretability techniques to help us identify which sets of activations or token-layer pairs impact the model getting the correct answer when it has the correct reasoning versus when it has the incorrect reasoning. The hope is to uncover systematic differences as to when it is not relying on its chain-of-thought at all and when it does leverage its chain-of-thought to get the correct answer. \[EDIT Oct 2nd, 2024\] This project intends to follow a similar line of reasoning as described in [this post](https://www.lesswrong.com/posts/HQyWGE2BummDCc2Cx/the-case-for-cot-unfaithfulness-is-overstated) and [this comment](https://www.lesswrong.com/posts/HQyWGE2BummDCc2Cx/the-case-for-cot-unfaithfulness-is-overstated?commentId=po6kmGDtTvzhodNJo). The goal is to study chains-of-thought and improve faithfulness without suffering an alignment tax so that we can have highly interpretable systems through their token outputs and prevent loss of control. The project doesn't necessarily need to rely only on model internals. Related work: 1. [Decomposing Predictions by Modeling Model Computation](https://gradientscience.org/modelcomponents/) by Harshay Shah, Andrew Ilyas, Aleksander Madry 2. [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://openreview.net/forum?id=EldbUlZtbd) by Peter Hase, Mohit Bansal, Been Kim, Asma Ghandeharioun 3. [On Measuring Faithfulness or Self-consistency of Natural Language Explanations](https://arxiv.org/abs/2311.07466) by [Letitia Parcalabescu](https://arxiv.org/search/cs?searchtype=author&query=Parcalabescu,+L), [Anette Frank](https://arxiv.org/search/cs?searchtype=author&query=Frank,+A) 4. [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://arxiv.org/abs/2305.04388) by Miles Turpin, Julian Michael, Ethan Perez, Samuel R. Bowman 5. [Measuring Faithfulness in Chain-of-Thought Reasoning](https://arxiv.org/abs/2307.13702) by Tamera Lanham et al. Methodology: 1. Curate a dataset of questions and answers where language models are known to provide correct answers but with flawed reasoning. 2. Use interpretability tools (e.g., attention visualization, probing classifiers) to analyze the model's internal representations and decision-making process for these examples. 3. Develop metrics and techniques to detect instances of correct answers with flawed reasoning. 4. Investigate the relationship between model size, training data, and the prevalence of flawed reasoning. 5. Propose and evaluate mitigation strategies, such as data augmentation or targeted fine-tuning, to reduce the occurrence of flawed reasoning. Expected Outcomes: - Insights into the underlying mechanisms that lead to correct answers with flawed reasoning in language models. - Metrics and techniques for detecting instances of flawed reasoning. - Empirical analysis of the factors contributing to flawed reasoning, such as model size and training data. - Proposed mitigation strategies to reduce the occurrence of flawed reasoning and improve model alignment. ## Sources Every external link in this piece that has a captured card, with what that page said when it was captured. The quoted lines below are not the author of this piece writing: they are the linked page describing itself, recorded by `bun run link-cards` on the date given, and kept so that a reader still has them if the original moves or goes away. - **Alignment Project Ideas** — docs.google.com Captured 2026-08-28. - **ARENA – AI Safety Education** — arena.education Captured 2026-08-28. > We aim to provide talented individuals with the skills, community, and confidence to contribute directly to technical AI safety. - **LASR Labs** — lasrlabs.org Captured 2026-08-28. > Join a 13-week research programme and write a technical AI safety paper in a small team with supervision from an experienced researcher. Work full time from the LISA offices in London alongside AI Safety organisations and programmes. - **MAIA** — multimodal-interpretability.csail.mit.edu Captured 2026-08-28. > A Multimodal Automated Interpretability Agent that autonomously conducts experiments on other systems to explain their behavior. - **zankner/CLoud** — zankner, github.com Captured 2026-08-28. > Critique-out-Loud Reward Models - **Simple synthetic data reduces sycophancy in large language models** — Jerry Wei and 4 others, arxiv.org, 2023-08-07 Captured 2026-08-28. > Sycophancy is an undesirable behavior where models tailor their responses to follow a human user's view even when that view is not objectively correct (e.g., adapting liberal views once a user reveals that they are liberal). In this paper, we study the prevalence of sycophancy in language models and propose a simple… - **The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models** — Alexander Pan, Kush Bhatia, Jacob Steinhardt, arxiv.org, 2022-01-10 Captured 2026-08-28. > Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. To understand how reward hacking arises, we construct four RL environments with misspecified rewards. We investigate reward hacking as a function of agent capabilities: model… - **Survival Instinct in Offline Reinforcement Learning** — Anqi Li and 3 others, arxiv.org, 2023-06-05 Captured 2026-08-28. > We present a novel observation about the behavior of offline reinforcement learning (RL) algorithms: on many benchmark datasets, offline RL can produce well-performing and safe policies even when trained with "wrong" reward labels, such as those that are zero everywhere or are negatives of the true rewards. This… - **Scaling Laws for Reward Model Overoptimization** — Leo Gao, John Schulman, Jacob Hilton, arxiv.org, 2022-10-19 Captured 2026-08-28. > In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law. This effect has been frequently… - **On the Sensitivity of Reward Inference to Misspecified Human Models** — Joey Hong, Kush Bhatia, Anca Dragan, arxiv.org, 2022-12-09 Captured 2026-08-28. > Inferring reward functions from human behavior is at the center of value alignment - aligning AI objectives with what we, humans, actually want. But doing so relies on models of how humans behave given their objectives. After decades of research in cognitive science, neuroscience, and behavioral economics, obtaining… - **the case for CoT unfaithfulness is overstated** — nostalgebraist, lesswrong.com, 2024-09-29 Captured 2026-08-28. > [Quickly written, unpolished. Also, it's possible that there's some more convincing work on this topic that I'm unaware of – if so, let me know. Also also, it's possible I'm arguing with an imaginary position here and everyone already agrees with everything below.] In research discussions about LLMs, I often pick up a… - **the case for CoT unfaithfulness is overstated** — nostalgebraist, lesswrong.com, 2024-09-29 Captured 2026-08-28. > [Quickly written, unpolished. Also, it's possible that there's some more convincing work on this topic that I'm unaware of – if so, let me know. Also also, it's possible I'm arguing with an imaginary position here and everyone already agrees with everything below.] In research discussions about LLMs, I often pick up a… - **Decomposing Predictions by Modeling Model Computation** — Madry Lab, gradientscience.org Captured 2026-08-28. > Research highlights and perspectives on machine learning and optimization from MadryLab. - **Verifying your browser | OpenReview** — openreview.net Captured 2026-08-28. - **On Measuring Faithfulness or Self-consistency of Natural Language Explanations** — Letitia Parcalabescu, Anette Frank, arxiv.org, 2023-11-13 Captured 2026-08-28. > Large language models (LLMs) can explain their predictions through post-hoc or Chain-of-Thought (CoT) explanations. But an LLM could make up reasonably sounding explanations that are unfaithful to its underlying reasoning. Recent work has designed tests that aim to judge the faithfulness of post-hoc or CoT… - **Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting** — Miles Turpin and 3 others, arxiv.org, 2023-05-07 Captured 2026-08-28. > Large Language Models (LLMs) can achieve strong performance on many tasks by producing step-by-step reasoning before giving a final output, often referred to as chain-of-thought reasoning (CoT). It is tempting to interpret these CoT explanations as the LLM's process for solving a task. This level of transparency into… - **Measuring Faithfulness in Chain-of-Thought Reasoning** — Tamera Lanham and 29 others, arxiv.org, 2023-07-17 Captured 2026-08-28. > Large language models (LLMs) perform better when they produce step-by-step, "Chain-of-Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for… ## Terms used The author's own definitions for the glossary terms this piece uses. These are his words, not a standard reference. - **API** — Application programming interface: the contract one program uses to call another. - **ARENA** — Alignment Research Engineer Accelerator: a training programme that teaches the engineering side of alignment research. See also: - **LASR** — London AI Safety Research: a research programme running collaborative alignment projects out of London. See also: - **LLM** — Large language model: a neural network trained on very large amounts of text to predict what comes next. - **MAIA** — Multimodal Automated Interpretability Agent: a system that runs interpretability experiments on another model on its own. See also: - **out of distribution** — Describes an input drawn from a different distribution than the one a system was trained on. The term is relative to a training set, so on its own it says nothing about how far away the input is or whether the system will fail on it. Source: Koh and others, WILDS: A Benchmark of in-the-Wild Distribution Shifts See also: - **ROME** — Rank One Model Editing: a technique that edits a single factual association in a language model by changing one weight matrix. See also: