QED Research

What are you working on?

I am building a non-profit research organization focused on identifying consequential cruxes in the transition to powerful AI, making sense of them, and generating evidence that would deconfuse key stakeholders (e.g., AI lab employees, AI safety researchers, grantmakers, policymakers, etc.) and enable higher-quality, high-stakes decisions.

Given the increasing complexity of the world driven by AI, we need organizations with the capacity to make sense of it all as we face challenges where even experts disagree on the details (e.g., the recent HuggingFace incident).

The organization will conduct the following types of work:

• Providing reports, podcasts, and short posts which serve the purpose of deconfusing thorny/cruxy scenarios with respect to AI safety. For example:
  • Making sense of the recent OpenAI incidents from a purely alignment perspective. What exactly do swarms change about the problem, if anything?
  • Reframing core alignment theory in a modern-era framing to help post-ChatGPT researchers become exposed to the ideas in a digestible format.
• Conducting technical AI safety research we believe will answer questions that are critical for decision-making (e.g. research prioritization, grantmaking, governance).
• Consolidating research projects that should be run inside the labs into executable specs. We expect outside researchers to get less and less access to the most capable models, so we will prepare the spec and ask a lab: can you run this on your most capable model?
• Host events and workshops to identify and operationalize cruxes for new or consolidated research.
  • Our intention is to design this process to improve coordination and collaboration among all stakeholders.
• Public list of open questions in AI safety for researchers to collaborate on.
• Scenario planning similar to the AI Futures Project, though we would likely develop our own methodology.
  • Note that I (Jacques) previously wrote in 2022 about strategic foresight as a method for identifying golden opportunities and taking advantage of future plausible scenarios by preparing in advance and sharing scenario planning reports.

We support the transparency of AI2040’s Plan A, but transparency can still lead to fundamental disagreements and different interpretations of experimental results. We want to position our org to manage these cruxes and conflicts more effectively. Ultimately, we expect that despite best efforts, some of these disagreements will not be resolved even with new tech solutions or sufficient evidence. That said, decision-makers will still make decisions, and our goal is to positively influence those key decisions even if we haven’t “solved” debate.

Our intention is to be laser-focused on presenting work designed to help key decision-makers make better decisions. Each piece is written for a specific audience, e.g., technical researchers inside labs, grantmakers, or government, rather than for a general reader. Examples include:

• Weighing the specific dangers of RL in AI training.
• There are researchers who believe Constitutional AI may be enough to robustly align superintelligence. We would articulate the crux of the disagreement: which part is values, which is risk tolerance, which is unsettled science, and which is one side missing context the other has (or something else, like a belief that governments will never go along with a proposal). Then we would list the experiments or other evidence needed to clarify the debate.
• Does Chain-of-Thought lead to more misalignment despite the added transparency? Is there an ideal balance?
• Providing clarity on LLM capability spikiness and its relation to AI safety.
• Why we might expect Automated Alignment research to go wrong.
• Making sense of the recent OpenAI/HuggingFace, Anthropic, and UK AISI incidents. Note that third parties are already being brought into this work: METR and Redwood Research ran an unpaid independent investigation of the incident and called it a precedent for independent investigation of misalignment incidents, and AVERI ran a double-blind third-party evaluation with Google DeepMind. Our reports build on that third-party role.
• Maintain a coherent go-to intro to AI safety evidence and arguments, written with capabilities researchers at frontier labs in mind.
  • We believe that more capabilities researchers will be shaken up by misalignment incidents in internal deployments. As a result, it seems prudent to prepare material we can share with them to help them make sense of the misalignment scenarios and perhaps steer them towards better safety work (or pushing for international treaties). We can share these with AI safety champions at frontier labs who are looking to provide such references to colleagues (who may have even asked for them).
• Collaborate with stakeholders to create well-evidenced, expert-endorsed “proto-standards” on specific frontier AI issues (as suggested by GovAI or requested here).

While we intend to initially prioritize questions relevant to technical safety research, we may also seek to cover cruxes related to:

• Autonomous AI weapons.
• Gaining better clarity on the public’s sentiments. As noted by Jasmine Sun, failing to account for the anti-corporate and class elements in anti-AI sentiment may lead companies and (more importantly) safety advocates to focus on the wrong things.
• Thought-to-Text.
• Mind uploading.
• Geopolitical considerations w.r.t. countries like China.
  • Note that I will be visiting China in October and would like to host a workshop/event that covers topics related to AI2040 and uncovers underlying assumptions held by the West and China.

What would you do with funding?

Base ask: $290,000 for 12 months. The Base ask will allow me to start treating this as an organization, though I would likely seek considerably more funding in 3-6 months to fund co-founders/hires.

Output:

• A series of blog posts and reports that point to specific cruxes and provide coherent explanations for disagreements among researchers, which would be helpful for all AI safety stakeholders. This includes many of the examples described in the previous section.
• Communication of key decision cruxes with consolidated evidence from research and other sources.
• Conduct a complete strategic foresight report on plausible scenarios.
• Collaboration with AI safety researchers to conduct experiments and follow-up research to answer questions and gather evidence for the cruxes.

My salary is $100,000. I’m Montreal-based (for now), though I expect to spend months in the Bay Area, and I’d rather the delta fund the organization. Travel and extended stays in SF and London run up to a month at a time, for collaboration and workshops.

Minimum viable: $125,000 (salary $75,000; fiscal/legal/ops $25,000; AI compute $15,000; contingency $10,000). The research and writing continue; workshops and travelling will have to wait.

Ideal ask: $700,000.

• +$200,000 to bring on a senior technical co-lead (which I would pay more than myself to hire someone great).
• +$40,000 to audit the AI Futures Project and host a scenarios workshop.
• +$150,000 to hire a technical AI safety researcher to conduct research that would inform our reports and the key decisions (e.g., a deep study into Constitutional AI and RL, studying where capabilities are coming from in LLMs and what this means for AI progress). The bigger the org, the more labour we will put towards tackling specific cruxes and important problems in ASI alignment.
• +$20,000 for additional operations.

Note that I am asking for less because I am trying to get this organization off the ground, and the reduced costs are largely due to salary cuts. Other comparable organizations pay senior staff more, so additional funding would largely make me more comfortable financially (I have taken considerable pay reductions as an independent AI safety researcher over the last few years), allow me to hire more staff, and potentially hire (or co-found with) more prestigious staff.

My aim would be to build the founding team in the first 6 months and then start hiring staff to work across all our pillars (which can each include different sub-areas in the lead-up to the singularity):

• Working with stakeholders to identify cruxes and write reports.
• Conduct research based on what we believe would move the needle for key decision-makers.

Currently $0 in funding. I will start reaching out to funders outside Lightcone Commons. For my previous work, I’ve received funding from LTFF and Lightspeed Grants (as an independent researcher), and Manifund (for my previous startup).

Who is involved?

Jacques Thibodeau. GitHub, LessWrong profile, LinkedIn, Website, X/Twitter, and Resume.

In the last year, I attempted to build an AI safety startup. I pivoted a lot (automated research for safety, monitoring, model drift, RL environments, data curation for robust reward models, formal verification / Spec IDE, and more) but ultimately decided not to pursue them further because I was unconvinced that the different paths would have been beneficial for superalignment (despite having investor interest). During that period I went through the Catalyze Impact incubator and Fifty Years’ 5050 program.

The main relevant output from this time is a desktop app I built that helps researchers clarify research ideas so AIs work on the right things, including a writing tool to draft research and get comprehensive reviews from an LLM pipeline. I’m now looking to make it specifically useful for AI safety; I added open questions, research agendas, codebase templates, etc.

As an Independent Alignment Researcher, I spent time investigating automated alignment research (including last year at PIBBSS), automated unsupervised evaluations (our 2023 research agenda outlined the importance of data, continual learning, and how automated research could go off the rails), and interpretability [1, 2]. MATS scholar in 2022 and Data Scientist in government before that. I’ve mentored for projects in AI Safety Camp and SPAR.

I believe I am a perfect fit to lead this organization for the following reasons:

• While working in various government departments from 2018 to 2022 (energy regulation, executive council innovation team, mental health & addictions), I learned a lot about how government works, hosted over a dozen workshops, and worked on systems thinking and strategic foresight.
• During my time as an alignment researcher, I’ve worked across almost all subfields (whether it’s engineering or conceptual) and have a concrete grasp of the core problems in alignment.
• I’ve hosted many talks and workshops on using AI to accelerate research and operations: paid workshops for MATS, talks and workshops for PIBBSS, and talks with Apart Research. When the subliminal learning paper came out, I replicated it in about half a day and taught the replication in one of those workshops. I will be able to lead this organization to be AI-native. As the organization grows, I’d expect we’ll make contributions to automating AI governance work (and AI for epistemics) and share our insights and internal products with other relevant organizations.
• I could have prepared for interviews and tried to get a good job at one of the AI labs, but I purposely never applied, so that I could stay independent and publicly candid about them.

Referrals

• Eyon Jang (Scale AI, ex-MATS): SPAR mentee on a project to automate interpretability. Email: yjang385@gmail.com
• Esben Kran (Seldon Lab): We co-organized a research augmentation hackathon for Apart Research, I helped organize and conduct the interview process for their research augmentation engineer, and we’ve had many conversations about AI safety. Email: esben@kran.ai
• Dušan D. Nešić (PIBBSS): I went through the PIBBSS programme last year, which he helps organize. Email: dusan@pibbss.ai
• Jake Mendel (Coefficient Giving): Coefficient (then Open Philanthropy) paid me to build the tool their team used to find and rank researchers to invite to their Technical AI Safety RFP. I also know Jake from conversations in co-working spaces like LISA and Constellation.

I’m happy to get more references if it would help. I’ve been in the AI safety field for a while and know a lot of people, even though I haven’t worked on direct projects with them. Some more well-known names you can reach out to: Ryan Kidd (MATS) and Marius Hobbhahn (Apollo Research).

Past work highlights

• But is it really in ROME? An investigation of the ROME model editing technique: Conducted with the mentorship of Alex Gray and William Saunders (former OpenAI employees). Documented the directional asymmetry of LLMs noted in the Reversal Curse paper using mechanistic interpretability 1 year before their paper.
• Alignment Research Literature Dataset: I wrote the majority of the code for scraping the alignment research literature. Managed contractors. We wrote up an arXiv manuscript for the work. This work has been used by organizations such as OpenAI, Anthropic, and StampyAI, as well as in other projects.
• BERDI Search Tool: Award-winning (Open Data Quality) search tool for accessing environmental and socio-economic data. I was the data scientist and LLM engineer for the project.

The first hire would be a senior technical co-lead, which I would select after (paid) trial projects. I also intend for an early hire to be someone more deeply connected to the frontier labs than I am (e.g., ex-OpenAI, ex-Anthropic, or ex-Google DeepMind), since working with labs depends on those relationships.

Anything else evaluators should know?

I do not have an official organization yet. My intention is to secure funds as an individual or be fiscally sponsored as soon as I receive confirmation of funding. I am not a US citizen, so a US entity would need either a US co-founder or a fiscal sponsor.

I will start working on this organization regardless. I will be preparing a list of cruxes to dive deeper into and start reaching out to potential co-founders.