Difficulties in Building an AI Safety Startup

Thread with discussion here.

Building an AI safety business that tackles the core challenges of the alignment problem is hard.

Epistemic status: uncertain; trying to articulate my cruxes. Please excuse the scattered nature of these thoughts, I’m still trying to make sense of all of it.

You can build a guardrails or evals platform, but if your main threat model involves misalignment via internal deployment with self-improving AI (potentially stemming from something like online learning on hard problems like alignment which leads to AI safety sabotage), it is so tied to capabilities that you will likely never have the ability to influence the process. You can build reliability-as-a-business but this probably speeds up timelines via second-order effects and doesn’t really matter for superintelligence.

I guess you can hone in on the types of problems where Goodharting is an obvious problem and you are building reliable detectors to help reduce it. Maybe you can find companies that would value that as a feature and you can relate it to the alignment-relevant situations.

You can build RL environments, sell evals or sell training data, but you still seemingly end up too far removed from what is happening internally.

You could choose a high-stakes vertical you can make money with as a test-bed for alignment and build tooling/techniques that ensure a high-level of guarantees.

If you have a theory of change, it will likely need to be some technical alignment breakthrough you make legible and low-friction to incorporate or some open source infra the labs can leverage.

You can build ControlArena or Inspect, open-source it, and then try to make a business around it, but of course you are not tackling the core alignment challenges.

Unless your entire theory of change is building infrastructure the labs will port into their local Frankenstein infra and that Control ends up being the only thing the labs needed for solving alignment with AIs. And I guess from a startup perspective, you recognize that building AI safety sabotage monitors doesn’t really relate 1-to-1 with what business owners care about right now. You essentially use your contract with Anthropic as a competence signal for VC money and getting costumers.

You can do mech interp, but again, when are you solving the superalignment problem?

So what do you do if you are under the impression that the greatest source of the risk is within the labs? Of course you can just drop the whole startup direction and do research/governance. Many end up inside the labs. You could keep doing a startup, but basically hope that your evals/monitoring product reduces some sources of risk and you might donate some of the money to fundamental alignment research.

I’m not really sure what to make of this and still have some startup ideas that I think will still be overall good for safety, but these are things I’ve been thinking a lot about recently and wanted to get my thoughts out there if anyone wanted to talk about it. The core thing is that it feels like there’s a lot of startups you can build as an AI company that would do things like robustify the world against AI, but tackling the core and conceptual problems and linking it to a venture-backed business is rough.

Another option I didn't mention is to build a company with the intention of getting acquired. This is generally bizarre, and VCs don't like it, since you'd be unlikely to deliver massive returns for them (most acquisitions are considered failures). That said, acquisitions in the AI space are quite high. Then again, VCs may be concerned the founders will just get acquihired instead.

One way that might work is to basically have no legitimate revenue for a few years and still build something the big labs really want at some point in the future (unclear what this is, but a non-AI safety company like Bun can get acquired with virtually no revenue afaict, though they only raised 7 million in funding in 2022). From an AI safety perspective, it's unclear how it would play out since your goal might be to have your tech disseminated across all companies.

...or you could build another startup which also leans heavily into capabilities like Anthropic.

I've had some updated thoughts since sharing the above, but in large part still currently think that the more important threat models I care about will likely make more progress in a non-profit like Redwood Research.

However, I did want to note Evan Hubinger's (Anthropic lead on model organisms for safety):

I think selling alignment-relevant RL environments to labs is underrated as an x-risk-relevant startup idea. To be clear, x-risk-relevant startups is a pretty restricted search space; I'm not saying that one necessarily should be founding a startup as the best way to address AI x-risk, but just operating under the assumption that we're optimizing within that space, selling alignment RL environments is definitely the thing I would go for. There's a market for it, the incentives are reasonable (as long as you are careful and opinionated about only selling environments you think are good for alignment, not just good for capabilities), and it gives you a pipeline for shipping whatever alignment interventions you think are good directly into labs' training processes. Of course, that's dependent on you actually having a good idea for how to train models to be more aligned, and that intervention being in the form of something you can sell, but if you can do that, and you can demonstrate that it works, you can just sell it to all the labs, have them all use it, and then hopefully all of their models will now be more aligned. E.g. if you're excited about character training, you can just replicate it, sell it to all the labs, and then in so doing change how all the labs are training their models.