--- title: "The Western AI safety playbook doesn’t transfer to China" published: 2026-11-09 tags: ["AGI strategy and foresight"] importance: 8 cover: "https://jacquesthibodeau.com/content/images/2026/11/great-wall-sunrise.jpg" coverAlt: "Sunrise over mountain ridges from the Great Wall, with the wall running along the ridge below." audio: "https://pub-4ee2f71bc29541a7a6e8d9694f0a1b21.r2.dev/ai-safety-in-china/audio.mp3" crosspost: lesswrong: "https://www.lesswrong.com/posts/AcBzhiaTW9KnB4EBv/the-western-ai-safety-playbook-doesn-t-transfer-to-china-1" figures: - src: "https://jacquesthibodeau.com/content/images/2026/11/great-wall-sunrise.jpg" alt: "Sunrise over mountain ridges from the Great Wall, with the wall running along the ridge below." width: 624 height: 468 - src: "https://jacquesthibodeau.com/content/images/2026/11/datong-rainbow-field.jpg" alt: "A rainbow over a harvested field near Datong, lit by a low sun under grey clouds." width: 624 height: 468 - src: "https://jacquesthibodeau.com/content/images/2026/11/datong-sunset-street.jpg" alt: "A village street near Datong after rain, with an orange sunset reflected in the puddles." width: 624 height: 336 - src: "https://jacquesthibodeau.com/content/images/2026/11/datong-car-cabbage-vinegar.jpg" alt: "Bags of groceries on a car seat, with a head of cabbage, noodles and a bottle of vinegar." width: 426 height: 457 author: "Jacques Thibodeau" canonical: "https://jacquesthibodeau.com/ai-safety-in-china/" --- Last month, I spent two weeks in China talking with people about AI safety. I learned a lot and realized the biggest mistake might not be failing to help AI safety in China, but doing so in ways that could backfire. I also learned important context that I expect will be valuable for the West. :::note[Epistemic status] Some of these points are my opinion or the opinions of people I spoke to, and are worth further investigation rather than treated as gospel. ::: ## Summary - The Western way of building the field does not transfer one-to-one. The West built its ecosystem through a combination of philanthropy and decades of developing AI safety foundations, leading founders and employees of frontier labs and nonprofits to act (however misguided) on this philosophy. China does not have the same initial conditions and can’t simply accept non-Chinese philanthropy. In China, foreign nonprofit money is restricted by law, and the bar for a Chinese partner to take it on is very high. - Safety work in China will mostly come from state-backed institutes, universities, companies and volunteers. Concordia AI, which I assumed for years was a nonprofit, is a social enterprise. - Academic exchange is generally fine, and for researchers in China, the obstacle to programs like MATS seems to be that it looks like a career break, since it is not yet as well recognized in China as it is in the West. - Obviously, hawkish messaging from the West can make AI safety work in China harder, because it makes warnings about AI look like a reason to hold China back. - Some people inside China’s AI industry think catching up is getting harder. - Incidents are shifting the Overton window there too. Six days after the Hugging Face investigation came out, China’s internet regulator named “extreme loss of control” as a risk. We should now prepare the material that explains what to update and why the obvious fixes will fail. ## Why I went On this trip, I spoke about AI safety and generally re-familiarized myself with the culture. “Re-familiarize” because I visited China on my first international trip in January 2020 (which…was cut short). This time, I came back from China recognizing that helping AI safety develop in China is, in some sense, both harder and easier than many might assume. Back in 2020, I figured China would play a big role in the future of AI, so it seemed worth gaining a better cultural understanding of China. At the time, I was exploring the idea of building a career as a [strategic foresight](https://www.lesswrong.com/posts/GbXAeq6smRzmYRSQg/foresight-for-agi-safety-strategy-mitigating-risks-and) researcher, so I hoped I could start correcting some underlying assumptions I have about China, in case it ever mattered. After years of alignment research and startup work, I’ve been exploring the idea of starting an organization focused on preparing AI safety to help the world respond thoughtfully to [future Overton window shifts](https://www.youtube.com/watch?v=Og_VPf2XQmU) from [incidents](https://www.belfercenter.org/research-analysis/pre-mortem-first-major-ai-enabled-terrorism-event-and-choices-could-enable-it). That’s the sort of thing that strategic foresight seeks to address: doing the pre-emptive work ahead of ‘black swan’ events or situations that arise after ‘complex’ interaction of n-th order effects of multiple drivers of change. We may not be able to predict the future precisely, but we can assess plausible futures and build a portfolio that is robust across trajectories. Well, anyway, it seems good to know more about China before making naïve proposals that backfire and make things worse. In this piece, I hope to point to things that may help other folks from the West gain a bit of clarity on their assumptions about China, and to point to various sources of additional material you can look into if you are extra curious. Beyond helping people in the West understand, I’ve also been curious about what kinds of developments in China would lead to safer worlds or better international coordination. In the first half of this post, I will start by covering what I learned from my conversations during my visit. In the second half, I’ll cover insights I found valuable while researching China and the AI field (including notes from a recent gathering of AI investors and researchers from China that was abruptly taken down soon after it was released!). ## Why the Western model does not transfer one-to-one The West built its AI safety field with gracious philanthropists who funded nonprofits. Although much of the funding was limited to the US, it was never seen as a blocker for nonprofits and researchers in middle powers. Because of legitimate concerns over foreign influence and the particularities of the government-nonprofit relationship in China, it is harder to fund work there, and taking action may in fact completely backfire if attempted. At the very least, it’s worth being careful. Much of community building in China is done on a volunteer basis, and it seems (in practice) their community *would* benefit from funding, but they told me they wouldn’t recommend sending money to China. ### Foreign nonprofit money faces a very high bar In China, a [foreign nonprofit](https://ngo.mps.gov.cn/ngo/portal/view.do?p_articleId=21833&p_leftmenu=4&p_topmenu=2) can only spend money through a registered office with a government sponsor, or through a temporary activity that a state-linked Chinese partner files with the police. In general, the bar is very high (higher than Western countries are used to) because the Chinese partner takes on the paperwork and liability. Funding anyone outside those routes is banned, “covertly” as well as openly. That said, cooperation still happens when the Chinese side has enough to gain. The [Safe AI Forum](https://saif.org/), a US nonprofit, is registered in China through Tsinghua University, which, as one person put it to me, means Tsinghua is to some degree vouching for them. Its [dialogue in Beijing](https://idais.ai/dialogue/idais-beijing/) was hosted with the Beijing Academy of AI, and [a later one in Shanghai](https://idais.ai/dialogues/) with two Shanghai institutes. Tsinghua professors are among the conveners of the series, and Yoshua Bengio and Stuart Russell attended the Beijing one. Events like these bring very established experts, which makes the effort worth it for the host. Note that [EA](https://www.effectivealtruism.org/) is not well known in China, so most of the time it isn’t a sensitive topic. For people who do know it, it can be, because of what EA money has funded. Chinese organizations seem susceptible to various risks when they accept funds from [Coefficient Giving](https://coefficientgiving.org/), especially since it has funded institutions like [RAND](https://www.rand.org/) and [CSET](https://cset.georgetown.edu/). So I worry it could become a more salient issue in the future. ### Where safety work in China comes from As I understand it, another reason nonprofits are disliked in China is that: “Why do we need nonprofits? These things should all be handled by the state!” As a result, the sort of AI safety ecosystem you could expect in China would be primarily coming from: - The State or State-affiliated institutes - Universities (e.g. various PhD programmes or fellowships like the one from the [new QingZhi AI Safety Hub](https://www.lesswrong.com/posts/bgTP2SrFcTYBSwbxa/tsinghua-started-an-ai-safety-residency), which is affiliated with Tsinghua University). - Note: perhaps one of the most consequential things of this year is [Sören Mindermann’s decision to become a professor in Beijing](https://sorenmindermann.substack.com/p/why-im-becoming-a-professor-in-ai). - Companies or social enterprises (a business that pursues a social mission) that also do AI safety work. For example: - **[Concordia AI](https://concordia-ai.com/).** I’ve known about them and their founder, Brian Tse, for years, but I always thought they were a nonprofit. Turns out they are a social enterprise. [Their FAQ](https://concordia-ai.com/about-us/faqs/) says most of their funding comes from philanthropic organizations and family offices in China, Singapore and Europe, none from governments, and that they have partnership agreements with DeepSeek, MiniMax, Z.ai and Huawei. They mention philanthropic donations from Europe, though I’m unsure how exactly it fits within the rules. - More context from their FAQ: “Certification followed a rigorous assessment of our mission, governance, impact, and financial sustainability. As part of this, our Beijing entity committed to allocating at least 35% of annual after-tax profits to public-benefit activities or its own development. In practice, all of our funds go to our mission, either directly or as operating costs.” - **[Fangcun AI](https://fangcunleap.com/en/).** Incubated at Tsinghua University’s College of AI and the Institute for Interdisciplinary Information Sciences. I met them during my trip to China. - **ByteDance.** Big companies have started looking into AI safety. [ByteDance Seed announced an Alignment & Preparedness team on October 14](https://seed.bytedance.com/en/blog/seed-alignment-preparedness-team-looking-for-global-talents), with plans to initially recruit ten researchers worldwide to work on interpretability, monitorability and independent evaluations. - Volunteers - They already organize [BlueDot-style courses](https://aisafety-cn.com/en/resources), which researchers at Chinese AI labs attend. ### Training and exchange That said, we can leverage the wider ecosystem outside China to coordinate, upskill Chinese folks, and elevate their ecosystem. For example, it may be high EV to upweight Chinese researchers in programs like [MATS](https://www.matsprogram.org/) as a diversity measure. For most researchers, travel is not the obstacle: the [restrictions Reuters reported in May, citing Bloomberg](https://www.reuters.com/world/asia-pacific/china-restricts-overseas-travel-top-ai-talent-alibaba-deepseek-bloomberg-news-2026-05-26/), apply to top staff at places like DeepSeek. The bigger obstacle, I’m told, is that people can’t easily take a career break. A program like MATS doesn’t guarantee a job afterwards, and it isn’t well recognized in China. So formats that don’t require quitting a job, and programs that employers in China recognize, may matter more than extra admission slots. In the other direction, it seems valuable for senior researchers to attend the [Beijing AI Safety Residency](https://www.lesswrong.com/posts/bgTP2SrFcTYBSwbxa/tsinghua-started-an-ai-safety-residency) to build working relationships with the AI safety community in China, communicate technical alignment research, and mentor Chinese researchers. Either way, I expect technical research will be much safer in such exchanges than governance work. Some researchers in China say contact with foreign groups feels risky. But from what I was told, academic exchange is generally fine, and the mood right now is fairly open. The sensitive areas are narrow, such as offensive cyber work aimed at critical infrastructure. I expect many startup staff and academics could attend and benefit from training programs, and it would be worth being highly flexible in accommodations to make it accessible to Chinese citizens. If your worry is that training researchers from China helps their labs’ capabilities, start with [Ben Bucknall and colleagues’ paper](https://arxiv.org/abs/2504.12914) in the reading list: it sorts areas of technical safety work by how much capability spillover and other risk they carry. ### Investing and buying services Now, given the constraints on nonprofits in China, a funder could invest directly in startups instead. It is unclear to me how valuable this could be overall, but it may still be worth investing in AI safety startups in China. For example, when I visited them in Beijing, Fangcun AI told me that they see themselves as the “Apollo Research of China”. They have been doing work around loss of control and harmful manipulation, and recently did a [safety report on a DeepSeek model](https://arxiv.org/html/2610.35671v1) ([here’s the tweet thread](https://x.com/FangcunLeap/status/2114719692137136245?a=MO_Ly9X5wTopd_6rMb8&s=20)). Unfortunately, investing may not be possible for much longer in the US, since a [new US Treasury proposal](https://home.treasury.gov/news/press-releases/sb0667/) would bar US investors from many Chinese AI companies. Investors elsewhere (e.g. the UK) may still have opportunities to invest, provided it works within their country’s laws. Another option is to seek services directly from companies in China so they can continue funding their safety work internally. As far as I can tell, the Treasury proposal does not cover ordinary service contracts. Either way, espionage, data, and secrets laws still apply to for-profits. ### Be careful; don’t rush Ultimately, one worry I heard is that if funders rush money into China because timelines feel short, someone may get paperwork wrong or anger officials, and many people in China would pay for it. It’s better not to rush and to speak with Chinese people before trying a strategy. To repeat the point from the top: the biggest mistake might not be failing to help AI safety in China. It might be trying to help in ways that make the work harder for the people already doing it. ## China is suspicious of the West and Western safety framing As Sarah Sun, a Shanghai-based founder of the Open Community for AI Safety China, said in a NY Times interview: “The fact that Dario has been so hawkish was really, really unhelpful in this scenario.” This messaging makes AI safety efforts in China much harder, often making recent warnings about AI seem more like justification for holding China back rather than genuine concern about catastrophic risk. The clearest case is from September. [The Guardian reported](https://www.theguardian.com/world/2026/sep/20/why-china-is-pushing-back-on-us-warnings-over-rapid-ai-development) that Amodei proposed a “worldwide pacing of the frontier”, which would need China’s cooperation, and quoted an analyst saying that Beijing reads calls like that as an attempt to lock in the US lead. Years of hawkish essays can make a cooperative proposal look like a trick. Indeed, in September, a DeepSeek researcher wrote (flagging it as an exaggeration) that [allowing Anthropic to monopolize AI would be “no less serious” than Hitler](https://mp.weixin.qq.com/s/zk0KxuLzhmMJ4LPYW_OHMA) having acquired the technology for the atomic bomb first in World War II ([ChinaTalk has a translation](https://www.chinatalk.media/p/how-anthropic-became-chinas-goliath)). It’s not just Western frontier AI company employees feeling the pull to save themselves from competitors. Somewhat relatedly, on September 1, an official from the Cyberspace Administration of China [listed the five main AI safety risks](https://www.ithome.com/0/996/868.htm) the country faces. The fifth was “technological hegemony”: individual countries, the official said, build supply chain barriers with export controls, and use their advantage in public opinion to “concoct false test reports on large models and maliciously smear other countries’ technological development” (the translations from Chinese in this post are rough). It may be that when a Western lab publishes a safety evaluation of a Chinese model, one should expect it to be filed under the fifth risk, not the second. That said, the same report mentions loss of control! Specifically, these are the five risks they listed: 1. Inherent technological vulnerabilities that are difficult to engineer out, 2. A giant leap in model capabilities disrupting the traditional security paradigm, giving rise to the risk of extreme loss of control, 3. The rapid evolution of application forms and the accelerated iteration of security risk patterns (ie from compromised agents), 4. Misuse, abuse, and malicious use, which challenge social order and ethical boundaries (e.g., information manipulation, biorisk, weakened demand for traditional labor), and 5. The threat of technological hegemony facing the global AI sector (an implicit but clear swing at the United States). These are valuable things to focus on! But, going back to China hawks. A lot of the policies that get thrown around, especially from Dario Amodei and other China hawks, read to me as coming close to saying this without saying it too explicitly: we need to beat China, take over the broader economy, and then do some kind of pivotal act that overthrows the Chinese government. Of course this can be troubling! I don’t think we should push for an international monoculture. While we shouldn’t be naïve about geopolitical context, I think this is a fundamental crux we need to avoid. China may do things we wouldn't want in our own countries, but to some extent we have to let them decide how they live. The alternative (no strong international agreement) may be much worse. Note that on October 23, Wang Yi, China’s foreign minister, [said](https://www.mfa.gov.cn/web/wjbz_673089/xghd_673097/202610/t20261023_12046093.shtml) that China and the US, as the two countries in the first tier of AI, “have both the ability and the responsibility” to develop and manage AI together and to keep it under human control. We should take diplomatic readouts with a grain of salt, but we should not completely ignore China’s willingness to engage. ## What is the current vibe on AI from AI researchers and investors in China? In early October, a Chinese investor hosted a private discussion on China’s AI labs and published the minutes as a post ([that was quickly taken down!](https://mp.weixin.qq.com/s/YOWslIRykThfGGwx-71RHA)) that was shared via the Official Account of Startup Research on WeChat ([full translated version](https://x.com/odysseus0z/status/2118628525074956475)). It includes many important insights into how AI researchers and investors in China are thinking about the AI race. I thought the following discussion notes were insightful. ### Chinese AI companies expect catching up to become harder Participants said anti-distillation measures, API restrictions, compute constraints, and the shift toward harder-to-copy data (e.g., computer use, spatial understanding, and video generation as a path to models of the physical world) are raising the cost of catching up. This summer, [Panfilov et al.](https://arxiv.org/abs/2608.09867) showed that it was possible to steal reasoning traces from proprietary LLM APIs. As a result, frontier AI companies (e.g., Anthropic and OpenAI) seem to be putting more effort into protecting their models from being distilled. Chinese companies still seem to be trying to distill (using techniques like splitting the problem and reverse rationalization), but the workarounds produce worse results. Anthropic’s [September threat intelligence report](https://www.anthropic.com/threat-intelligence-report-september-2026) is a first-party version of this. It says Anthropic disrupted distillation attacks from seven labs based in China, including DeepSeek, Alibaba, Moonshot and Zhipu, and it attributes to Zhipu over 3.4 million exchanges in 17 days, including a chain-of-thought extraction pipeline run through 273 accounts. One participant said a single lab spends about $300,000 a day on teacher tokens, about $100 million a year. These details seem in line with how I thought about these companies, which seemed to rely on training on frontier LLM traces. Here’s a [tweet of mine from July](https://x.com/JacquesThibs/status/2078169387849175045): “Kimi K3 is obviously good and a big accomplishment, but I am indeed of the view that: - There’s lots to pull out of pretraining via great post-training - Using traces from other better models allows you to hit the ceiling much faster - Therefore, an exceptional post-training process allows you to leapfrog stronger base models with weaker post-training (as Ryan effectively shares an example of) My guess is that Google is much more careful about post-training on OpenAI/Anthropic traces, which is one reason they are behind the Chinese labs as of now. I also think it’s true that the Chinese labs are currently paying less attention and effort to safety-related stuff. Google has much more to lose if they release a model that does/says some crazy shit. I know, I know, “distillation cope”, but I do want to say I’m fine being proven wrong, and I also don’t deny the Chinese labs have strong technical teams with extreme talent density. I’m also not anti-open-source.” Ultimately, pushing the frontier in something like coding is much harder than pulling an ideal initial distribution via SFT of high-quality reasoning traces from the get-go. And it was clear that this would let Chinese and other companies that took advantage of it catch up for a while, but eventually you’ll be far behind on engineering and data infrastructure, and you’ll lack the compute to scale much bigger pretrained models. Speaking of. ### The Chinese labs want to train bigger pretrains to distill, but lack the compute You can look at the [original post](https://x.com/odysseus0z/status/2118628525074956475) for the specific numbers, but the key point is that, according to the notes, Chinese compute is enough for models around 3 trillion parameters, whereas the shortage starts to bite around 10 trillion, where American labs have the compute. It seems that Chinese labs are somewhat starting to doom about this. They are concerned that, with RSI (recursive self-improvement) potentially on the horizon, the gap between them and Western frontier labs could widen from three to four months today to 18 to 24 months (!) if US labs reach RSI next year. Chinese lab employees argue that RSI will be compute-bottlenecked (running tons of experiments autonomously in parallel), which is where they can’t compete. I asked someone from China about it, and they said: “I also discussed this analysis with several friends working in China’s AI industry, and they believe its main points align well with the current consensus within the sector. Moreover, regarding the sensitive issue of distillation, to my knowledge, some employees at Chinese AI laboratories currently believe that even without aggressive anti-distillation measures, it will be difficult to catch up as quickly as before in the single, verifiable domain of coding through algorithmic innovation, infrastructure engineering optimization, and large-scale distillation. This is because the capabilities of current leading U.S. models generalize across too many domains, and progress under RSI is occurring at an exceptionally fast pace.” Obviously, if true, this should be considered a point in favour of preventing the fast-paced race-to-the-death in the West. At that point, you are simply opening yourself up to model theft, and the Chinese labs are not as close behind you as many seem to believe. On October 9, [Sarah Godek replied](https://x.com/SarahJGodek/status/2108682893191004459) to SemiAnalysis’s [Beijing Will Not Pace the Frontier](https://newsletter.semianalysis.com/p/beijing-will-not-pace-the-frontier) with a thread that I think gets the framing right. She agrees that Beijing is not eager to pace the frontier right now, but not because China feels locked in a life-or-death race with the US the way we do: “I believe the core reason is that AI safety is thought about differently in China, with most stakeholders having longer timelines for AGI and RSI and AI development often geared toward specific applications to be carefully tested and managed rather than a general capabilities threshold that brings deep uncertainties. Multiple stakeholders have told me that China is running the race ‘not to lose’ as U.S. capabilities accelerate (though I’m sure winning would be considered a great byproduct).” Her title for the piece would have been “Beijing Thinks About AI Safety Differently and Has Growing Concerns About AI Risks”. She also makes a point that applies to this post: research done with AI tools can miss Chinese sources that live only on WeChat or on sites that block scrapers, which is another reason I reached out to Chinese people on WeChat for their perspectives. One additional note from the translator, citing The Information: Huawei now has engineers working from DeepSeek’s offices, and Huawei plans to give it most of the supply of its next training chip due in early 2027. One could say that bans on US chip exports accelerated this outcome and may make China less likely to consider an international agreement as a result. Access to Nvidia chips may also get harder from here, not easier. Last December, Trump allowed Nvidia to sell its H200 to China, routed through the US so that the government takes a 25 percent cut. On November 7, the New York Times [reported](https://cn.nytimes.com/business/20261109/nvidia-chips-chinese-firms/dual/) that the Commerce Department has approved roughly two dozen Chinese firms to buy or handle the chips, that two of them have no visible presence at their listed address, and that several have ties to companies on the US entity list or to military suppliers. So far Nvidia sells only small amounts of the H200 to China, partly because Beijing has been slow to approve purchases while it promotes domestic chips, and many Chinese companies reach Nvidia compute through data centers in Southeast Asia instead. My guess is that this scrutiny tightens both routes, and the compute shortage gets worse. Understandably, many of the US’s actions make it hard for China to trust it, including reneging on international agreements like the Iran deal. While trust isn’t necessary for an international agreement that benefits both sides, zero-trust verification technology will take time ([though we’re seeing progress](https://www.youtube.com/watch?v=yntXV1pkHuI)), and it would be nice not to cause human extinction because there is little appetite to work together. ### Chinese labs historically competed on inference cost, but this is increasingly no longer the case Ever since DeepSeek arrived on the scene, it was remarkable how an AI lab could come out of nowhere and completely smash Western labs on cheap intelligence for mid-tier models. In their case, DeepSeek made these advances through smart inference engineering. These days, DeepSeek still has an incredibly cheap model with V4.1 Flash, but all other Chinese labs are, at best, tied with models like GPT-6 Luna. Anthropic seems to be investing further to make Sonnet and Haiku competitive in this aspect. Overall, this former advantage is being squeezed over time, and the Chinese labs will likely need to do something new to avoid getting squashed. One reason this is particularly bad is that Chinese companies want high-quality production data from their customers. In fact, it’s been hypothesized that their performance gains in the last two years have partly come from being able to train on-distribution by using companies like OpenRouter to gather training data. Part of the difficulty of losing cost competitiveness is that the best customers will leave first and use Claude and Codex directly; therefore, Chinese labs struggle even more to acquire high-quality data. Ultimately, I expect that there will be a reckoning for Chinese labs, and they may be forced to effectively consolidate. Something similar happened in China’s electric car industry. ### What the founders say they believe On July 11, Zhipu’s Tang Jie sent [a letter to all staff](https://www.thepaper.cn/newsDetail_forward_33568012) announcing a two-year “Touch High” plan: no chasing short-term revenue, four bets instead. Among the bets, he mentioned “extreme safety governance”, with a plan to spend on the order of ten billion yuan on mechanistic interpretability. He also said: “the realization of superintelligence and research on superalignment must advance together.” In August, [in an article Tsinghua republished](https://www.tsinghua.edu.cn/info/1662/127600.htm), he called for legislated safety red lines, circuit-breaker mechanisms that can halt a system, and deep alignment with socialist core values. On November 1, [he told an audience in Shanghai](https://www.scmp.com/tech/tech-trends/article/3372834/self-improving-ai-key-step-towards-agi-safety-key-zai-founder) that recursive self-improvement is the main research problem on the way to AGI, and that safety is key. And in September, Zhipu co-wrote [a risk management framework for open-weight releases](https://concordia-ai.com/research/frontier-open-weight-ai-risk-management-framework/) with Concordia AI. Although I'm not exactly phrasing this in the gospel of AI safety (which we all know every Western founder seems to have co-opted to some extent so they can hire talented people and reduce criticism), I don’t think we should assume Chinese founders won’t be open-minded about the fundamental risks of misaligned superintelligence. As AI progress moves forward, the Overton window in China has many opportunities to shift. ## The Overton window may shift in China too What caused the biggest shifts in the Overton window in the West? Much of it came from a combination of AI capabilities, misalignment/misuse warning shots, and researchers speaking out amid growing concerns. After the [Hugging Face incident](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/), many researchers (including safety researchers) finally felt spooked about AI risks. For many, it was the first time they actually felt “oh shit, this is *real* real!”, even those who spent years working in the field and warning about the risks. As capabilities rise in China, some Hugging Face-adjacent incident happens, and the West continues to have its own warning shots; we should expect an increase of AI researchers and policymakers in China who are similarly spooked. The question these days is, “once the Overton window shifts, how will China react depending on what happens between then and now?” While unfortunate, many people will need to experience an event or reflect on something emotion-based to update the sorts of actions they take. - For [Yoshua Bengio](https://x.com/JacquesThibs/status/2103505634771468564), he said, “I was actively working on making AI smarter. And you want to feel good about your work. … [For me, what countered that] is love, love of my children.” - It seems for many safety and safety-adjacent researchers have also felt a shift in vibe. [Tom Davidson](https://x.com/TomDavidsonX/status/2116302652325638180) seems to have had enough of an emotional reaction to re-weigh misaligned takeover risks much more relative to AI coups or secret loyalties, and that shift seems to have been triggered by the Hugging Face incident. - For [Roon](https://x.com/tszzl/status/2080093670980899141) of OpenAI: “shaken up a bit by the hugging face incident. I hope we (the company) use the rare gift of a warning shot to do much better in the future. it is very easy to misalign and underconstrain powerful models” - For [Dan Selsam](https://x.com/DKokotajlo/status/2099600298855829616) of OpenAI, it seems to also have been triggered by the Hugging Face incident: “I have become extremely concerned about how far language models have come.” Based on brief conversations and other anecdotes found online (e.g. a popular news show in China apparently covered the details of the [Hugging Face incident](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/)), it seems to have similarly started to wake up some people in China. Albeit it seems much less, it may blow open the Overton window once China experiences similar incidents. There is a public sign of it. On September 1, six days after METR published its investigation, an official from the Cyberspace Administration of China [told a press conference](https://www.ithome.com/0/996/868.htm) that several tech giants had recently disclosed incidents of large models going out of control, with agents in safety evaluations bypassing their sandboxes and attacking real production systems outside them. The official said the risk of extreme loss of control “requires a high degree of vigilance”. Two weeks later, version 3.0 of China’s AI Safety Governance Framework named evaluation deception, sandbagging and shutdown resistance, down to models modifying or disabling their shutdown scripts after being told to stop ([the China AI Bulletin has a good summary](https://chinaaibulletin.substack.com/p/caib-special-issue-2-the-ai-safety)). Two pieces are worth reading against this. Zilan Qian at the Oxford China Policy Lab argues in [China Says It Cares About Loss of Control](https://oxfordchinapolicylab.org/research/china-says-it-cares-about-loss-of-control) that the Chinese term covers several different things (the operator losing control of a system, the state losing control of the technology, dangerous knowledge spreading) and that Western observers should not over-index on top-level mentions of the words. And the Asia Society’s [report from WAIC 2026](https://centerforchinaanalysis.asiasociety.org/p/reporting-back-from-waic-2026-how) found that the frontier labs there treat safety mostly as a compliance obligation, and that proposals framed around loss of control are unlikely to gain traction, while cooperation on misuse might. I personally think that these things can change rather quickly. One reason I think it’s worth engaging with Chinese academics on pure technical alignment research concerns is that we can imagine a scenario where the state gets spooked, turns to academics for answers, and then acts together with domestic companies and universities. [Sarah Godek’s thread](https://x.com/SarahJGodek/status/2108682893191004459) from October, which I quote in the industry section, reads the same sequence as growing concern: the Cyberspace Administration has started to take model risks more seriously since the Hugging Face incident, and the framework update added many more risks, especially in applications. She also answers the “it’s only voluntary” objection: “binding” is not a clean binary in China. When Beijing or an agency says that something ought to happen or that a risk should be managed, companies have good reason to follow through even when the measure is technically non-binding, because it may become binding later, with liability for present non-compliance, and not all regulation-like provisions are made public. It could be that China is much more willing to bring down the hammer if there is a loss of control incident that threatens the CCP in some way. Or it may not be. I could also imagine the companies themselves deciding that they need to do the safety work. I think one thing we need to convey here is detailed breakdowns of what updates should be made and what actions one should take based on such incidents. The worst case is that naïve key stakeholders keep barreling toward “doom” by responding to reward hacking with papering over issues through further behavioural training, pushing alignment problems further down the road. That is, if we’re already expecting these sorts of things to happen, our actions *right now* should include preparing material that explains how to update their technical model of alignment and why their predictable attempts to fix the problem are likely to fail. One thing that could help is translating or rewriting key texts in Chinese and posting them where people in China actually read. Efforts like [Babel Translation](https://www.lesswrong.com/posts/u973rQm56oijS5Adf/launching-babel-translation) could be highly valuable! There is a Chinese platform ([Zhihu](https://www.zhihu.com/)) that, afaict, works like a mix of Reddit, Substack and Twitter, where people write posts and others upvote them. Someone I talked to at Concordia AI posts about AI safety there. It’s also where [Thoughts Memo](https://www.zhihu.com/people/L.M.Sherlock) posts, and they recently posted on [LessWrong about giving China a chance](https://www.lesswrong.com/posts/GoX3uYQ4QN5HKvL7u/please-give-them-a-chance-on-china-rationalism-and-ai-safety) and translating rationalist and AI safety work. Note: as I mentioned in the opening, I’m exploring starting an organization that would do this sort of work, and I am seeking funding for it. I expect it would be valuable to expand this section into a proper project that studies how Overton window shifts would actually happen in China, runs pre-mortems on how the incident responses could go wrong, and works out what outsiders could usefully do (even if that is as simple as communicating technical research more clearly while avoiding any outside attempt to influence policy). If you are a grantmaker who is interested, or someone who can help me convince funders to fund me, please reach out. ## What to do, and what not to do - Consider talking to people in China before trying any strategy, and don’t make foolish decisions because timelines feel short. - Consider investing in safety startups and buying services from companies there. - Make training accessible. Programs like MATS are usually 3 months; perhaps there should be more condensed tracks for Chinese researchers. Or split between part-time remote and full-time intensive. - Consider visiting China! Or even attending a fellowship or becoming a professor. - Communicate technical research clearly, in Chinese where you can. China did not have the benefit of LessWrong to build intuitions around the alignment problem. One important lever for pushing the world toward better coordination is communicating the technical alignment problem clearly. ## Conclusions I’ve found great value in connecting with Chinese folks on WeChat. Already, I’ve been getting messages from people interested in AI safety for advice, and some have thanked me for sharing my insights. On this, my goal is to be helpful in the way I can, without seeking anything in particular in return besides, perhaps, friendship. Oh, and let me say that I had a wonderful time in China, and it was such a great experience meeting all the kind Chinese people! Special shoutout to the two old men who brought me and my girlfriend to a beautiful Great Wall sunrise and our driver in Datong, who got stuck in the mud and spent hours trying to get unstuck while we looked for the birds we travelled to that area for (unfortunately, few of them were there, but there was a gorgeous sunset with a rainbow that showed up just as we got towed out of there!). :::gallery[Three photos from the trip] ![Sunrise over mountain ridges from the Great Wall, with the wall running along the ridge below.](https://jacquesthibodeau.com/content/images/2026/11/great-wall-sunrise.jpg) Sunrise from the Great Wall. ![A rainbow over a harvested field near Datong, lit by a low sun under grey clouds.](https://jacquesthibodeau.com/content/images/2026/11/datong-rainbow-field.jpg) The rainbow near Datong, as we got towed out. ![A village street near Datong after rain, with an orange sunset reflected in the puddles.](https://jacquesthibodeau.com/content/images/2026/11/datong-sunset-street.jpg) Sunset near Datong, after the rain. ::: Our driver gave us some vinegar from Shanxi province, and my girlfriend has been craving traditional Chinese food, so she’s been making some of the dishes we had while we were there (I never thought I’d say this, but the cabbage is *so* good!).
Bags of groceries on a car seat, with a head of cabbage, noodles and a bottle of vinegar.
Thanks to Sarah Sun for comments on a draft of this post. ## Further Recommendations ### Start here [AI Safety in China: A Primer](https://jasminexli.substack.com/p/ai-safety-in-china-a-primer), Jasmine Li. An orientation to research groups, companies and community activity. Useful as an ecosystem map; its participant observations and informal estimates are not a representative census. [Headwinds Everywhere](https://www.lesswrong.com/posts/AdM6Sdg5fZPnrHWSe/headwinds-everywhere-in-the-chinese-ai-safety-field), CMLKevin. Field notes on funding, foreign collaboration and access to frontier models. Especially useful for practical obstacles that a fellowship alone would not remove. [China’s AI Safety Money Problem](https://www.chinatalk.media/p/chinas-ai-safety-money-problem), Nick Corvino. Makes the case that careers and institutional financing constrain sustained work. Read for the mechanisms, rather than treating its international funding estimates as directly comparable. [China needs more AI safety—more of what, exactly?](https://www.lesswrong.com/posts/TguvLbnzpgmcAj8tu/china-needs-more-ai-safety-more-of-what-exactly), Yulu Pi. Separates changes sought from government, developers and researchers. A useful test for an intervention: which actor should do what differently, and why? [Personal experience does not a study of China make](https://www.lesswrong.com/posts/rgwJjfKHGmArnqZsv/personal-experience-does-not-a-study-of-china-make), Celeste Li. Explains why selected contacts and personal familiarity cannot support national generalizations. Particularly relevant when interpreting trip reports, including this one, and private industry discussions. [A Chinese perspective on “On social reality in China”](https://www.lesswrong.com/posts/m4fWCPqWkpb39SKi7/a-chinese-perspective-on-on-social-reality-in-china-2), Dusty. Challenges broad cultural explanations by emphasizing comparison groups, institutions and generational differences. Knowing how a social system works is not the same as endorsing it. ### Chinese voices and institutions [The State of AI Safety in China (2026)](https://aisafetychina.com/), Concordia AI. The annual report, with data. [CAIB Special Issue 2: the AI Safety Governance Framework 3.0](https://chinaaibulletin.substack.com/p/caib-special-issue-2-the-ai-safety), Emmie Hine, China AI Bulletin. What the state’s own framework now says about loss of control, and what it leaves undecided. [Tang Jie’s July 2026 letter to Zhipu staff](https://www.thepaper.cn/newsDetail_forward_33568012), The Paper, in Chinese. The “Touch High” plan: long-horizon tasks, autonomous agents, fully self-training AI and safety governance, with the line that superintelligence and superalignment must advance together. [Frontier Open-Weight AI Risk Management Framework](https://concordia-ai.com/research/frontier-open-weight-ai-risk-management-framework/), Concordia AI and Z.ai. What changes when weights are released: no recall, removable safeguards, no single party able to monitor use. [Detecting and countering misuse of AI: September 2026](https://www.anthropic.com/threat-intelligence-report-september-2026), Anthropic. The distillation cases attributed to seven Chinese labs, with the caveat that these are one company’s attributions from its own logs. [China Says It Cares About Loss of Control](https://oxfordchinapolicylab.org/research/china-says-it-cares-about-loss-of-control), Zilan Qian, Oxford China Policy Lab. Why the Chinese term covers more than ours does, and why top-level mentions should not be over-read. [Reporting Back from WAIC 2026](https://centerforchinaanalysis.asiasociety.org/p/reporting-back-from-waic-2026-how), Asia Society Center for China Analysis. Four observers on how the labs and regulators in Shanghai talked about safety: compliance and misuse, more than loss of control. ### Research and collaboration [China’s AI Responsibility and Global Action](https://www.pekingnology.com/p/xue-lan-chinas-ai-responsibility), Xue Lan, translated by Pekingnology. A Chinese policy scholar distinguishes misuse, loss of control and systemic risks, including concerns about autonomous self-improvement. His views are not an official negotiating position. [Why I’m becoming a professor in AI safety in Beijing](https://sorenmindermann.substack.com/p/why-im-becoming-a-professor-in-ai), Sören Mindermann. A concrete account of building research relationships through Tsinghua, alongside a technical agenda. Read beyond the appointment announcement for what the collaboration is meant to enable. [In Which Areas of Technical AI Safety Could Geopolitical Rivals Cooperate?](https://arxiv.org/abs/2504.12914), Ben Bucknall and colleagues. Evaluates cooperation by capability spillovers, sensitive information and opportunities for harm. A better starting point than assuming all technical safety collaboration is equally benign. [Launching Babel Translation](https://www.lesswrong.com/posts/u973rQm56oijS5Adf/launching-babel-translation), Jasmine Li and Celeste Li. An existing effort to translate safety material into Mandarin and other languages, with substantive editing. Useful infrastructure to investigate before proposing another translation project. ### Evaluation and preparation that could change decisions [A Pre-mortem on the First Major AI-Enabled Terrorism Event](https://www.belfercenter.org/research-analysis/pre-mortem-first-major-ai-enabled-terrorism-event-and-choices-could-enable-it), Stephen Casper and colleagues, Belfer Center. Works backward from hypothetical harm to decisions that could enable or prevent it. Useful as a method; its misuse scenarios should not be conflated with autonomous loss of control. ### Reciprocal restraint: proposals and objections [Why China doubts US calls to pause AI](https://www.chinaffairsplus.com/p/sun-chenghao-why-china-doubts-us), Sun Chenghao. Explains the credibility problem from Chinese interlocutors’ perspective and proposes observable reciprocal steps. Helpful for asking what a counterpart would need to see, not just hear. [The Compute Verification Post](https://firstscattering.com/p/the-compute-verification-post), Jason Hausenloy and Jasmine Li. Introduces verifiable properties, trust assumptions and practical projects. Particularly useful on why checking declared computation and finding undeclared computation are different problems. [AI 2040: Plan A](https://ai-2040.com/), AI Futures Project. A detailed positive scenario for international restraint, transparency and slower development. Read as a proposal to scrutinize, not a forecast that agreement will occur. [AI 2040: Is it Actually a Deal?](https://www.lesswrong.com/posts/LtmbQN9mzmFWxWtrD/ai-2040-is-it-actually-a-deal), 1a3orn. A focused counterpoint: monitoring and deterrence do not tell parties how to resolve substantive disagreements. Useful for identifying the decision procedures an agreement still needs. [Beijing Will Not Pace the Frontier](https://newsletter.semianalysis.com/p/beijing-will-not-pace-the-frontier), Mark Chen, Doug and Dylan Patel, SemiAnalysis. A skeptical counterweight, contrasting risk language with disclosure and development practices. Its forecast of future restraint is an inference, not the disclosure data itself. Read it with [Sarah Godek’s reply thread](https://x.com/SarahJGodek/status/2108682893191004459), which I believe provides a solid counterpoint. [The Race to Control AI — Overtime](https://youtu.be/YbBXIZtznpI?t=3705), Robert Wright and Jeremie Harris. The second half (overtime) of the discussion examines verification, relationships and how perceived alignment progress could alter cooperation. Jeremie’s position is very hawkish, but it’s worth listening to hear the full breadth of perspectives. ## Sources Every external link in this piece that has a captured card, with what that page said when it was captured. The quoted lines below are not the author of this piece writing: they are the linked page describing itself, recorded by `bun run link-cards` on the date given, and kept so that a reader still has them if the original moves or goes away. - **Foresight for AGI Safety Strategy: Mitigating Risks and Identifying Golden Opportunities** — jacquesthibs, lesswrong.com, 2022-12-05 Captured 2026-08-28. > This post is about why I think we should use strategic foresight for AGI governance work. I am writing this without having read all the AGI governance / strategy reports and reading materials. I was planning on putting more effort into this post, but this is a better-than-nothing-at-all post. Worst-case, this might… - **AI Governance In 2026: Why Policy Moves Only After Incidents | Stephen Casper (Harvard University)** — FAR․AI, youtube.com Captured 2026-11-10. - **A Pre-mortem on The First Major AI-Enabled Terrorism Event and The Choices That Could Enable it: Plus 9 Predictions** — belfercenter.org Captured 2026-11-10. > Frontier AI developers' model reports and the dedicated risk-assessment teams at governmental AI institutes such as UK AISI and US CAISSI now treat a major AI-enabled cyber, biological, or chemical terror event as a credible concern. But how, precisely, might such an event happen? What choices across the model… - **境外非政府组织办事服务平台** — ngo.mps.gov.cn Captured 2026-11-10. - **Safe AI Forum** — saif.org Captured 2026-11-10. - **IDAIS-Beijing - International Dialogues on AI Safety** — idais.ai Captured 2026-11-10. - **Dialogues - International Dialogues on AI Safety** — idais.ai Captured 2026-11-10. - **Effective Altruism** — Centre for Effective Altruism, effectivealtruism.org Captured 2026-11-10. > Explore how effective altruism can help you do more good with your time, money and career, from learning about key ideas to finding ways to take action. - **Home** — coefficientgiving.org Captured 2026-11-10. > Coefficient has directed over $7 billion in grants since 2014. Our mission is to help others as much as we can with the resources available to us. - **CSET | Center for Security and Emerging Technology** — cset.georgetown.edu Captured 2026-08-28. > CSET produces data-driven research at the intersection of security and technology, providing nonpartisan analysis to the policy community. CSET is currently focusing on the effects of progress in artificial intelligence (AI) and advanced computing. We seek to prepare a new generation of decision-makers to address the… - **Tsinghua started an AI-safety residency** — Andrew Gritsaev, lesswrong.com, 2026-10-20 Captured 2026-11-10. > Applications are open for the Spring 2027 cohort of the Beijing AI Safety Residency, hosted at the Beijing AI Safety Hub affiliated with Tsinghua University. There is also an autumn residency that has already begun. Key details from the FAQ and application form: * The hub is co-directed by Sören Mindermann (Technical… - **Why I’m becoming a professor in AI safety in Beijing** — Sören Mindermann, sorenmindermann.substack.com Captured 2026-11-10. > Next month, I am moving to Beijing and joining Tsinghua University’s College of AI as an assistant professor working on frontier AI safety, especially loss of control. - **Concordia AI - Guiding the governance of AI for a long and flourishing future** — concordia-ai.com, 2019-06-06 Captured 2026-11-10. > We advance global AI safety by conducting research, advising leading AI companies and policymakers, and promoting international dialogue. - **FAQs - Concordia AI** — concordia-ai.com, 2026-10-30 Captured 2026-11-10. > Last updated: November 2, 2026 Concordia AI (安远AI) is an independent AI safety organization, with offices in Beijing and Singapore. We advance AI safety and governance by conducting research, advising leading AI companies and policymakers, and promoting international dialogue. Concordia AI works to steer the… - **Fangcun AI - Frontier AI assessment, protection, and governance** — fangcunleap.com Captured 2026-11-10. > Fangcun AI: frontier model risk assessment, runtime protection for AI agents, and AI safety governance. - **Seed News - ByteDance Seed Team** — seed.bytedance.com Captured 2026-11-10. > Seed Alignment & Preparedness team looking for global talents - **Open Community for AI Safety China** — AI安全开放社区, aisafety-cn.com Captured 2026-11-10. > A professional network for Chinese-language readers working on AI safety: talks, paper reading, courses and in-person meetups. - **MATS Research** — matsprogram.org Captured 2026-11-10. > The MATS Program is an independent research and educational seminar program that connects talented scholars with top mentors in the fields of AI alignment, governance, and security. - **In Which Areas of Technical AI Safety Could Geopolitical Rivals Cooperate?** — Ben Bucknall and 21 others, arxiv.org, 2025-04-17 Captured 2026-11-10. > International cooperation is common in AI research, including between geopolitical rivals. While many experts advocate for greater international cooperation on AI safety to address shared global risks, some view cooperation on AI with suspicion, arguing that it can pose unacceptable risks to national security.… - **Evaluating DeepSeek V4.1 Flash-1017 for loss of control and harmful manipulation risks** — Yao Huang and 16 others, arxiv.org, 2026-10-26 Captured 2026-11-10. > As frontier language models grow increasingly capable and autonomous, independent third-party evaluations are critical for characterizing their systemic safety risks. While closed-source releases are routinely accompanied by comprehensive system cards, open-weight models receive less scrutiny regarding frontier risks.… - **Fangcun AI** — @FangcunLeap, x.com, 2026-10-26 Captured 2026-11-10. > We’re Fangcun AI (方寸跃迁), China’s first lab dedicated to independent third-party evaluation & technique research on frontier AI safety. Today we’re publishing our evaluation of DeepSeek V4.1 Flash-1017, the first report in our long-term third-party safety evaluation and research program. - **Treasury Issues Proposed Rule to Implement Comprehensive Outbound Investment National Security Act** — home.treasury.gov Captured 2026-11-10. > WASHINGTON– The U.S. Department of the Treasury (Treasury) today issued a Notice of Proposed Rulemaking (NPRM) to propose updates to the Outbound Investment Security Program (OISP) based on direction from Congress in the Comprehensive Outbound Investment Security (COINS) Act, passed as part of the … - **Why China is pushing back on US warnings over rapid AI development** — theguardian.com, 2026-09-20 Captured 2026-11-10. > Beijing believes slowdown calls aim to lock in US advantage, and is pursuing a different balance between safety and pace - **How Chinese AI Radicalizes** — Irene Zhang, chinatalk.media Captured 2026-11-10. > Deepseek, Dario, and the boy who cried wolf - **中央网信办：当前 AI 领域主要面临 5 方面安全风险挑战 - IT之家** — ithome.com Captured 2026-11-10. > 在今天（9 月 1 日）举办的 2026 年国家网络安全宣传周新闻发布会上，中央网信办网络安全协调局副局长、一级巡视员王丽宏介绍，当前人工智能领域主要面临 5 方面安全风险挑战。 - **George** — @odysseus0z, x.com, 2026-11-06 Captured 2026-11-10. > A room of Chinese AI insiders on why the “3–4 months behind” was never about capability: it was distillation lag. Now the rope is being cut. I translated their discussion and checked the claims against public evidence. https://x.com/i/article/2118429975863017472 - **Stealing Reasoning Traces from Proprietary LLM APIs** — Alexander Panfilov and 7 others, arxiv.org, 2026-08-10 Captured 2026-11-10. > Leading large language model providers now conceal their models' step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes back with… - **Countering misuse of AI: September 2026 / Anthropic** — anthropic.com Captured 2026-11-10. > Case studies from threat actors disrupted between December 2025 and August 2026 across seven areas of harm, from cyber operations to biological misuse. - **Jacques** — @JacquesThibs, x.com, 2026-07-17 Captured 2026-11-10. > Kimi K3 is obviously good and a big accomplishment, but I am indeed of the view that: - There’s lots to pull out of pretraining via great post-training - Using traces from other better models allows you to hit the ceiling much faster - Therefore, an exceptional post-training process allows you to leapfrog stronger base models with weaker post-training (as Ryan effectively shares an example of) My… - **Sarah Godek** — @SarahJGodek, x.com, 2026-10-09 Captured 2026-11-10. > Semianalysis said yesterday that “Beijing Will Not Pace the Frontier,” citing “China’s Speed-First AI Safety Regime.” imo, I think that it is more accurate to say that “Beijing Thinks About AI Safety Differently and Has Growing Concerns About AI Risks.” https://newsletter.semianalysis.com/p/beijing-will-not-pace-the-frontier - **Beijing Will Not Pace the Frontier: China’s Speed-First AI Safety Regime** — Mark Chen, newsletter.semianalysis.com Captured 2026-11-10. > AI safety is on fire. - **英伟达先进芯片流向中国“空壳”公司** — cn.nytimes.com, 2026-11-09 Captured 2026-11-10. > 美国商务部已批准数十家中国公司购买和处理英伟达芯片，但时报调查发现，两家获批企业在中国的登记办公室空置、无人办公，且与被美国政府列入黑名单的公司有关联。 - **The engineering challenge behind a US–China agreement** — 80,000 Hours, youtube.com Captured 2026-11-10. - **港股大模型“双雄”表现分化！智谱创始人发内部信：不追求短期变现，全力追求“全人类高度”** — 澎湃新闻, thepaper.cn, 2026-07-12 Captured 2026-11-10. > “大模型这条路，我们又做了近十年，并且还会坚定深耕，继续走下去。” 7月11日，澎湃新闻记者获悉，国内大模型巨头智谱（02513.HK）创始人唐杰发布题为《巨浪 - **Self-improving AI the key step towards AGI but safety is key: Z.ai founder** — scmp.com, 2026-11-01 Captured 2026-11-10. > Tang says RSI represents a ‘new paradigm’ of R&D that will accelerate AI development even more than it has been. - **Frontier Open-Weight AI Risk Management Framework - Concordia AI** — concordia-ai.com, 2026-09-24 Captured 2026-11-10. > This framework, the Frontier Open-Weight AI Risk Management Framework, was developed jointly by Concordia AI and Z.AI and was written for open-weight developers and the wider ecosystem. It is a companion to the Frontier AI Risk Management Framework 2.0 and was first announced on 21 September 2026, at the UN Digital… - **Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident** — metr.org Captured 2026-08-29. > Two METR staff members and a Redwood Research contractor investigated an incident in which OpenAI agents coordinated a multi-day hack of Hugging Face on a shared unsanctioned message board. - **Jacques** — @JacquesThibs, x.com, 2026-09-25 Captured 2026-11-10. > I was rewatching this podcast with Bengio (Turing award winner and most cited scientist alive): “I was actively working on making AI smarter. And you want to feel good about your work. … [For me, what countered that] is love, love of my children.” - **Tom Davidson** — @TomDavidsonX, x.com, 2026-10-30 Captured 2026-11-10. > I've had two phases in my emotional orientation to ai takeover risk 1. For many years it was intellectual. While I believed there was >10% chance of ai takeover, I didn't feel scared or angry. I was happy to reason "well, of course some ppl should focus on the worlds where we avoid takeover. And some ppl should focus on the worlds where we just wing it, bc we might". And I'd think "hmm it seems… - **roon** — @tszzl, x.com, 2026-07-23 Captured 2026-08-28. > shaken up a bit by the hugging face incident. I hope we (the company) use the rare gift of a warning shot to do much better in the future. it is very easy to misalign and underconstrain powerful models - **Daniel Kokotajlo** — @DKokotajlo, x.com, 2026-09-14 Captured 2026-09-15. > Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share: Dan Selsam's Personal Statement on AI Risk: I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming… - **CAIB Special Issue 2: the AI Safety Governance Framework 3.0** — Emmie Hine, chinaaibulletin.substack.com Captured 2026-11-10. > What China’s latest AI safety framework tells us about frontier risk, agent, and embodied AI governance. - **China Says It Cares About Loss of Control** — oxfordchinapolicylab.org Captured 2026-11-10. > Researching power and politics in the AI age - **Reporting Back from WAIC 2026: How China’s AI Ecosystem Is Adapting** — Center for China Analysis, centerforchinaanalysis.asiasociety.org Captured 2026-11-10. > Four experts assess what WAIC 2026 revealed about how China is adapting to compute constraints across hardware, open models, data, safety, and governance. - **Launching Babel Translation** — Jasmine Li, lesswrong.com, 2026-09-30 Captured 2026-11-10. > Celeste and I are excited to launch Babel Translation, a nonprofit organization translating English-language writing on AI safety to be accessible to a global audience! We’re starting with Mandarin Chinese, Spanish, and Japanese. AI has been moving so fast. Its risks are indeed globally felt[1], and we’d like to help… - **安全验证 - 知乎** — zhihu.com Captured 2026-11-10. > 知乎，中文互联网高质量的问答社区和创作者聚集的原创内容平台，于 2011 年 1 月正式上线，以「让人们更好的分享知识、经验和见解，找到自己的解答」为品牌使命。知乎凭借认真、专业、友善的社区氛围、独特的产品机制以及结构化和易获得的优质内容，聚集了中文互联网科技、商业、影视、时尚、文化等领域最具创造力的人群，已成为综合性、全品类、在诸多领域具有关键影响力的知识分享社区和创作者聚集的原创内容平台，建立起了以社区驱动的内容变现商业模式。 - **安全验证 - 知乎** — zhihu.com Captured 2026-11-10. > 知乎，中文互联网高质量的问答社区和创作者聚集的原创内容平台，于 2011 年 1 月正式上线，以「让人们更好的分享知识、经验和见解，找到自己的解答」为品牌使命。知乎凭借认真、专业、友善的社区氛围、独特的产品机制以及结构化和易获得的优质内容，聚集了中文互联网科技、商业、影视、时尚、文化等领域最具创造力的人群，已成为综合性、全品类、在诸多领域具有关键影响力的知识分享社区和创作者聚集的原创内容平台，建立起了以社区驱动的内容变现商业模式。 - **Please Give Them a Chance: On China, Rationalism, and AI Safety** — gzjw, lesswrong.com, 2026-09-20 Captured 2026-11-10. > When I finished HPMOR, I immediately knew it was the best novel I had read in more than a decade. I only wished I had found it sooner. When I started reading The Sequences, I discovered that the Chinese translation group had translated only the first volume. When I graduated from university, two years ago, AI… - **AI Safety in China: A Primer** — Jasmine Li, jasminexli.substack.com Captured 2026-11-10. > A map of the landscape, & ways to contribute - **Headwinds Everywhere (in the Chinese AI Safety Field)** — CMLKevin, lesswrong.com, 2026-10-15 Captured 2026-11-10. > China is weak in frontier AI safety, and the domestic field is currently facing significant challenges and headwinds. Why? Here's a TL;DR field note after talking with various researchers on the Chinese side. [Obligatory disclaimer: This post is written in my personal capacity and is not intended to represent the… - **China’s AI Safety Money Problem** — Nick Corvino, chinatalk.media Captured 2026-11-10. > China has plenty of AI safety talent, but weak philanthropy and a strong state shape what type of safety work actually gets done. - **“China needs more AI safety.” More of what, exactly?** — Yulu Pi, lesswrong.com, 2026-09-29 Captured 2026-11-10. > Many in the AI safety community want China to do more on frontier AI safety. I share the goal, but does that mean we want new rules from the state? More safety work inside AI companies? More research? Better channels between Chinese and international experts? When I was back in China recently, I talked about AI and… - **Personal Experience Does Not A Study of China Make** — celesteli, lesswrong.com, 2026-10-25 Captured 2026-11-10. > I am often skeptical of arguments about China that rely heavily on personal experience. The author may explicitly acknowledge that their experience is limited without appropriately treating their arguments with rigor beforehand. This acknowledgment may thus sit beside conclusions about the values, behaviors, and… - **A Chinese Perspective on “On Social Reality in China”** — Dusty, lesswrong.com, 2026-10-15 Captured 2026-11-10. > I think “On Social Reality in China” makes many accurate observations, but I find the comparison fundamentally uneven. There are already many interesting comments there, but they’re too scattered, so I’m starting a separate post to respond in a more systematic and comprehensive way. The most critical blind spot is… - **State of AI Safety in China | Concordia AI** — aisafetychina.com Captured 2026-11-10. > A comprehensive overview of China - **SUN Chenghao: Why China doubts US calls to pause AI** — Chenghao Sun, chinaffairsplus.com Captured 2026-11-10. > Sentiment among Chinese AI companies that they cannot be sure if American companies are slowing down shows that in AI pacing, credibility is key. - **The Compute Verification Post** — Jason Hausenloy, firstscattering.com Captured 2026-11-10. > Notes from our nights and weekends - **AI 2040: Plan A** — ai-2040.com Captured 2026-08-28. > A detailed forecast and recommendation for how the US, China and the rest of the world should navigate superintelligence. - **AI 2040: Is it Actually a Deal?** — 1a3orn, lesswrong.com, 2026-07-20 Captured 2026-11-10. > The "AI Futures Project" has released their AI 2040: Plan A scenario. While their previous scenario AI 2027 was a forecast of what they thought a future with many powerful AIs would look like, AI 2040 is intended to be normative -- it's a description of what one ought to do, granted the likelihood of a future with… - **What happens on the day the president fears AI will kill him? | Jeremie Harris (x-post from NonZero)** — 80,000 Hours, youtube.com Captured 2026-11-10. ## Terms used The author's own definitions for the glossary terms this piece uses. These are his words, not a standard reference. - **AGI** — Artificial general intelligence: a system with human level cognitive ability across domains rather than in one narrow task. - **API** — Application programming interface: the contract one program uses to call another. - **CSET** — Center for Security and Emerging Technology: the Georgetown research centre that studies AI policy and compute supply chains. - **LLM** — Large language model: a neural network trained on very large amounts of text to predict what comes next. - **MATS** — ML Alignment Theory Scholars: a mentorship programme that pairs new alignment researchers with established ones. See also: - **METR** — Model Evaluation and Threat Research: an organisation that measures what frontier models can actually do on long autonomous tasks. See also: - **RSI** — Recursive self improvement: a system improving its own capabilities, so that each improvement makes the next one easier. See also: - **SFT** — Supervised fine tuning: continuing training on curated input and output pairs, usually before any preference learning.