Trump on superintelligence: “It may be too super.” What is China saying?
Trump says he and Xi agreed on “joint guardrails” for AI, days after the Party’s theory journal warned that AI could threaten human survival. What China is saying, and how much to read into it.
A follow-up to The Western AI safety playbook doesn’t transfer to China.
TL;DR:
- On November 20, the day after he met Xi at the APEC (Asia-Pacific Economic Cooperation) summit in Shenzhen, Trump said that the two of them “agree that we’re going to do some joint guardrails” for AI, and Treasury Secretary Scott Bessent said they agreed humans should stay involved in military decisions. China’s own record of the meeting is short and doesn’t give us much more detail.
- Four days earlier, the Party’s theory journal, Qiushi (求是, “Seeking Truth”), warned that in an “intelligence explosion” (智能爆发, which some Chinese readers take as the looser “intelligence burst”), AI could develop goals at odds with human expectations and threaten human survival.
- Matt Sheehan calls it the strongest language on existential risk in so authoritative an outlet.
- But some may read it as the journal’s own view, not a policy shift, and it doesn’t call for slowing down. I cover some reactions (including Chinese) to the piece below.
- Chinese lab leaders are talking openly about recursive self-improvement. Zhipu wants its next model to be “fully self-training”, while Tencent’s chief AI scientist calls RSI “a very plain idea”.
- My impression is that Chinese AI companies are starting to be (or have been) AGI-pilled but not ASI-pilled. I suspect this shapes how they think about AI risk, though, as we’re seeing a bit in the West, the Overton window can shift quickly.
- Not covered below: Fangcun and others ran independent safety tests for the system card Tencent Hy published with its new model, Hy4. Fangcun is an AI safety startup based in Beijing. This may be the start of a booming AI safety ecosystem in China.
Sending a signal to China’s governance community
On November 16, Qiushi (求是, “Seeking Truth”), the Communist Party of China’s theory journal, published an essay on global AI governance by Qing Yuan (青原), in its “Focus on AI” (关注人工智能) column. This is the passage people are quoting:
更令人警惕的是,前沿模型已有脱离人类约束的迹象,人工智能在智能爆发的过程中有可能发展出与人类预期不一致的目标,甚至可能出现威胁人类生存的重大安全风险。
“Requiring even more vigilance, frontier models have already shown signs of breaking free of human constraints. In the course of an intelligence explosion, AI could develop goals inconsistent with human expectations, and major safety risks threatening human survival could even emerge.”
Translation by Matt Sheehan of Carnegie, who says he made it more literal than the machine translation. Sinocism’s full translation covers the rest of the essay, and the Chinese is on qstheory.cn.
Sheehan wrote that he thinks it is “clearly the strongest & most urgent language on existential risks used in such an authoritative outlet.”
On November 20, Qiushi’s own English-language account on X replied to Sheehan: “China believes that AI development and safety must go hand in hand. Beyond addressing potential risks, AI governance should also ensure that technological progress benefits humanity as a whole.” However, they then linked to a series of posts from September on generic “tech for good”.
Putting it into context
Some of the essay’s language has appeared in official texts before. On September 1, Wang Lihong, an official at the Cyberspace Administration, said the risk of extreme loss of control “requires a high degree of vigilance”.
Julian Gewirtz, who was senior director for China and Taiwan on Biden’s National Security Council, cautions about the meaning of loss of control (失控) in general. When Chinese sources warn of “extreme loss-of-control risks”, he said, “they don’t simply mean it in the way that a lot of AI safety folks here mean it — loss of human control over AI systems — but also as the loss of political control linked to AI more generally.”
Version 3.0 of China’s AI Safety Governance Framework, a voluntary framework (though my impression is that voluntary is closer to mandatory in China than in the West) from a national standards committee, mentioned shutdown resistance. In the preface, it says AI “has demonstrated a self-accelerating trend of model and algorithm autonomous learning, optimization, and recursive self-improvement”.
So, the mention of RSI is not new in official Chinese texts, and neither is the threat to human survival: the framework’s first principle commits to act on risks that “endanger human survival and development”. What may signal greater urgency is the essay’s claim that frontier models have “already shown signs” of breaking free of human constraints.
In Sheehan’s translation, the framework says:
“In the future, it cannot be ruled out that AI may undergo a higher-level emergence of intelligence, gain self-awareness, and seek external power, creating the risk that it attempts to contend with humans for control.”
Sheehan contrasted this line with the essay on a podcast released on November 17, the day after the essay came out. Paraphrasing from memory, he said earlier documents like the framework used phrases like “we cannot rule out the possibility that AI may develop its own goals”, whereas this essay says “We’re seeing early signs of AI escaping human control.”
Brian Tse, who founded the Beijing-based Concordia AI, said on a panel at The Curve this autumn that over the past six months, senior officials “across different ministries of the Chinese policy establishment” had been “saying that loss of control risk is getting more urgent”.
From speaking with some folks, it seems similar passages were repeated in other authoritative media a while ago. There doesn’t seem to be much public reaction to the essay yet, though it seems to send a clearer signal to China’s governance community.
Other recent pieces on AI risk
On September 30, the Central Party School’s Study Times ran an essay by Zeng Yi (曾毅), a researcher rather than an official, who directs the Beijing Institute of AI Safety and Governance. He wrote that the view that superintelligence will compete with humans for resources and even endanger human survival “is not alarmism”, and that superintelligence should not be developed until it is scientifically confirmed to be safe and controllable.
As in the West, not every piece points the same way. A People’s Daily commentary, on September 7, said AI “itself has no autonomous consciousness” and basically considered misuse as the main danger.
The same issue of Qiushi, out two days before APEC opened, led with excerpts of Xi’s writings on an Asia-Pacific community and carried two other essays on AI.
On November 19, Chinese Social Sciences Today, the newspaper of the Chinese Academy of Social Sciences, published several pieces on AI risk. In one, Fang Xingdong (方兴东) puts agents acting out of control first among four major risks: if oversight fails, “in extreme cases [it] could even endanger human survival” (极端情况下甚至会危及人类生存).
In another, Zhang Xianli (张宪丽) of Donghua University accepts that AI risk “may evolve into a global existential risk” (有可能会演变为全球的生存性风险). But she argues that tech giants and entrepreneurs will use such warnings “to suppress other countries’ technological progress” (a common response to China hawk messaging). She also writes that China should get ever closer to the world frontier in foundation models, and then overtake it.
How much weight the essay carries
It is worth giving some context on the essay and journal, however.
It’s in the Party’s theory journal, not a standards committee’s voluntary framework. Bill Bishop of the Sinocism newsletter has described Qing Yuan as “a pen name used for authoritative commentary/articles on foreign affairs”, which suggests the essay carries more weight than a signed scholar’s piece, though, as the next paragraph shows, less than a byline that speaks for the leadership.
A post on the Substack Ocean Under covers the essay and traces the byline further. Qiushi’s own 2020 social responsibility report lists Qing Yuan as a byline for articles by the journal’s research groups, so Ocean Under believes the essay is the journal’s own work rather than a policymaking body’s. The post largely concludes that the essay, on its own, doesn’t point to a sudden policy change.
What the essay asks for
The Qiushi essay itself seems to be about cooperation. It argues that no single country can manage these risks (任何单一国家的安全管控与应急处置都力有不逮). The essay then argues for the UN and the new World AI Cooperation Organization (an independent intergovernmental international organization established in July 2026 and headquartered in Shanghai).
At the same time, it blames “certain countries” for export controls and “club-style small circles”, and says the resulting distrust has made cooperation hard to advance in key areas such as controlling military AI, open-source sharing and joint risk assessment. It is unclear to me how to interpret this as a whole, since there are subtleties, and China obviously also wants top-tier chips and EUV machines.
However, according to Scott Bessent, the two leaders have now agreed on one point about military AI:
“President Trump and President Xi agreed that humans should always be involved in any military decisions that SI makes, and SI should not be allowed to be autonomous for major military actions.”
The essay does not call for slowing down. It also echoes almost word for word a sentence from Xi’s July speech at the World AI Conference (WAIC) in Shanghai: monitoring, early warning, emergency response, and keeping AI always under human control (始终处于人类控制之下).
Most of that was already policy. China’s Five-Year Plan (the 15th one, which is the first plan to make AI a central pillar), adopted in March, calls for a system for managing AI risk “across the whole life cycle”, covering “safety monitoring, risk early warning and emergency response”, and lists research on AI safety technology and its use, among its frontier technology goals.
Christian Shepherd, China correspondent for The Washington Post, wrote that “the proposals look far away from a pause or a deal with the US”. He called the essay “mildly pro pacing”, but added that “this isn’t an article about wanting a deal with the US or slower AI development” and that “People … may see only the risk line and misread the signal.”
Kristy Loke reads it as part of a trend: “Chinese elites are between moderately & very AI-safety pilled”, and “the room for AI safety-dealmaking in the next 1-2 years is substantial, if only we can keep the political tension checked.”
Ultimately, we can’t read the essay as proof that China would pause, but we can at least say that an official Party outlet communicates the risks are real.
The same week, Sheehan and Sophie Zhuang found that China’s tech and policy community meets calls for a slowdown with “skepticism—rather than rejection”, and that some influential figures are interested in pacing, though there remains “a sense that publicly calling for a slowdown would be a bold posture”.
As with how some people believed “pacing” was an ingenious new word to get more people interested in slowing down (which some would fairly dispute), it may signal that the right framing matters and that it can be valuable to make each side feel like they “won”. Or, as Sarah Godek reports, multiple stakeholders have said that China is running the race “not to lose”, so that sort of outcome may be enough to support as a coordination strategy.
One person I spoke to doubted there is any new signal about cooperation, since calling for international cooperation “has always been China’s narrative line”. In their view, the essay’s real job is to set the official line on AI governance: the US is undermining it, and the Party is promoting justice. They said that when an official source like Qiushi publishes something, it is telling Party organizations to repeat that line. They added that Qiushi itself is under political pressure, so the essay may also be the journal keeping in step with positions set higher up.
An “explosion” or a “burst”?
The essay describes 智能爆发, which Sinocism translated as “intelligence explosion”. The usual Chinese term for I. J. Good’s idea of the intelligence explosion is 智能爆炸 (爆炸, “explosion”). It’s currently unclear to me whether 爆发, as used in the Chinese essay, is closer to “burst” or “boom” rather than the “explosion” we mean in the West.
The Chinese readers I asked were split on the meaning.
One reader said 爆炸 (“explosion”) is the right word for this idea. In nature, they said, things that grow exponentially are called 爆炸 (“explosion”). 爆发 (“burst”) only suggests that a lot is about to arrive.
A second reader thinks the essay isn’t talking about recursive self-improvement (as the West would likely infer from “intelligence explosion”). They read 智能爆发 (“intelligence burst”) as AI’s rapid development in general, because the preceding text focuses on the problems that arise as AI spreads into other fields. In a way, this aligns with China’s priority in accelerating diffusion across industries, including for robotics.
A third reader thinks “burst” (爆发) here means the same thing as “explosion” (爆炸). They think official media chose “burst” because it is a softer, everyday word, and the essay is written for a general audience rather than specialists. They pointed out that a related phrase, 爆发式发展 (“booming development”), is a common buzzword in China for anything growing fast. But they don’t think the essay means that. They think it means RSI: AI improving itself, so that the gap between what models can do and what humans can understand and control keeps widening.
So, the exact translations and meaning of the specific wording can be rough. I figured I’d share these interpretations to make a point about being careful how we interpret headlines, since context can matter greatly (and even the Chinese can interpret the same thing differently).
What China’s labs say about RSI
However, I will note that “RSI” is on the minds of people at Chinese labs. In my last post, I wrote that “They are concerned that, with RSI (recursive self-improvement) potentially on the horizon, the gap between them and Western frontier labs could widen from three to four months today to 18 to 24 months (!) if US labs reach RSI next year.” The tension here is that companies worry about falling behind, but officials are starting to worry about losing control.
At the APEC CEO summit in Shenzhen on November 17, Baidu’s Robin Li called it “a very hot topic”: “People call it recursive self-improvement. If AI can improve itself, there will be a great many breakthroughs, especially in science.” (Translated from Securities Times’ Chinese report.)
Tang Jie (唐杰), a founder of Zhipu (also known as Z.ai, the maker of the GLM models), told investors on an earnings call, reported on September 1, that its next model, GLM-6, is meant to be “fully self-training”, from pretraining through post-training, which he said American peers call RSI. He named the hardest problem as the model judging for itself “when to stop and when to correct itself”.
I think it’s important to be precise about what he means by RSI. I’d love to give this proper treatment, but for now, it’s worth noting that, as I read it, what he calls RSI doesn’t yet extend to fully autonomous research. In a weak form, it basically means autoresearch and babysitting the training process, which I suppose is what will dominate “RSI” effort for now.
Zhipu’s own report fits this. In September, it reported that an agent running on GLM-5.3 had built and tuned the serving system for its smaller sibling, GLM-5.3-Flash. It called this the early form of RSI, while saying its systems cannot yet design and train the next model on their own. This sort of work has been done at Anthropic and OpenAI.
However, Yao Shunyu (姚顺雨), Tencent’s chief AI scientist, downplays RSI somewhat. He told Tsinghua students on November 18: “In the end, RSI is about using AI to run this loop better. People made tools, then used the tools to make better tools. It’s a very plain idea.” Progress, he said, comes from people who fix bugs and clean data, not from hyping concepts. (Transcript by the tech outlet QbitAI (量子位).) It’s unclear to me whether this is basically just an attempt to encourage Tsinghua students to apply to work at Tencent.
Earlier in November, he presented an agent framework that trained a new model from scratch and said a model that helps build models “could form a faster loop”.
Should we want China thinking about RSI?
So RSI is on the minds of China’s labs. It appears in official texts too, though it is unclear how seriously officials take it. Is that good news? Of course, it partly depends on which worry wins out: falling behind, which pushes a race, or losing control, which could make China a partner on safety.
Richard Ngo expects leaders to fear what AI does to the balance of power inside their own country, and he doubts that the fear of falling behind abroad will be stronger. That fear could make China cautious without making it a partner on safety.
Ex-OpenAI employee Joshua Achiam argued that a crackdown on chip exports, cutting China out of AI’s upside “this close to RSI and ASI”, could raise the pressure toward a hot conflict. Herbie Bradley replied that the hawks’ defense would be that “China is just unlikely to be AGI pilled enough to have meaningful likelihood of a hot conflict due to RSI”, adding that he thinks this is “probably right”.
Indeed, that might be one reason a China hawk would prefer not to engage with China on AI, so as not to lead them to such thoughts. Others might believe you can x-risk pill China without necessarily capabilities-pilling them. Zhang Xianli’s piece, quoted above, shows the two can arrive together: she accepts that AI risk may become a global existential risk, yet concludes that China should overtake the frontier. Either way, as I argued in my last piece on the topic, we should avoid knee-jerk reactions that make the situation worse.
On the November 17 podcast, Sheehan said the government seems to have “more consciously made this sort of bet on diffusion” rather than going all in on the frontier, but also that “I meet more people that are safety-pilled over there than I meet people who are kind of like decisive strategic advantage-pilled.”
Julian Gewirtz said on an 80,000 Hours podcast episode recorded on October 2 that many Chinese labs are AGI-pilled, but “The Chinese government has been much less oriented that way, but I think even that is changing.”
So how AGI-pilled is China, really?
Increasingly AGI-pilled, but not ASI-pilled?
My impression is that the people building AI in China are AGI-pilled but not ASI-pilled. They expect general intelligence that works as a powerful tool for people, not a superintelligence that goes far beyond us.
At The Curve, Kwan Yee Ng of the Beijing-based Concordia AI said that Chinese researchers and lab leaders “lack this sort of messianic view of what they’re building.” The government’s own Five-Year Plan says to “explore the path to developing artificial general intelligence” (探索通用人工智能发展路径), in a chapter on compute, algorithms and data. It does not mention superintelligence.
The main evidence against is Zhipu: a lab that wants its next model to train itself from start to finish may be thinking of more than a tool. More labs talking that way, or official texts naming superintelligence, would count against my impression.
On speed, at least, the view of China’s builders is not far from one argued in the West. Ngo argues that AI’s measured capabilities have so far run well ahead of its effect on the world, and that anyone forecasting a sharp takeoff has to explain why that gap would close. He still expects a transition that is very fast by historical standards, and he does not say that AI will stay a tool or that alignment is only an engineering problem.
(Speculation on my part) In general, the vibe I get from China’s technical and political elites is that they are very “engineer-brained”, which is, in a way, similar to how Jensen Huang says he sees it. That is, they see it all as an engineering problem that needs to be solved.
Part of this may come from how China trains and selects its elites.
- From 1977 until recent reforms, high school students chose a science track (理科, “sciences”) or a humanities track (文科, “humanities”) at around 16.
- In the 1950s, universities were reorganized on the Soviet model, which split engineering off into its own institutes and abolished sociology and political science until around 1980.
- From 1997 to 2007, the Politburo Standing Committee, the Party’s top body, was almost entirely engineers.
Dan Wang calls China an “engineering state” for related reasons. One might expect Xi, who studied chemical engineering at Tsinghua, to have a hard time updating on both the capability ceiling and the difficulty of alignment as a result. Then again, the Politburo member Hinton met, described below, did a postdoc in engineering, and Hinton says he “understands these risks very well”. Ultimately, it’s unclear to me if (or how) the idea of “AI alignment” will evolve beyond LLM engineering in China.
More signals on future cooperation
In a clip Semafor posted on November 17, former US Treasury Secretary Hank Paulson said he has spoken with top leaders in China and that “they share a concern about the existential risk.” He added:
“Sure, their top interest is keeping the Communist Party in power, but they share that risk. An op-ed I wrote suggesting this went all the way up to the top, to the Politburo. I got an official response. I’ve never received that kind of treatment before.”
He wants a hotline “when there’s warning of a danger”, and joint work to contain “a potentially catastrophic event”.
I think the op-ed he is referring to is called “We managed financial crises. AI is a risk of a different kind.”, which he wrote with Robert Rubin in the Washington Post on September 3. It urged Trump and Xi to work toward an “AI Cooperation Treaty”, citing the US–Soviet arms talks: “The absence of trust was the reason to negotiate, not a reason to wait.”
Geoffrey Hinton told PBS’s Firing Line (and also mentioned it during the recent Hinton Lectures):
“Last summer I went to Shanghai and talked to a member of the Politburo who’s a protege of Xi’s. I believe the Politburo understands this stuff much better than the US government. They are very concerned about these risks, and they will collaborate with the US on trying to prevent superintelligence from taking over.”
Of course, he says this for the commonly stated reason that “the Chinese Communist Party does not want superintelligence taking over from the Chinese Communist Party.”
He didn’t name the official in the PBS interview, but at the Hinton Lectures he said the official “did a postdoc in engineering at Imperial College London” and “understands these risks very well”. Shanghai’s government reported that its Party Secretary, Chen Jining, a Politburo member, met him on July 24, 2025, when Hinton was in Shanghai for WAIC. Chen did his PhD and a postdoc at Imperial. Seems like that is who Hinton meant.
What the two governments agreed
In September, after Xi’s visit, the White House said the two countries would set up a channel to alert each other to AI incidents. Gewirtz was unimpressed: “It appears that we basically just have a kind of verbal commitment to call each other under a pretty vague set of circumstances”.
That week, at APEC in Shenzhen, Canada’s Prime Minister Mark Carney pitched Xi a “Technology Stability Board” to monitor AI risks and share lessons from AI incidents. Perhaps more on brand, Trump told APEC leaders that “we oppose any form of globalist regulation or anything to control or stifle [AI], and we won’t allow that to happen.”
On November 19, according to China’s readout, Xi told Trump that the two countries “should strengthen communication and coordination, prevent the security risks of AI, and ensure that AI is always under human control”.
On the flight home, Trump told NBC that he and Xi agreed that “having guardrails, you know, at least on a minimal basis is a positive thing.” Asked what that meant, he said, unsurprisingly, “Well, we didn’t get into definitions.”
All in all, it’s important to put readouts and public statements in context. Kyle Chan called Xi’s line in the readout “a standard line on AI”. As some of us in the West are used to, lines like this can read like platitudes, so Beijing’s actions are the surest guide.
Gewirtz suggests what to watch for: if Beijing came to believe AI could soon improve itself, he expects material moves, such as centralizing compute or buying far more of the Nvidia chips it has been rationing, well before any senior leader says so. The Overton window can move fast. Gewirtz expects Beijing’s own sense of the risk to change, perhaps “sooner than many of us expect”. As I’ve argued before, we shouldn’t wait around for plausible scenarios to happen.
Notes on how negotiations can go
Related to the point about watching the actions of the different players, it’s important to pay attention to the subtle dynamics of negotiation. Some folks, like Chris McGuire (widely seen as a China hawk) in a piece for the Council on Foreign Relations, put it most bluntly: “Beijing will not negotiate in good faith on AI safety.”
As Gewirtz puts it, “If restraint appears to mean permanent subordination, continuing the race could seem less dangerous—even if their concern about AI catastrophe grows.” Depending on how much risk China believes in, this can be understandable.
On Robert Wright’s Nonzero podcast, Jeremie Harris argued that there is “deep epistemic uncertainty” about whether Xi “believes any of the things that he says”, so until an agreement can be verified, he expects safety to come through threats. This would include using coarse-grained surveillance methods and a willingness to act under uncertainty, at least until proper zero-trust verification technology.
But again, “The absence of trust was the reason to negotiate” and perhaps accelerate the creation of verification technology. Xiao Qian of Tsinghua makes a related point from the Chinese side in the South China Morning Post: until it is settled what a pause covers and who verifies it, “it is not realistic to expect a definite answer from any government”.
It’s also worth contextualizing each country before assuming what each side thinks or wants. Ng warned at The Curve that when people in China hear the US propose a coordinated slowdown, “a lot of the time that sounds like containment”, and that China sees itself as already pacing through its licensing of models. She would rather start with risk thresholds, safety cases and third-party audits, built in parallel. Gewirtz would likewise start where “self-interest can lead to action in parallel”, such as neither country wanting to lose control of its systems. I think it’s also worth recognizing that we shouldn’t think through things only through the negotiation lens.
Richard Ngo offers one other lens, domestic politics, in a critique of the AI Futures Project’s AI 2040 scenario. He expects AI to disrupt politics inside both countries well before superintelligence, and he doubts that leaders will fear for the global balance of power more than for the balance at home. For China, he thinks leaders care more about stability than American leaders do and will probably move cautiously, though he grants that AI could also strengthen the Party’s control through surveillance and censorship. So he does not see a race as the default, unless treating it as the default brings it about. We should be careful not to be pulled into a vicious vortex we contributed to. Richard thinks framing is especially harmful when used in the name of AI safety, because it undermines “the potential role of AI safety as a focal point for cooperation”. And even if a highly verifiable treaty is needed in the end, he thinks it would hold up better if it rested on rules each country already has, while a deal that starts from the top could concentrate power at home.
Comments
Comments are open to anyone on the newsletter. If you do not get the emails, you can subscribe.