AI Safety Cannot Be Defined by AI Companies Alone

On 29 September, the White House held a lunch meeting on artificial intelligence regulation and safety. Those present included Donald Trump, House Speaker Mike Johnson, Vice President JD Vance and numerous technology business leaders. Public reporting indicates that the gathering went well beyond the “six tech giants” described online. Nearly 20 executives attended, including Meta’s Mark Zuckerberg, Google’s Sundar Pichai, Nvidia’s Jensen Huang, OpenAI’s Greg Brockman, Anthropic’s Dario Amodei, and leaders from Microsoft, Amazon, AMD, xAI and other companies.

The meeting produced a voluntary document on AI safety. Trump described it as “morally binding”: a moral commitment rather than a regulatory system with clear legal force. Participating companies pledged to establish internal controls, auditing and safety mechanisms. The discussion also covered third-party reviews and the possible creation of an oversight body of about ten people.

These discussions certainly have value. But they raise a more fundamental question—who should define AI safety?

Technology companies must be involved. Without them, governments would struggle even to understand how far the most advanced models have progressed. Developers hold much of the information about unusual model behaviour, problems discovered in safety tests, capabilities that have not been publicly disclosed, and changes in training scale, computing resources, data and deployment methods.

The problem is not that technology companies have seats at the table. It is that, when they occupy most of those seats, AI safety can easily become an exercise in industry self-governance.

There is a clear structural problem here.

A company develops a technology and invests tens of billions of dollars in infrastructure, seeking market dominance, a higher valuation and more users. That same company then judges whether the technology is safe, which risks are acceptable and how quickly development should continue. We do not need to assume malicious intent. Even if everyone involved acts responsibly, the conflict of interest remains.

Consider gambling regulation. Casino operators should certainly be consulted, because they understand how casinos work. But if casino owners also largely determine the social harms of gambling, the criteria for addiction, the scope of regulation and which activities should be restricted, the institutional arrangement itself is flawed.

The same applies to AI safety.

Technology companies should explain what machines are capable of doing. They should not decide alone what machines are permitted to do.

These questions may sound similar, but they belong to quite different levels of discussion.

What tasks can a model complete? Can it bypass permission boundaries? Does it have cyberattack capabilities? Can it deceive human supervisors? These are primarily technical questions that experiments and testing can help answer.

But how much risk should society accept? Which tasks should be delegated to machines? How much autonomy should an AI system have? When must people retain the final decision? These are no longer questions for computer science alone.

They involve law, economics, psychology, sociology and philosophy.

Some of the AI governance arrangements now being established already recognise this.

The US National Institute of Standards and Technology’s AI Risk Management Framework was not developed by technology companies alone. NIST explicitly states that it emerged from an open process lasting 18 months, with contributions from more than 240 organisations across industry, academia, civil society and government. NIST also emphasises that AI risks affect individuals, organisations and society as a whole. Identifying and managing them therefore requires the perspectives of different participants throughout the AI lifecycle.

The UK’s AI Security Institute has established a government capability for testing frontier models. Since 2023, it has independently evaluated large frontier models, examining cyber capabilities, dangerous capabilities and other behaviours with implications for public and national security. Its purpose is clear: governments need empirical evidence independent of companies and cannot rely entirely on companies’ own safety judgements.

The European Union’s structure is particularly instructive. Under the AI Act, it has established a Scientific Panel of 60 independent experts to assess systemic risks from general-purpose AI models and advise the AI Office and national regulators. A separate Advisory Forum brings together industry, civil society and academia. Technical experts assess capabilities and risks, but governance is not left to technical experts alone.

This reveals an easily overlooked point about AI safety—a “third party” cannot simply mean another group of AI engineers.

AI safety has at least three levels.

The innermost level is technical safety. Can a model bypass permissions, carry out dangerous actions autonomously, acquire new cyberattack capabilities or behave in ways that cannot be reliably controlled? These questions require computer scientists, security engineers, red-team testers and genuinely independent model evaluation organisations.

The next level concerns social risks. Changes to employment and education, privacy and copyright problems, information pollution, and the concentration of wealth and power cannot be resolved by adjusting a few model parameters. Economists, sociologists, teachers, doctors, artists, workers and ordinary users should all be part of the discussion.

The outermost level, often overlooked, is philosophy.

Any discussion of AI safety eventually reaches a question—what exactly are we trying to protect?

We are protecting more than servers against attacks or preventing models from producing dangerous answers. We also need to ask how much human autonomy should be preserved, how far machine agency can develop while remaining subject to human constraints, who bears responsibility as AI participates in more social decisions, and what place human judgement retains as machine cognitive abilities begin to exceed those of ordinary people.

No technology company has an inherent right to answer these questions.

Even computer science cannot answer them on its own.

They concern more than how to build AI. They concern what kind of society we want to live in once AI is part of it.

This is why I remain wary when a group of technology giants is brought together to discuss AI safety. Their participation is necessary, but they can too easily become the centre of the entire discussion. In media coverage, AI governance often appears as a president, government officials and the CEOs of a few of the world’s largest technology companies sitting together. Ordinary people, who will bear AI’s long-term consequences, are absent from the table.

AI differs from many earlier technologies.

A nuclear power station’s risks are concentrated mainly around the station. A medicine’s risks are concentrated mainly among those who take it. AI, however, is gradually entering education, healthcare, the military, justice, finance, media, research, business management and personal life. It is becoming new infrastructure for society as a whole, extending well beyond a single industry.

Its governance therefore should not be confined to a single industry either.

A sound arrangement would distinguish several roles.

Developers provide models, data and internal test results. Independent technical institutions repeat tests and verify findings. Governments establish legal responsibilities and public institutions. Different social groups provide evidence of actual impacts. Philosophy, the humanities and the social sciences help examine value boundaries that engineering tests cannot resolve.

There should even be some tension between these roles.

Oversight should not depend on every participant having perfectly aligned interests.

It is natural for AI companies to want technological development to continue. It is natural for investors to seek returns. Governments’ desire to maintain national competitiveness is also understandable. Precisely because these incentives are so strong, an independent counterweight is needed—one whose direct interests are not tied to the speed of the technology race, company valuations or market share.

What AI safety lacks is not more statements about safety from technology CEOs.

It lacks a social institution capable of maintaining distance from these companies while genuinely understanding the technology.

Companies should certainly have seats at the table.

But they should not occupy the whole table.