AI Self-Regulation: Should Big AI Be Allowed to Police Itself?

Quick answer: AI self-regulation should not mean allowing leading AI companies to police the entire field. Leading AI companies should share safety information and test one another’s models. But they should not be allowed to define the rules, judge compliance and determine which AI systems reach the public. That would not be neutral regulation. It would turn existing technological power into private governing authority.

Elon Musk has proposed that leading AI companies review one another’s most advanced models before release. He argues that rival developers possess the technical knowledge and commercial incentive to identify risks that conventional regulators may miss.

Musk is not proposing that AI companies formally govern the entire field. The danger arises if peer review becomes the primary mechanism through which they define safety and control market access. 

The proposal sounds practical. The companies building frontier AI understand their systems better than almost anyone else.

That is also the problem.

If the same companies build the models, identify the risks, write the standards and decide whether competitors have passed, they become more than developers. They become inspectors, judges and gatekeepers.

The central danger is that a small group of dominant laboratories could end up governing AI on behalf of everyone else. 

Why does AI self-regulation sound so convincing?

Every technically complex industry can make a persuasive case for self-regulation.

Its companies employ the specialists. They possess confidential information. They understand how their products behave. External institutions often move more slowly and lack comparable expertise.

Musk’s proposal draws strength from this knowledge gap. A rival AI laboratory may recognise a dangerous model capability faster than an outsider. Regular safety discussions could also prevent companies from discovering the same vulnerabilities separately.

But expertise answers only one question: who understands the technology?

It does not answer the more important questions: whose interests will shape the standards, what information will be disclosed and who will be excluded by the rules?

A company can understand a danger perfectly while still having strong reasons to minimise it, redefine it or delay acknowledging it. Its survival may depend on releasing a model before competitors, satisfying investors and protecting its market position.

Technical knowledge does not remove that conflict. It makes the conflict harder for outsiders to detect.

Amba Kak, co-executive director of the AI Now Institute, has stated the problem directly:

“A closed-door deliberation with corporate actors resulting in voluntary safeguards isn’t enough.”

Her criticism matters because voluntary commitments tend to address problems companies can accept without changing the foundations of their business models. AI self-regulation may therefore recognise certain technical dangers while leaving questions of competition, discrimination, privacy and corporate power outside the room.

What does history tell us about industries policing themselves?

History does not show that every private standard fails. It shows that voluntary oversight becomes unreliable when public welfare conflicts with the industry’s commercial interests.

The early internet provides a direct technology example. For years, companies argued that voluntary privacy policies could protect users without slowing digital innovation. Yet a 1998 Federal Trade Commission review found that only 14% of surveyed commercial websites disclosed their information-collection practices, while roughly 2% offered a comprehensive privacy policy. The FTC concluded that voluntary industry efforts had “fallen short of what is needed to protect consumers.”

Credit-rating agencies reveal a second danger: private reviewers can become market gatekeepers. Before the 2008 financial crisis, issuers paid agencies to rate the securities they wanted to sell. SEC Commissioner Caroline Crenshaw has called this a “fundamental conflict of interest.” Agencies had incentives to satisfy paying clients, while investors and the wider economy relied on their judgments.

The lesson is not that AI will reproduce either case exactly. It is that self-regulation changes the position of private companies. They cease to be only market participants and begin deciding what counts as trustworthy, compliant or safe. 

Once their approval becomes necessary, commercial judgment begins to function like law.

Could AI safety standards become tools of market control?

The greatest risk is not simply that AI companies might overlook an unsafe model. It is that they could use safety standards to control the direction of the entire industry.

A group of frontier laboratories could decide which evaluations are mandatory, how much testing is sufficient and what level of computing security every developer must maintain. Those requirements may appear neutral while favouring companies that already possess enormous budgets, specialised staff and access to advanced infrastructure.

A large laboratory can afford extensive evaluations, legal teams and secure data centres. A university, start-up or open-source research group may not.

When dominant developers define what “safe” means, they can create standards that smaller companies and independent researchers cannot afford to meet. Coordination then begins to resemble market control. 

The credit-rating precedent matters here. Rating agencies did not manufacture the financial products they assessed, yet their approval became a commercial passport. An industry-controlled AI safety process could become even more conflicted because the laboratories issuing judgments would also compete against the companies being judged.

A rival could identify a genuine safety problem. It could also benefit commercially when another developer’s release is delayed, restricted or discredited.

Both things can be true at once.

Are we sleepwalking into a technological oligarchy?

Frontier AI is not developing in an open field where every company possesses equal power.

The largest laboratories are closely tied to major cloud providers. An FTC study of partnerships involving Microsoft and OpenAI, Amazon and Anthropic, and Google and Anthropic identified more than $20 billion in cumulative investment. It found that these relationships could affect access to computing resources and engineering talent, increase switching costs and give major cloud companies privileged access to sensitive technical information.

These firms already influence several layers of the AI economy: cloud infrastructure, computing capacity, models, distribution and consumer applications.

Giving leading companies authority over safety standards would add another layer—the power to define legitimate AI development.

Former FTC Chair Lina Khan framed the central choice directly:

“Will a handful of dominant firms concentrate control over these key tools, locking us into a future of their choosing?”

An AI oligarchy would not require a secret agreement between technology executives. It could emerge through an apparently responsible process.

A few companies would meet regularly. They would develop shared standards. Their evaluations would become trusted by investors, businesses and public institutions. Over time, their approval would determine which products entered the market and which developers were considered irresponsible.

The system might still contain competition. But it would be competition managed by the most powerful competitors.

Would consumers receive the best AI or only what AI gatekeepers permit? 

AI self-regulation is often presented as a way to give consumers safe products without blocking innovation.

But safety is not the only value at stake. Consumers also benefit from competition, affordability, openness, privacy, choice and the ability to move between providers.

A private AI-governance system could weaken those benefits.

Dominant laboratories may prefer closed models because closed systems protect intellectual property and recurring revenue. They may oppose forms of open development that allow users and smaller companies to run, modify or study models independently. They may design safety conditions that keep customers inside proprietary platforms.

The result could still look innovative. Consumers might receive faster assistants, better interfaces and increasingly capable subscription products.

What they may not receive is meaningful control.

They could become dependent on a small number of firms that determine how AI may be used, what features are permitted and how much access costs. Smaller competitors might disappear before they can offer different models of ownership, privacy or consumer choice.

The danger is not that every restriction will be dishonest. Some limits will address real risks. The danger lies in giving commercially interested gatekeepers the final authority to distinguish legitimate protection from inconvenient competition.

A safe gate is still a gate.

Does the promise of an “age of abundance” hide the ownership question?

The abundance narrative does more than predict prosperity. It helps establish the leading AI companies as indispensable custodians of the future. Once that image is accepted, allowing them to govern their own technology can begin to look inevitable.

Musk presents AI and robotics as technologies capable of producing extraordinary abundance. Goods and services could become cheaper. Productivity could rise. Work itself could eventually become optional.

That vision may prove partly correct. But the language of abundance can hide the institutional questions that determine whether technological gains become public prosperity.

Who will own the models?

Who will control the data centres, chips and energy infrastructure?

Who will set the price of access?

Who will decide which communities, workers and businesses receive the benefits?

Lower costs do not automatically create fair access. Concentrated ownership can produce technological wealth alongside severe inequality. 

Economists Daron Acemoglu and Simon Johnson place the issue in historical context. Technological progress can create enormous wealth without distributing that wealth broadly. Their warning about the current direction of AI is direct:

“The current path of AI is neither good for the economy nor for democracy.”

Their deeper point is not anti-technology. It is that innovation does not arrive with a built-in system for sharing its gains. Institutions, ownership and power determine who benefits.

The promise of abundance should therefore increase scrutiny of concentrated power, not reduce it. 

That is precisely when their power deserves the most scrutiny.

What role should AI companies have in AI governance?

AI companies must participate in safety work. Excluding their expertise would be reckless.

They should exchange information about serious vulnerabilities. They should permit credible external testing. They should report dangerous capabilities and provide evidence about how their models were evaluated.

But participation is not ownership.

The companies making AI should not possess final authority over the standards used to judge it. They should not decide alone which competitors can enter the market, which research is permitted or what consumers are allowed to receive.

Their technical knowledge should inform decisions. It should not give them sovereignty over those decisions.

Musk’s proposal begins with a sensible idea: competitors can help identify one another’s weaknesses. The danger begins when peer review quietly becomes AI self-regulation, and AI self-regulation becomes gatekeeping.

Society could then wake up to find that a few companies do not merely produce its most important technology.

They govern access to it.

Key takeaways

• AI self-regulation cannot make commercially interested companies neutral judges of their own conduct.
• Historical self-regulation has often weakened when consumer welfare conflicted with commercial incentives.
• Industry-defined safety standards could exclude start-ups, universities and open-source developers.
• Frontier laboratories already benefit from concentrated access to capital, cloud infrastructure, compute and technical talent.
• Allowing those companies to judge competitors could turn market dominance into private governing power.
• The promise of AI abundance does not answer who will own, distribute or control its benefits.
• AI companies should contribute evidence and expertise without becoming the final gatekeepers of AI.

Frequently asked questions

What is AI self-regulation?

AI self-regulation is a system in which companies voluntarily create safety standards, review models, share risk information and determine acceptable development practices without independent authorities controlling the process.

What did Elon Musk propose?

Musk proposed regular safety discussions between leading AI companies and peer review of advanced models before release. He argues that rival developers are better positioned to recognise technical risks in frontier systems.

Why is AI self-regulation a conflict of interest?

The companies evaluating safety would also be competing for customers, investment, computing resources and market leadership. A decision presented as a safety judgment could also benefit the reviewer commercially.

How could AI self-regulation create an AI oligarchy?

A small number of powerful companies could define safety standards, influence market access and determine which models are considered acceptable. Their existing economic power would expand into rule-making authority.

Could AI safety rules harm competition?

Yes. Standards that require expensive infrastructure, extensive testing or access to scarce computing resources may be manageable for frontier laboratories but unaffordable for smaller developers and independent researchers.

Would consumers benefit from industry-led AI regulation?

They might receive safer products in some cases. But they could also face fewer providers, less open technology, greater platform dependence and reduced control over how AI systems are used.

Is the argument against all cooperation between AI companies?

No. Companies should share genuine safety information and help test advanced models. The objection is to allowing those companies to become the primary or final judges of their own conduct and that of their competitors.