Anthropic let METR inside: Independent AI evaluation depends on access and publication rights

Short answer

Anthropic has given six METR staff company hardware, office space, and access to internal tools roughly comparable to its own risk assessment teams. Some areas remain excluded, including customer data, attorney-client privileged information, and details of model architectures. METR can publish its findings. Anthropic cannot veto publication, although it can require redactions in narrow, predefined categories.[1][2]

That is more substantial than a conventional external benchmark arrangement. It is still too early to say Anthropic is now independently audited. METR's own documents say these exercises are not audits, compliance assessments, or formal oversight. The company participates voluntarily, negotiates scope, controls access, and can redact sensitive material.[3][4]

The durable lesson goes beyond model safety. When an AI company says "independent evaluation," ignore the label for a moment and inspect four rights: what the evaluator can see, how it can speak to staff, what it can publish, and whether company redactions remain visible.

This article is based on the terms published by Anthropic and METR. I have not accessed Anthropic's internal systems, and METR has not yet published the first report produced under this arrangement.

What changed?

Anthropic's October 15, 2026 announcement says the embedded METR team has six members. They use company devices and office space. Except for explicitly excluded areas, Anthropic encourages staff to share information with them as they would with an employee working on alignment risk.[2]

The publication rules matter. METR can publish analysis of risk levels, incidents, and company practices. Anthropic does not have a general veto over that work. It may demand redactions for predefined categories such as customer data, privileged legal information, or model architecture. METR can publicly describe the redactions and say whether it considers them justified without revealing the protected material.[2]

This is not an open window into the company. Anthropic says the work is limited to a small team under enforceable agreements, with other METR employees siloed from the project.[2] The arrangement offers controlled access that is deeper than an outside model test, not full transparency.

Anthropic also describes three access tiers. At the first tier, the company chooses what to provide and approves publication. At the second, evaluators receive more information and can publish without company approval. The third tier combines employee-like access with publication rights similar to the second tier; that is the METR arrangement.[2]

Independence comes from rights, not logos

"Third party" and "independent" are not synonyms. An outside organization may be legally separate yet provide weak public assurance if it only tests a vendor-selected dataset and must obtain approval before publishing.

METR's October 14 policy offers a more useful standard. It commits to describing scope and duration, access limits, redactions, formal constraints, and informal pressure. It also says relevant conflicts of interest will be disclosed and that METR does not accept payment or donations from frontier AI companies for this work.[3]

The same distinction helps enterprise AI buyers. I would not treat "independently evaluated" as a positive signal on its own. These questions carry more information:

1. Did the evaluator see only the final model, or also training and monitoring systems?
2. Who set the scope, and how long did the work last?
3. Can findings be published without vendor approval?
4. Will readers learn that redactions happened and how they affected the conclusions?
5. Who funds the evaluator, and where are conflicts disclosed?
6. Are findings connected to an owner, a remediation plan, and a retest?

This resembles the issue in AI content transparency labels. A label does not create trust by itself. Trust depends on the process, evidence, and accountability behind it.

This is not an audit

Anthropic compares the arrangement to resident examiners in large banks.[2] The analogy explains why access matters, but the difference in authority should remain visible. A bank examiner acts with public authority. METR's access inside Anthropic comes from a voluntary contract.

METR states the limitation plainly. Companies can end an engagement, choose an assessor they expect to be less demanding, and decide which access requests to grant. METR also says these exercises are not audits, formal compliance assessments, or regulatory oversight.[4]

Deeper access does not guarantee a correct conclusion either. METR says it may rely on information it cannot independently verify and acknowledges that evidence supplied by a company could be filtered.[3] The science of loss-of-control risk is not mature enough to produce a simple safety certificate. METR frames its job as bringing better evidence into public view, not declaring a system safe.[4]

When the first report appears, the metadata around the conclusion will matter: which systems the team saw, how long it stayed, whom it could interview, which requests were denied, and what was redacted.

Who is affected, and what changes in practice?

The bar rises for frontier model companies. Publishing a system card or a selected benchmark table will look less persuasive on its own. Competitors may face pressure to say whether independent evaluators receive comparable access and publication rights.

Enterprise buyers should add a new section to vendor reviews. Alongside security certificates, retention terms, and SOC reports, ask how model behavior and internal monitoring systems were assessed. For critical workflows, the evaluator's independence matters, but so do report scope, redaction rules, and whether remediation is tracked.

Developers and end users do not get a new product feature today. This announcement does not prove that Claude is more accurate or safer. The first concrete value will arrive when METR publishes. Its report may confirm Anthropic's risk narrative, narrow it, or challenge it.

The boundary from my piece on human review and chat privacy applies here too. Short claims such as "humans have access" or "an independent body checked it" remain incomplete until we know which data, for what purpose, under which publication rights.

Conclusion

Anthropic's METR arrangement is one of the more serious voluntary models for independent AI evaluation so far. Its value is not that six people can enter an office. It is that Anthropic gives up a general publication veto and METR can disclose the existence and character of redactions.[2]

It is not a safety seal. Anthropic still controls access, the relationship remains voluntary, and the assessment focuses on a specific set of loss-of-control risks. The real test begins with the first difficult finding: Can METR describe it plainly, what will Anthropic redact, and will the public be able to see those limits?