Summary

• Frontier AI companies and AI evaluation organizations should publish their evaluation contracts alongside any public evaluation report. 
• These contracts should be subject to minimal redactions for sensitive information and set out the key terms that shaped the evaluation, including what evaluators can access, what they can publish, how they are paid, and what conflict-of-interest rules they must follow.
• Publication of evaluation contracts is needed to provide transparency into the conditions evaluators work under and to make evaluation findings easier to interpret and compare. Where the terms for evaluators are poor, stakeholders can take this into account when assessing the credibility of evaluation results and put pressure on companies to improve these arrangements. 
• Potential downsides of publishing contracts may include creating a false sense of assurance and making companies more reluctant to grant extensive access.
• These downsides are outweighed by the benefits of publication. Without legal requirements on evaluation conditions, the publication of evaluation contracts is the most direct way for stakeholders to verify that adequate conditions were met. 
  ‍

Introduction

Third-party evaluations already play an important role in assessing risk under voluntary commitments from frontier AI companies and under frontier AI regulations. Recent proposals to embed independent evaluators with employee-like access within frontier AI companies could substantially strengthen external oversight of companies’ safety claims and internal practices.

However, the value of these assessments – as well as other types of external evaluations – depends partly on the contractual terms that govern them. These terms determine, among other things, what an evaluator can access, how they can publish or escalate findings, how they are paid, and how assessments can be terminated or renewed. Without knowledge of these terms, stakeholders cannot determine whether contractual constraints limited an evaluator’s access, compromised their independence, or otherwise placed pressure on them, making the results harder to interpret.

To address these concerns, AI evaluators and frontier AI companies should publish their evaluation contracts alongside public evaluation reports. 

The most direct way to achieve transparency about the terms under which evaluations are carried out is to publish the full evaluation contract, with redactions limited to proprietary technical information, security-sensitive information, and personal/third-party information.¹ Where full publication is not possible, companies and evaluators should at minimum disclose the terms most important for interpreting an evaluation. Table 1 sets out a non-exhaustive list of these terms and why they matter for assessing evaluation credibility. 

Contracts for frontier AI evaluations have been published before. For example, Transluce published the template agreement for its August 2026 mental-health evaluation with Anthropic, OpenAI, and Google DeepMind, but this was not an embedded assessment, and publishing evaluation contracts is not yet standard practice.²

There is also precedent for publishing contracts in other sectors. For example, under the EU’s Digital Services Act, audit agreements must be annexed to published audit reports (subject to specified exceptions for confidential information).³ 

The analysis that follows sets out the benefits and downsides of publishing evaluation contracts and outlines the terms that are most important to disclose. 
‍

What are the benefits of publishing evaluation contracts?

Publication of evaluation contracts offers four main benefits: 

1. Helping the public judge how much confidence to place in evaluation findings. Accurately interpreting evaluation results requires understanding the context in which they were produced, including the evaluators’ access, independence, control over publication, and any operational constraints. These conditions are likely to be defined in the contract. For example, contracts that show evaluators were protected from retaliation, could escalate concerns, and had significant editorial control would give readers greater confidence in the independence of their findings and their ability to report them freely. Contracts could also reveal safeguards against conflicts of interest (e.g. restrictions on evaluators selling other services to the company, or fees being fixed regardless of findings), reassuring stakeholders that evaluators did not have incentives to soften their conclusions. 

2. Making evaluation results easier to compare. Knowing what access evaluators had and what limitations were placed on their work can help stakeholders compare evaluations of similar safety claims across different companies. All else being equal, an evaluation conducted with broader access and more significant editorial independence may provide greater assurance than one conducted under tighter access or publication constraints. Publishing the contract could therefore help stakeholders compare companies’ risk profiles and judge how much assurance their evaluations provide.

3. Reducing incentives for both parties to accept or propose weaker terms. Publication of contracts would reveal cases where companies and evaluators agreed to inadequate access, restrictive publication rights, or weak independence protections. Because visibly weak terms could undermine stakeholder confidence both in companies’ safety claims and in evaluators’ findings, publishing contract terms gives both parties an incentive to agree to stronger terms.

4. Strengthening evaluators’ negotiating position. Published contracts would give evaluators evidence of what companies have agreed to in the past, which they could use as a baseline when negotiating evaluations of a similar scope. This seems important as there is evidence that evaluators already receive different levels of access and publication rights, but the full extent of these differences is not public.⁴ 
   ‍

What are the downsides of publishing contracts?

At the same time, publishing contracts presents three key downsides that will be important to address. 

1. Publishing evaluation contracts could create a false sense of assurance. Stakeholders might interpret strong contractual protections as evidence that an evaluation was rigorous and independent, even where this was not the case in practice. An evaluator might have broad access, editorial independence, and escalation rights, but fail to exercise these rights during the evaluation. For example, they might fail to fully elicit a model’s capabilities, overlook important lines of inquiry, or fail to escalate significant concerns. These shortcomings may not be apparent from either the public report or the published contract.
   
   This challenge will require a combination of approaches but may be difficult to address fully. One potential mitigation would be for companies and evaluators to commit to reporting which contractual rights were used or left unused, and why. Evaluators should also describe their methods, evidence, and limitations in their public evaluation reports. Both measures could help readers assess whether evaluators made effective use of their contractual rights. However, these mitigations rely on the good faith of companies and evaluators to report these details. Parties that do not make full use of their contractual terms may be unlikely to disclose this voluntarily.
   
   
2. Publishing evaluation contracts could make companies more reluctant to grant extensive access. Generous access terms, once published, could become an expected industry standard, even for assessments that do not require comparable access. They could also become a baseline for future regulation. Companies may therefore be less willing to grant extensive access when contracts must be published, fearing that voluntarily agreed terms could become lasting expectations or legal requirements. 
   
   To reduce this risk, each published contract or accompanying letter of engagement could explain the assessment’s purpose, the access provided, and why that level of access was appropriate. This could help stakeholders and policymakers distinguish justified differences between engagements from unjustified ones. And where an assessment’s narrower scope warrants narrower access, the explanation could reduce the risk that extensive access becomes an expectation or legal requirement for assessments that do not need it. Where the stated reasons do not justify the access provided, readers could see that the assessment terms were too weak. Companies that justify their access decisions in this way could also gain reputational benefits, which may in turn encourage other companies to provide access in line with the scope and purpose of a given assessment.
   
   The concern that companies might provide more restrictive access if contracts are published may also be less severe than it first appears, for two reasons. First, stakeholders currently cannot tell how much access evaluators receive. Publishing contracts with restrictive terms would therefore give stakeholders more information about the conditions under which evaluators worked, which would help them interpret evaluation results. Second, knowledge that evaluation contracts will be published could encourage some companies to grant more extensive access.

3. Companies may lack incentives to publish evaluation contracts. Publicly disclosing contracts could invite scrutiny of agreed evaluation terms and expose weaknesses that attract public criticism. Companies may see little benefit in voluntarily inviting this scrutiny, particularly if competitors keep their contracts private. As a result, they may be reluctant to disclose contracts even where doing so would help the public interpret evaluation findings and could provide greater external assurance about the credibility of the evaluation results. 
   
   While publishing contracts could expose companies to criticism, it could also bring reputational benefits. Companies that publish strong evaluation contracts could earn greater stakeholder confidence in their safety claims than competitors that do not. Companies that publish their contracts could also help establish stronger industry standards for evaluation transparency and create pressure on other companies to meet them. For example, Anthropic was the first frontier AI company to publish a safety framework, in September 2023. Within a year, OpenAI and Google DeepMind had published their own, and more than a dozen companies committed to do so at the AI Seoul Summit 2024.
   ‍

What aspects of the contract should be disclosed?

There are two broad approaches companies and evaluators could take to publishing their evaluation contracts. The first, and preferable, approach is to publish contracts in full, redacting only information that falls within a narrow set of predefined categories. The second approach is to disclose, at minimum, the terms most important for interpreting an evaluation.

Under the first approach, contracts would be published in full, with redactions kept to a minimum.⁵ For AI evaluation contracts, redactions should be limited to three main categories:

• Proprietary technical information: Details whose disclosure would reveal commercially sensitive aspects of a company’s systems (e.g. model architectures, training methods, or names and details of unreleased models).
• Security-sensitive information: Information whose disclosure could create security risks for the company, the evaluator, or third parties (e.g. names of partner organizations providing secure compute or testing facilities, or the security protocols governing evaluator access).
• Personal and third-party information: Sensitive information about individuals or other organizations that is not needed to interpret the contract (e.g. names and pay of individual staff, or the names of competitor companies the evaluator is prohibited from working with).

The categories of information in scope for redaction should be contractually agreed between the parties before the evaluation begins. This prevents companies from redacting unfavorable terms once findings are known and gives evaluators certainty about what they can publish.

Under the second approach, companies unwilling to publish full contracts with predefined redactions should at minimum disclose the terms set out in Table 1. These cover (1) evaluation scope, (2) access rights, (3) publication, redaction, and escalation rights, (4) compensation and funding, (5) independence requirements, and (6) contract duration and renewal.

Conclusion

To help the public interpret and compare evaluation findings, frontier AI companies and evaluators should publish their evaluation contracts alongside evaluation reports. Doing so would reveal cases where evaluators lacked adequate access, publication rights, or independence, and create incentives for both companies and evaluators to agree to stronger terms. Publishing contracts carries some potential downsides, including creating a false sense of assurance or making companies more cautious about granting access. However, these downsides can largely be mitigated if evaluators publish detailed accounts of their assessments and companies explain the reasoning behind the access terms they provide. While companies may lack strong incentives to publish contracts, there are some positive reputational incentives to do so.

Acknowledgements: I am grateful for valuable comments from Markus Anderljung, Ben Garfinkel, Elias Groll, John Halstead, Kevin Wei, and Sophie Williams (in alphabetical order).

‍