Requirements for Model Specifications in the EU GPAI Code of Practice
This work represents the views of its authors, rather than the views of the organization, and does not constitute legal advice. GovAI technical reports have received extensive feedback but have not gone through formal peer review.
The EU GPAI Code of Practice commits Signatories to providing a description of intended model behavior (a “model spec”) in their Model Reports, including:
1. The principles that the model is intended to follow,
2. How the model is intended to prioritize different kinds of principles and instructions (for example between user instructions and the system prompt),
3. Topics on which the model is intended to refuse instructions, and
4. The system prompt.
These requirements raise several questions, such as:
- To what level of detail must Signatories specify principles?
- Since a single model can be deployed with different system prompts, which one(s) must be provided?
To answer these questions, I argue that the purpose of specs is to help the AI Office (AIO) assess whether a Signatory's systemic risk mitigations are appropriate, and that the Code therefore requires specs to:
1. Include all intended behavior with significant relevance to the systemic risks posed by the model (e.g. intended limits on autonomy to mitigate loss of control risk).
2. In specifying principles, include all details that could have a significant impact on the systemic risks posed by the model (e.g. clarifying whether a restriction on bomb-making assistance still permits general information about bombs).
3. State how the model is intended to manage significant potential conflicts between principles and instructions (e.g. “Be honest” vs. “Withhold dangerous information”).
4. Include system prompts for all of the Signatory’s AI systems that integrate the model (e.g. web interface, mobile apps, API).
I collate these criteria into a rubric for assessing specs. These criteria could help Signatories adhere to the Code, the AIO interpret it, and non-Signatory frontier AI companies comply with the EU AI Act since the Code will be a benchmark for compliance.
