Assessing Risk Relative to Competitors: An Analysis of Current AI Company Policies
Executive summary
● When frontier AI companies assess the risks from their models, they increasingly focus on marginal risk. This aims to measure how much their models increase risk compared to some baseline (e.g. a situation where an actor doesn’t have access to any AI model). In general, this is a sensible approach.
● Recently, some companies have started to assess marginal risk relative to their competitors’ models. These companies have provisions in their safety frameworks that allow them to lower their mitigations if a competitor has released a model with similar capabilities but weaker mitigations. The basic idea is that this wouldn’t meaningfully increase the total level of risk in the ecosystem.
● We discuss ways in which this approach could, in fact, increase the total level of risk. For example, in practice, it’s very difficult to accurately evaluate the capabilities of a competitor’s model or to be confident that its mitigations are indeed weaker. As a result, companies may wrongly conclude that they can lower their own mitigations without increasing risk.
● Furthermore, the total level of risk would only stay the same under certain conditions. For example, accident risks – such as models giving adverse health advice or assisting suicide attempts – are likely to scale with the amount of users. As such, if a company were to lower their mitigations in response to others doing so it’s likely to increase risk. We also identify reasons why other types of risk could increase.
● Even if these conditions hold, assessing marginal risk relative to one’s competitors could still erode safety standards. Such erosion might happen suddenly, if a single defector triggers many other companies to lower their mitigations at the same time, or through a “boiling frog” effect, where small, incremental increases in risk go unnoticed, but accumulate over time.
● Given these concerns, this particular approach to marginal risk needs further scrutiny and should not be accepted as a best practice without significant analysis and public discussion.
