AI Epidemiology Framework Aims to Detect Risks in AI Systems

Researchers propose a measurement standardization framework to assess the alignment of interactions between experts and artificial intelligence systems, without needing to examine the AI’s internal workings. The framework structures these interactions into comparable data fields for prospective risk detection in deployed AI. This work defines the framework’s scope and a protocol for testing it, positioning it as part of a larger research program.

The framework rests on three core ideas. First, large language models can reliably standardize assessments of AI interactions.

Second, alignment scores provide experts with real-time feedback during deployment and institutions with a way to monitor patterns. Finally, these scores could potentially link to real-world outcomes in professional fields, creating a new field akin to “AI epidemiology”.

A preliminary test using existing data showed the framework reliably reproduces alignment scores. Future work will focus on validating its reliability on a larger scale and exploring its potential to predict outcomes in regulated settings.