About
Team
Ben is an assistant professor in the Department of Psychological and Behavioural Science at the LSE.
Kobi is a research scientist at the UK AI Security Institute and a PhD student at the University of Oxford.
Luke is a research fellow at the nonprofit AI evaluator Transluce and a senior research fellow at Stanford University.
Frequently asked questions
Generative artificial intelligence is a powerful new technology that may transform society. How society chooses to respond to this technology depends in part on what the public and decision makers believe about its effects on human psychology and behavior. For example, it matters whether we believe its effect will be positive or negative on diverse outcomes like mental health, public opinion, medical diagnosis and treatment, workplace productivity, and student learning, among many others.
Research can help us understand these effects, but is currently limited in two fundamental ways. First, because AI can affect diverse outcomes, the relevant research is fragmented across different fields, and is often buried in academic articles sitting behind a paywall. Second, the technology is a rapidly moving target—AI systems are changing all the time—rendering static research studies or even field-wide summaries potentially out of date after just one or two years.
The AI Trials Observatory exists to overcome these limitations. The Observatory keeps a systematic record of what randomized trials show about how generative AI affects human outcomes like those above. The record is regularly updated and free, and we prioritize clear communication so societal decision-making about AI can run on the most current evidence.
A randomized trial is a type of study design for identifying cause and effect. In a trial, the researcher assigns study participants to one of at least two groups at random (e.g., by flipping a coin). In one group, participants are exposed to a “treatment” — such as an interaction with a generative AI model — whereas those in the other group are not. After exposure, the researcher measures what happens to everyone (the “outcome”, such as their mental health). Because the participants were randomly assigned to the groups, there should be no systematic difference between the groups’ outcomes unless the treatment affects the outcome. Therefore, if there is a systematic difference, the researcher can conclude this was caused by the treatment. Randomized trials are a powerful study design for identifying cause and effect, but they are not the only one.
Randomized trials are not the only way to study cause and effect. However, the Observatory focuses on randomized trials for two reasons, one scientific and one pragmatic.
To understand the effect of generative AI, we can’t simply compare the outcomes of those who use generative AI to those who do not use it, or who use it less. Users and non-users may differ in an outcome, such as their mental health, but this difference may be driven by something other than their AI usage. For example, younger and older people use AI at different rates and in different ways. They may also have different mental health for reasons unrelated to AI use. Naively comparing users and non-users could make it look like usage has an effect on mental health, when in fact it does not—it’s really just an effect of age. There are often many confounding variables like this, and it is challenging to account for them all.
Randomized trials get around this problem by randomly assigning people to different groups and systematically varying their exposure. This requires us to assume that the random assignment worked as intended. Other types of study design rely on different techniques for identifying cause and effect. Those techniques typically require us to make stronger and harder-to-verify assumptions in order to believe we have identified cause and effect. For this reason, the Observatory focuses on randomized trials only. This doesn’t mean that other types of evidence are not valuable—on the contrary, triangulating evidence from different approaches is generally the best way to uncover cause and effect.
So why doesn’t the Observatory also include evidence from these other approaches? The second reason is pragmatic. Widening the evidence scope would dramatically increase workload, reducing the quality and speed of our updates to the evidence record. The Observatory seeks to maximize quality and speed for the evidence from randomized trials; we trust others and the division of labor to do the same for evidence from alternative approaches.
Despite their strengths in identifying cause and effect, randomized trials have several limitations worth emphasizing. These limitations do not make trial evidence useless—all study designs have limitations—but keeping them in mind is important to avoid overconfidence in what trials tell us about the effect of generative AI on human psychology and behavior.
Snapshot in time. Trials take place over a defined time period and therefore can be rendered obsolete if their object of study changes over time. With generative AI, this is very much the case: AI systems are a rapidly moving target. The result of a trial on ChatGPT in 2023 may not hold for the ChatGPT of 2026. The AI Trials Observatory exists in part to address this problem, by keeping a regularly updated record of what the latest trial evidence says.
Long-term cumulative effects. Most trials take place over the short or medium term, limiting their ability to measure the long-term effects of cumulative exposure to generative AI. This limitation is most important for outcomes where we have reason to believe long-term effects of cumulative exposure might be qualitatively different than short- or medium-term effects. Ambitious and highly resource-intensive trials conducted over the long term can partially overcome this limitation.
Research conditions. In many (but not all) trials, the participants know they are taking part in research. This could cause them to act differently than they would outside of a research setting. Furthermore, the people who choose to participate in research trials are often likely to be systematically different from the population of people among whom we would like to understand the effects of generative AI. This could limit the generalizability of the results of the trial.
Equilibrium effects. Trials are necessarily limited in scope—for example, it is generally not possible to randomly assign entire societies to different groups. As a result, a trial cannot show what happens when an entire society is exposed to generative AI. This limitation is most important for outcomes where we have reason to believe the effect of exposure might be qualitatively different when everyone is exposed compared to when only some people are exposed.
We currently don’t use strict formal criteria for choosing outcomes. Instead, we try to prioritize tracking outcomes which have generated (or may generate) considerable public interest regarding the effect of generative AI and that have a pre-existing evidence base of randomized trials which will plausibly grow in the future. We already have several outcomes lined up for future releases, but are always open to suggestions—you can find our contact information in the Contact section below.
Currently every 2-3 months. To get an email each time it updates, sign up on the Latest Updates page.
The Observatory does not currently receive funding from any outside individual or group.
Each time we update the evidence record for a particular outcome, we publish a self-contained and frozen release report describing the methods and results for that update. Each report ships with a unique digital object identifier (DOI), which permanently resolves to it, meaning you can cite the numbers contained within the report and they won’t disappear or change over time. Visit the Methodology page to find the release reports for each outcome we track.
Contact
Corrections, missing trials, data questions and press inquiries: [email protected].