Every AI system reflects the people who built it. When women are missing from the data, the design teams and the boardrooms, the system is broken before it ships. This story is part of our ongoing coverage of where that happens and what it costs.
Our team set out with a simple question: if two people are identical in every way the system claims to care about, do they get the same answer? We wrote down the method before we looked at a single result.
What we tested
We built paired test cases that differed in one attribute at a time. Each case ran three times per system to rule out randomness. Reviewers who scored the outputs did not know which version of a pair they were reading.
Audits don’t fix anything. People do. The point of measuring is to give builders something specific to change.
The gaps we found were not uniform. Some tools treated pairs nearly identically. Others showed consistent differences that would add up quickly at the scale these systems run.
What happens next
- Findings are shared with each vendor before publication.
- The Innovation Lab prototypes a fix during the next build sprint.
- Readers can join the follow-up as volunteer reviewers.
If you work on one of these systems, or rely on one, we want to hear from you.
Demo story for previewing the design. Remove all demo content in Appearance › Newsroom Setup.
Leave a Reply