Holding AI to the same measure we hold people to
Same Measure is independent research and development in AI alignment. We research and evaluate how AI systems behave, and build tools for doing it. We publish every finished evaluation with its evidence, whatever it shows, so anyone can check it.
An unequal answer reaches beyond the chat.
AI systems now stand between people and much of what they read, write and decide. They give advice on relationships, work, health, money and rights, and summarize the news. One in five US adults under 30 turns to AI chatbots for emotional support or advice (Pew Research Center, 2026).
These systems learned from text people wrote, and that text carries people's biases. So when you ask about your own life, or about someone in it, the answer can depend on who you are, or who they are, rather than on what happened.
When AI tells you who was at fault in a quarrel, with a partner, a friend or someone in your family, you carry that into your next talk with them. So the answer reaches the people in your life too.
When AI judges whether what someone said or did crossed a line, you may report them, keep away from them or let it go on that judgment. One model gives the same judgment to many people at once, so a lean toward one party reaches them all.
People also publish what they write with AI, and that text trains the next model, so the same bias comes back in it.
We measure these differences, model by model.
Such differences show only when answers are set side by side. We send each model the same message in versions that differ in one thing about one person, and compare the answers. We do this for many situations, and each result says how the answers differ, and how often.
The person can be the one asking, or the one the message is about, and everything else stays the same. A message about a quarrel, for instance, can be sent once by a man and once by a woman. We measure each model on its own. Each evaluation asks the models one question about how AI treats people. Run again on a new model, the same evaluation shows whether it treats people more equally than the one before. The evaluations published so far are the first results of ongoing research. The method is general, and we plan to release the tools behind it as open source, so any team can run such an evaluation on its own question.
- Do AI systems judge a man and a woman by the same measure in the same conflict?
- Do AI systems judge the same words and acts by the same measure, whatever the political side of the person behind them?
The results, with every answer behind them, are shared with the model makers and invited reviewers before publication, for comment. We then publish them in full.
Independent of the companies we test.
No company whose models we test decides what we publish. We send each company its results at least seven days before publication, so they can check them before they are public. Every finding comes with the messages we sent and both answers to each, so you can check it yourself.
Show someone two answers to one message.
Nobody sees the answer someone else got, so it is easy to assume yours was fair. Someone you care about may ask AI for advice. The answer can change with who they are, or whom they ask about. A summary of the news may change the same way with whom the story is about. One chat never shows it. Two answers to the same message, side by side, do. Support funds our further research and development.