このノートにはまだ日本語版がありません。原文(英語)を表示しています。
If you have ever had a question that really mattered, you have probably done this. You typed it into ChatGPT. Then you opened Claude and pasted it again. Then Gemini. Three tabs, three confident answers, and ten minutes of reading them side by side to work out which one to believe.
That habit is correct.
It is also a lot of work, and it hands the hardest part of the job to the person with the least information to do it.
The habit is common enough that one of the field’s best-known engineers turned it into a weekend project. Andrej Karpathy’s LLM Council sends one question to several models, has each review the others’ answers with the names hidden, and asks a chair model to write the final response. A guide from UC Berkeley’s School of Information states the truth underneath it plainly: “The honest answer to ‘which LLM should I use’ is not which model is best, but which model is best for what.”
Why comparing tabs is harder than it looks
When three answers disagree, you have to decide which model to trust on this particular question. That depends on knowing what each one is good at, and the public signals for that are thinner than they look.
Every release arrives as the latest and the greatest, with benchmark tables and arena rankings to match. None of that is dishonest. It is simply not about your question. Stanford’s 2026 AI Index found the top four labs “clustered within 25 Elo points when rated against one another by human voting in the Arena Leaderboard.” As of March 2026, the highest-rated and the fourth-rated sat at 1,503 and 1,481. On the Elo scale those ratings use, a 22-point lead means winning roughly 53 votes in 100.
At the very top, the leaderboard is close to a coin toss.
The benchmarks underneath move just as fast. The same report notes that “evaluations intended to be challenging for years are saturated in months.” A chart that settled an argument in spring can be irrelevant by autumn.
So “which model is best” is the wrong question, and a ranking cannot answer the right one. Models are not evenly strong. The team behind RouterBench starts from exactly this: “no single model can optimally address all tasks and applications.” One is sharper on dense documents, another on careful reasoning, another on pulling many sources into one argument, and which is which shifts from release to release.
What the Researcher does instead
myOrbit’s Researcher does the three-tab job, with two things nobody sitting in front of three tabs has.
The first is a working picture of what each model is actually good at, built from our own runs rather than launch-day charts. A research question is not one task. It is several: breaking the question apart, searching, reading, weighing sources, writing the synthesis. Each part goes to the model best suited to it, across Anthropic, OpenAI and Google. Which labs we use, and why covers the roster.
The second is you. The same question from two different people is not the same question. What you are deciding, and what would change your mind, shape what a good answer looks like. That is alignment to the user, and it is what IntellectoAlign is for: the part of myOrbit that keeps an answer pointed at the person asking rather than at an average of everyone. I have written before about why intelligence is not alignment. This is where the two meet.
Put together, each part of your question is placed by capability and by fit to you, not by a public ranking. How that placement works is the part I keep private. That it happens is not.
A different lab checks the work
The step that checks a report’s claims runs on a different lab from the one that wrote it. Every run, in every mode.
There is a body of research behind that choice. Models judging text are not neutral about who wrote it. Panickssery, Bowman and Feng found that models can recognise their own writing, and reported “a linear correlation between self-recognition capability and the strength of self-preference bias.” The better a model knows its own work, the more it likes it.
Asking a model to fix its own reasoning does not rescue this. Huang and colleagues found that models “struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction.”
Diversity helps. Verga and colleagues replaced a single judge with a panel drawn from different model families, and found the panel “exhibits less intra-model bias due to its composition of disjoint model families.” Kim and colleagues traced where errors cluster, naming “shared architectures and providers” among the causes, and found that “on one leaderboard dataset, models agree 60% of the time when both models err.”
Here is the part I would rather say myself than have someone point out. A different lab does not make a check independent. The same paper found that “larger and more accurate models have highly correlated errors, even with distinct architectures and providers,” and Goel and colleagues warn that “model mistakes are becoming more similar with increasing capabilities.” Crossing the lab line lowers the odds of a shared blind spot. It does not remove them.
Which is exactly why I would never let the writer and the checker share a lab on top of everything else they already share.
The receipt, and what it leaves out
Every report ends with a receipt that lists the models used to produce it.
It is a list, not a breakdown. The receipt deliberately does not say which model wrote which part, and the reason is human rather than technical. Show someone that a paragraph came from one lab and a perfectly reasonable person starts arguing with the attribution instead of the paragraph: that section should have gone to the other one. The placement was made on evidence the reader cannot see. A per-part scoreboard would invite second-guessing of the one decision the system is best placed to make, and pull attention away from the only thing that matters, which is whether the report is right.
So you see every model that worked on your question. You do not get a scoreboard.
Research you keep
A finished Report goes into your Library, where you can come back to it, or listen to it instead of reading. Before a new run starts, the Researcher looks for research you already have on the same question and offers to use it, refresh it, or start fresh. Paying twice for an answer you already own is a small indignity, and it was an easy one to remove.
When the answers differ
Run the same question through the Researcher and through the research agents the big labs ship, and you will sometimes get different answers. That is expected. Each lab’s agent is excellent, and each is built on one lab’s models.
Research is rarely done for low stakes. It is the contract, the offer you are weighing, the market you are about to enter. At that moment the useful question is not which answer sounds best. It is which one had Anthropic, OpenAI and Google each working on the part they do best, placed for you rather than for a leaderboard, and checked by a lab that did not write it.
That is the answer the Researcher was built to give.
One lab’s answer is a good answer. It is still one lab’s answer.
References
Comparing models, and what rankings measure
- Andrej Karpathy. LLM Council. GitHub repository.
- UC Berkeley School of Information. Which LLM Should I Use? A Data Scientist’s Guide to ChatGPT, Gemini, and Claude. 13 August 2026.
- Stanford Institute for Human-Centered Artificial Intelligence. AI Index Report 2026: Technical Performance. 2026.
- Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, Shriyash Kaustubh Upadhyay. RouterBench: A Benchmark for Multi-LLM Routing System. arXiv, 2024.
Why the checker comes from a different lab
- Arjun Panickssery, Samuel R. Bowman, Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations. arXiv, 2024.
- Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou. Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024.
- Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, Patrick Lewis. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv, 2024.
- Elliot Kim, Avi Garg, Kenny Peng, Nikhil Garg. Correlated Errors in Large Language Models. ICML 2025.
- Shashwat Goel, Joschka Struber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, Jonas Geiping. Great Models Think Alike and this Undermines AI Oversight. arXiv, 2025.
Related systems
- Anthropic. How we built our multi-agent research system. 13 June 2025.
- Google. Gemini Deep Research.
- Sakana AI. Fugu: a multi-agent system, delivered as one model.
この考えを持ち帰る
正規のソースを共有するか、使っている AI のための読書プロンプトをコピーできます。
— orbiteer1