Compare Models runs the same prompt through two or three models at once and shows the answers side by side as they arrive.
It exists because every other way of choosing a model is guesswork.
Why it is a separate surface
Normally a request goes to whichever model is configured. That is the point of having a setting.
Here, that routing is bypassed deliberately and each model is called directly, in parallel. That is why the answers stream next to each other rather than one after another, and why the comparison is fair: same prompt, same moment, no cache in between.
What to compare on
The prompt you use decides whether this tells you anything.
Use questions people actually ask. A clever test question tells you which model is better at clever test questions.
Use your own data. Model differences on general knowledge are well documented and irrelevant to you. What matters is which one handles your vocabulary, your record structure and your edge cases.
Use a hard one and an ordinary one. The hard one shows the ceiling; the ordinary one shows what your team experiences forty times a day. They frequently point at different winners, and the ordinary one should usually win the argument.
What to actually look at
Not just which answer reads better.
Speed. Visible immediately because they stream together, and it is the difference most likely to decide whether people keep using the assistant. A two-second answer that is slightly worse beats a nine-second answer that is slightly better, every time, for an assistant used constantly.
Whether it stayed on your data or drifted into generic advice. This separates models more than raw capability does, and it is the failure that damages trust fastest.
Whether it admitted not knowing. A model that invents a plausible figure is dangerous in a business tool in a way that a model saying "I do not have that" is not. Test this on purpose by asking something the answer genuinely is not available for.
Length. Some models pad. Over hundreds of daily answers, padding is a real cost in reading time and in credits.
When to use it
Before changing the configured model. Ten minutes here prevents a fortnight of people quietly finding the assistant worse and not saying so.
When somebody complains about answer quality. Run their question through two models. Either the alternative is better, which is useful, or both are equally poor, which is much more useful, because it means the problem is knowledge rather than the model. See Teaching the assistant.
When a new model becomes available. New is not better for your work until it is tested on your work.
Cost of a comparison
Running two or three models on one prompt spends two or three times what a normal question does.
Trivial for a handful of tests, and worth being aware of if somebody discovers the page and spends an afternoon in it. It is a decision tool used occasionally, not a nicer chat window.
When not to use it
As a daily tool. Every comparison spends several times a normal question, because it runs several models. It is a decision aid, not a way of working.
To settle a taste argument. If two answers are both correct and somebody prefers one, that is persona, not model choice, and changing model to fix tone is the long way round.
A fair comparison
Three things that quietly ruin one.
Same prompt, exactly. Retyping it with a small improvement for the second model is the most common way people prove what they already believed.
No leading. A prompt written to suit how one model behaves will be won by that model, and the result tells you nothing about ordinary use.
Judge the output, not the reasoning. Some models narrate their thinking and some do not, and narration reads as thoroughness. What matters is whether the answer was right and usable.
It is also worth having somebody who did not set up the test read the answers without knowing which model produced which. That sounds excessive for a ten-minute decision, and it changes the conclusion often enough to be worth it on a decision that affects everybody daily.
Test the refusals too
Ask something the assistant should decline, and check both models decline it. A model that answers a blocked topic is a finding worth more than any quality difference.
Reading the result honestly
Two traps.
One prompt is not evidence. Models vary run to run. Three or four prompts before concluding anything, and if the results disagree, the models are closer than they look.
The longer answer looks better. It usually is not. Compare against what was asked, not against each other, or you will systematically pick whichever model writes most.
Comparing three rather than two
Three is supported and is usually one too many.
Two answers are compared. Three are skimmed, and the middle one is rarely read properly, so what you actually learn is which of the outer two you preferred.
Use three only when you genuinely have three candidates, which is rare. Most decisions are "the one we use" against "the one somebody suggested".
Keep a note of what you found
Model decisions get revisited every few months, usually by somebody who was not there the first time. Three lines saying what you compared, on what, and why you chose what you chose saves the whole exercise being repeated from scratch.
What to do with the answer
If one is clearly better, change the setting in Choosing your model, then watch latency and error rate in usage for a week.
If they are close, pick the faster one. Closeness means the quality difference will not be noticed and the speed difference will.
If both are poor, stop looking at models. The problem is what the assistant knows, and no model choice fixes that.
Did this answer your question?
No, ask a person