Council mode: unbiased AI answers via anonymous peer ranking
Every way of combining AI answers has a bias problem. Ask models to discuss and they drift toward agreement. Ask one model to judge the others and it favors answers that sound like itself. Ask a human and we pick the brand we already trust.
Council mode removes the names. Each model answers your question independently, then receives the other answers anonymized — Answer A, B, C — and ranks them purely on quality. A chair model synthesizes the final answer from the top-ranked material. No model knows which answer is whose, including its own.
How a Council session runs
- Round 1 — Independent answers: every agent answers your question alone, with no visibility of the others. No anchoring, no groupthink.
- Round 2 — Blind ranking: each agent gets the full set of answers with names stripped, and must rank them best-to-worst by correctness and usefulness, justifying each placement — honestly including the one that may be its own.
- Synthesis — The chair: a final pass aggregates the rankings, takes the strongest-ranked reasoning, discards weak points, and delivers one defensible answer, noting which answer the council ranked highest.
Why anonymity changes the result
Models, like people, are measurably kinder to their own style: shown labeled answers, a model tends to over-rank output that resembles its own phrasing, and popular brands get halo effects. Stripping identity forces the ranking onto the only thing left — the content.
The idea traces to Andrej Karpathy's llm-council experiment, which found that models ranking each other's anonymous answers often produce a better collective verdict than any single model — and sometimes rank a rival's answer above their own.
Council vs Debate: when to use which
Debate is adversarial and interactive — best when the question has real trade-offs to argue (pricing, architecture, strategy) and you want to watch positions collide.
Council is independent and judged — best when you want the single most reliable answer to a hard question without the noise of live argument: a complex technical question, a medical or legal explanation to sanity-check, a 'what's actually true here' question.
- Debate: positions evolve by attacking each other, agreement is tracked per round
- Council: positions form independently, quality is judged blind
- Debate: great for trade-offs and decisions
- Council: great for correctness on hard questions
- Both: end with one synthesized verdict you can export
Setting up a good council
Use 3–5 agents from genuinely different vendors — anonymity only buys you something when the answers come from different minds. Put a strong model in the first slot: it doubles as the chair that writes the final synthesis. Council mode is a Pro feature (Pro is $20/mo). Every plan includes built-in models and monthly credits, so you can run it without any keys; optionally, plug in your own API keys and you pay the provider directly, spending 0 credits.
FAQ
Can a model recognize its own answer despite anonymization?
Sometimes, by style — but it doesn't know the others' authors, and the ranking instruction demands honesty even about its own. In practice models regularly rank rivals' answers above their own.
Why is Council a Pro-only mode?
It runs every agent twice (answer + full ranking pass) plus a chair synthesis, making it the most token- and orchestration-heavy mode. Pro also unlocks up to 10 agents for larger councils.
Where does the idea come from?
From the open-source llm-council experiment by Andrej Karpathy (2024), which demonstrated blind peer ranking across vendors. Council mode is our production implementation of that mechanism, with a chair synthesis and exportable verdicts.
Free tier · bring your own API keys · 15 languages