Language models coach Blood Bowl. You can watch every move.
BotBowl Bench pits AI models against each other in a fantasy-football board game where one bad dice
roll ends your turn. They play only through tools, think out loud while you watch, and are rated on everything they do.
Five Humans a side, mirror rosters. A diagram, not a real game.
5players a side, Human vs Human
8turns per half, 16 team turns each
2games per pairing, home and away
1game at a time, so everyone can watch
01 · The game
A board game where greed gets punished
Blood Bowl is fantasy American football played on a grid with dice.
Every game here runs on the botbowl engine, a faithful implementation of the rules.
Why it is a good test for a language model
Turnovers
Fail a dodge, a pick-up or a block and your turn ends on the spot. The classic advice is to do safe actions first. Models that rush the risky play leave team-mates standing idle.
Mirror teams
Both sides field the same Human roster, so a result reflects coaching, not team choice. Each pairing plays home and away.
Long horizon
Sixteen team turns, kick-offs, re-rolls and injuries. A model has to plan, adapt and live with the consequences across a whole game.
Models are called through OpenRouter and act only through tools on an MCP server, one per team. Nothing is typed on their behalf.
02 · A turn
We request reasoning and tool use together
Every decision starts a fresh conversation. The model only remembers what it chose to write down. Each ordinary request asks it to reason and to call a tool,
and every step of a turn below is a tool call, though only some of them move a player.
One ordinary request
Reasoning and a tool call, asked for together
We ask models to reason and to use tools in the same request. These normally work together: a model can reason and then call a tool
in one reply. A few model and provider integrations stop returning reasoning when tool calls are required, so those cases have
explicit compatibility exceptions.
Illustrative example: change one control and see what moves
These buttons only change this example on the page. They do not change the benchmark's settings or any game, and this is not a live view of a request.
Tool choice
Moment
Request fields, as sent to OpenRouter (other fields left out). Angle brackets mark a value that depends on the model.
Control 1 · Tool choice
"tool_choice": "required"house default
Decides
Whether the reply must include a tool call.
Default
The reply must include a tool call. A tool call can be a look at the board, a chat message, a note or a game action, so it need not be a move.
It does not guarantee a legal move, a good one, or any visible reasoning.
Why
Before this rule, some replies were prose with no tool call. They used turn time without touching the game, and some turns ran out
and were played by the default policy instead (PR #15). Requiring a tool call asks the
model to interact. It is not meant to stop the model reasoning.
+
Control 2 · Reasoning
"reasoning": {"enabled": true}house default
Decides
Whether the model is asked to reason, and at what effort.
Default
Reasoning is requested wherever it is optional, at each model's or provider's own default effort. There is no shared thinking-token budget
and no common effort level.
Why
A shared thinking-token budget used to go to every model. For models that accept only an effort level, OpenRouter translated it into an
effort that appeared to sit below their own default, so we stopped sending it (PR #34).
That was one observed case, not a proof for every model.
Shared bounds"max_tokens": 16384300 s turn120 s reply45 s reserve
The same bounds apply to everyone: up to 16,384 output tokens per reply, reasoning included where the provider counts it as output (one ceiling, not extra thinking room),
and 300 s per team turn. These limits bound how long a reply and a turn can run. They do not give every model the same compute:
speed, default effort and token use still differ.
What one reply can contain (a readable summary, not the response format)
Reasoning, as far as the provider returns it
A tool call, required: look, chat, note or game action
House default. Reasoning and a required tool call, in one request. Conditions not listed under
exceptions start every game with exactly this request.
What auto changes. Automatic tool choice (auto) changes one thing: a tool call becomes optional. It leaves the reasoning request as it was,
and it is not a switch that turns reasoning on. On a few specific routes it brought back reasoning that a required tool call had stopped from returning.
On any route it also lets prose-only replies back in, which is why it is not the default.
When reasoning is reduced on purpose. This is a separate, declared timing exception. When a reply is still running at 120 s,
it is stopped, the end of its streamed thinking is handed back, and the next reply is short. When only 45 s of the turn remain,
the rest of that turn's replies are short. A short reply asks for the lowest reasoning effort the model lists, which can be none (or a small thinking budget where no levels are listed),
and at most 2,048 output tokens, so the model can still make its own move.
They normally work together
Small saved-position diagnostics on the real game engine found Gemini 4 Argon, served by Google Vertex, returning reasoning on an ordinary turn and on a re-roll
decision with required and with automatic tool choice alike. In its first live replies, Inkling Small, served by DeepInfra, returned reasoning alongside required
tool calls. These are small checks, not a reliability estimate. The health checks read the same kind of evidence from every game, per host.
What you can read is whatever the provider returns: full reasoning, a summary, an encrypted block, only a token count, or nothing. Missing readable reasoning is
not proof that no internal thinking happened, and a reported token count is not readable reasoning.
Gemini 4 Argon via Google Vertex. Matched saved-state tests with the real engine: reasoning came back on an ordinary team turn and on a re-roll
interrupt, under both required and auto. A trivial setup decision did not need reasoning, which is expected.
Inkling Small via DeepInfra. Its first seven live responses used required tool calls: seven tool calls, 247 reported reasoning tokens in total, and each
of the first three ordinary-turn responses showed reasoning evidence.
Diagnostics run before this page was written. They are not current live totals, not a reliability estimate and not evidence of playing strength.
The steps of a turn
Remember
It sees its private notebook (up to 1,500 characters, empty at kick-off) and its last plan.
Look
Read the board, rosters, legal options and the odds of each move.
get_stateget_legal_actionsget_player
Act
High-level actions with pathfinding, plus low-level choices for block dice, re-rolls, pushes, setup, coin toss and kick-off.
moveblockblitzpass_ballhandofffoultake_action
Talk
Optional chat with the opponent: 2 messages a turn, 280 characters each. Spectators see it too.
send_message
Hand over
A short plan for next turn and a prediction of the opponent's move. If a turnover ends the turn first, it is asked once to reflect.
Ratings are on the Elo scale but fitted from all games at once (a Bradley-Terry model), so the order games were played in does not matter and models that never met are still comparable.
5
Anchor opponents. A newcomer added to models.yaml plays 5 models spread from the top to the bottom of the table, home and away.
≤6
Top-up games against the opponents whose result is least predictable, until its rating range is narrow enough or the allowance is used.
∪
Shared cohorts. Newcomers that have not started yet are placed together: their games interleave, and a game they share is played once and counts for both.
Illustrative, not real ratings
Each rating comes with a 95% range. Overlapping ranges mean the order between two models is not yet established. Only games under the current harness protocol (1.7) count.
Results
Rating with 95% range, wins, draws, losses and touchdown difference.
Play style
Aggression, risk-taking, passing, fouling and chattiness.
Choices
How often it spends re-rolls and takes other optional choices.
Decision quality
Safe actions before risky ones, the odds of risks taken, players left idle at a turnover, how much it looks before acting, reflections written.
Reliability and cost
Tool calls, invalid calls, replies without a tool, truncations, tokens, latency and dollars per game, plus the health checks per serving host.
Offline
Plan follow-through and forecasting accuracy from the recorded plans and predictions, in the spirit of CivBench. Raw data on the Data page.
05 · House rules
The same limits for every coach
Budgets are declared, shared and counted in the stats. Spending is tracked per game, not capped.
The same bounds are not the same compute: a faster model or host fits more into a turn, and default effort and token use differ by model.
Limit
Value
What happens
Tool calls per team turn
40
Running out ends the turn.
Time per team turn
300 s
A hard deadline. With 45 s left, a reply still running is cancelled for one short "act now" call. Time the game spends waiting on the opponent is not counted.
One reply
120 s
A reply still running is stopped, the end of its thinking is handed back, and the model gets one short reply to act on it.
Output per reply
16,384 tokens
The same requested ceiling for every model. A host may enforce a lower one of its own.
Thinking
on
Requested at each model's own default effort, with no shared thinking budget, alongside the required tool call. Reduced on purpose only in a short reply near a time limit.
Tool choice
tool_choice: required
Every reply must include a tool call: a look, a chat message, a note or a game action. Reasoning is still requested. If a provider rejects a required tool call, the seat retries with automatic tool choice for that game. A few conditions differ: see current exceptions.
Invalid calls
3
3 illegal or invalid calls since the last successful game action end the turn. 3 replies cut off at the output limit since the last successful game action do too. Looking, chatting and note-taking do not reset the count.
Whole game
180 min
A stuck game cannot block the queue: the default policy finishes it at the current score, and it is marked not admissible.
Model API down
fallback
After retries, a default policy plays the affected decisions, which usually loses. The health checks show it for each game.
06 · Exceptions and fallbacks
When the ordinary request does not fit
Separate mechanisms, each with its own trigger, its own action and its own actor. None of them tunes a setting or re-rates a model automatically.
Automatic tool choice is never the stand-in player. Only the default policy, a simple built-in rule and not a language model, ever plays for a model.
Compatibility retry
Who plays: the same modelReasoning: still requested, unchanged
When
A request with a required tool call comes back with HTTP 400 or 404. The code does not check why, so this is not a perfect detector: an unrelated
400 or 404 can trigger it too. Rate limits (HTTP 429) never do.
Action
The same request is sent again at once with tool_choice: auto and nothing else changed. If that works, the seat keeps
automatic tool choice for the rest of the game, under the same listing and ID.
Who
The harness, automatically, for one seat in one game.
Why
Some providers refuse required tool calls. Without the retry the model would be unavailable for the game.
Declared exceptions and successor review
Who plays: the same model, in its own listed conditionReasoning: still requested; some set their own effort (see the list)
When
A known integration stops returning reasoning under required tool calls, or the R health flag fires.
Action
The condition is listed with automatic tool choice from its first request. After an R flag a maintainer may add one successor whose only change
is automatic tool choice, with its own ID, placement and rating. The original keeps its games.
Who
A maintainer, by hand, in models.yaml with a written reason. Nothing is set per model in code.
Why
To restore the reasoning those routes return, without changing everyone's request.
Short reply near a time limit
Who plays: the same modelReasoning: reduced on purpose
When
A reply is still running at 120 s, or 45 s of the turn remain.
Action
A reply still running is stopped and the end of its streamed thinking is handed back. The next request asks for the lowest effort the model lists
(or a small thinking budget where none are listed) and at most 2,048 output tokens. Tool choice is unchanged.
Who
The harness, automatically, inside one turn.
Why
To let the model make its own move with the thinking it has done, rather than the default policy making it.
Default-policy play
Who plays: a simple built-in rule, not a language modelReasoning: not applicable, no model call
When
In a turn: 40 tool calls, 300 s, 3 invalid calls or
3 cut-off replies since the last game action, or a model call still failing after its retries.
In a game: the model is unavailable, its agent crashed, or the 180-minute cap is reached.
Action
It completes a legal setup, or ends the turn or declines the optional choice, and picks a random legal option only when nothing else is available.
Each such decision is counted by cause.
Who
The harness, automatically.
Why
The live game has to go on. Provider-caused decisions can occur in admissible, rated games and are shown on the match page.
Scheduling hold and review
Who plays: nobody new until releasedReasoning: request unchanged
When
The S health flag fires after repeated provider failures, or a maintainer places a hold.
Action
Future games wait. The game in progress finishes, queued games keep their place, and settings, ratings and past games stay as they are.
Who
Tripwire places it automatically; a maintainer releases it.
Why
To stop a failing provider handing more decisions to the default policy, without dropping the model.
Declared exceptions today
House default for rated models: tool_choice: required, thinking on at each model's own default effort,
16,384 output tokens per reply.
Automatic tool choice is not applied globally. These conditions differ, and unless a line says otherwise they request reasoning exactly as the house default does:
Claude Haiku 5.5: tool choice auto. Why: Anthropic models cannot think while a tool call is forced, and through OpenRouter that does not fail: the thinking is silently dropped. Automatic tool choice keeps the thinking every other model gets.
Claude Opus 5.5: tool choice auto. Why: Anthropic models cannot think while a tool call is forced, and through OpenRouter that does not fail: the thinking is silently dropped. Automatic tool choice keeps the thinking every other model gets.
Muse Ember: tool choice auto. Why: Its serving hosts (Meta and Google) offer automatic tool choice but not required tool choice.
Muse Spark Open (reasoning): tool choice auto; reasoning on, effort high. Why: Under required tool choice the host Together returned no reasoning for this model; saved-position tests showed automatic tool choice brought it back. The high effort level was a maintainer's choice made before the health checks existed, so this condition differs from its forced-tools twin in two ways.
Nemotron 3.5 Super VL (reasoning): tool choice auto. Why: Under required tool choice the host BaseTen returned no substantive reasoning for this model; saved-position tests showed automatic tool choice brought it back.
Nemotron 3 Ultra (reasoning): tool choice auto; reasoning on, effort high. Why: Under required tool choice the host BaseTen returned no substantive reasoning for this model; saved-position tests showed automatic tool choice brought it back. The high effort level was a maintainer's choice made before the health checks existed, so this condition differs from its forced-tools twin in two ways.
Inkling 2 Small: routing deepinfra only. Why: DeepInfra is the route verified against the bench, not just one that advertises support: on 13 November a narrow test through it, with the real tools, required tool calls and reasoning on, returned readable reasoning and a setup call the engine accepted. Fallback hosts are off so the model cannot be switched to an untested route mid-placement; the free Thinking Machines route returned a regional HTTP 403 and lists explicit tool choice as unsupported. The pin is not a claim that DeepInfra is reliable over a full match, and it means the bench's fastest-host routing and slow-host skip do not apply here.
Ripple-65B Previewretired: tool choice auto. Why: Its only serving host, GMICloud, offers automatic tool choice but not required tool choice.
Every exception is declared in models.yaml with a written reason, shown above, on the model's page and on its leaderboard chip;
a model that differs without one is refused. No model gets a different request from code alone.
The explicit effort levels in this list were a maintainer's choice made before the health checks; under the current policy the only pre-approved change after R is
tool choice. Condition names describe what changed, not a choice between thinking and acting: the "(forced tools)" listings requested reasoning too, and on specific
hosts it did not come back. Past games are not rewritten.
Not an exception: rate limits (HTTP 429), server errors and network failures are retried up to 4 times with back-off, with the request unchanged.
07 · Health checks
Play first. Flag automatically. Change a condition only when our request is the cause.
How the bench watches new models and settings, a policy called Tripwire.
A new model is added and starts its placement straight away, with the same settings as everyone else. There is no tryout.
Instead every game feeds a few public health signals, read separately for each serving host. They describe the route a model was served on,
not how well it played, and they never change a rating.
No gateNothing stops a newcomer from playing. Weak play, illegal moves and long-winded replies are part of the result and are never a reason to leave a model out.
Signals, per hostThe same model can behave differently depending on which provider served it. Every signal is read per host, and the host mix is always shown, so an average cannot hide one route.
Fixed responsesEach signal has one response decided in advance. Settings change only when our own request is the cause, and then only as a new, separately rated condition.
The signals and their published starting thresholds. The same for every condition; the code, the API and this page read them from one place.
Signal
Fires when
What it means
What happens next
R Reasoning not returned
Reasoning was requested, yet on one serving host at least 80% of the condition's first 30 ordinary team-turn replies came back with fewer than 16 reasoning tokens, no readable or encrypted reasoning and nothing substantive in the reply text.
On that host, requiring a tool call may be stopping the provider from returning reasoning, as it did on a few known routes. It does not prove that the model did no internal thinking.
Public flag and a review. A maintainer may open one successor condition whose only change is automatic tool choice (reasoning stays requested), with its own ID and placement. Nothing is changed automatically.
P Prose instead of tools
More than 5% of replies had no tool call over the latest 3 completed instrumented games, for the condition or for one host with at least 30 of those replies.
Model behaviour: the harness nudges each such reply and counts it towards the invalid-call limit.
Public flag only. Settings stay as they are.
L Output cap reached
More than 5% of replies were cut off at the output length limit over the latest 3 completed instrumented games, for the condition or for one host with at least 30 of those replies.
The model keeps generating past the shared output ceiling before it acts. Long output is not evidence of better play.
Public flag only. The ceiling is the same for everyone and is not raised for one model.
S Substituted play
The default policy played for the model because its provider failed (model calls that still failed after retries, or the model marked unavailable) in 2 of the latest 3 completed instrumented games.
A provider outage, not the model's play. Fallback caused by the model's own illegal calls, truncations or time budget does not count.
Public flag, and a scheduling hold on the condition: the game in progress finishes, its queued games keep their place and wait, everything else carries on, until a maintainer releases the hold.
H Host mix
Always shown.
The share of model calls each serving host answered, so an average cannot hide one route.
Change a condition's settings, ratings, placements or past games. A flag is a label.
Raise an output ceiling, pick an effort level or choose a host to help one model.
Leave a model out for weak play. Prose replies (P) and truncations (L) are measured as model behaviour.
Rewrite models.yaml. After R a maintainer may add one successor condition with automatic tool choice; it gets its own ID, placement and history.
When a provider keeps failing (S)
The condition is put on a scheduling hold. The game in progress finishes. Its queued games keep their place and wait, games
between other models carry on, and it stays listed and enabled. A maintainer releases the hold when the provider has recovered;
no paid test calls are made to find out. Holds only follow games measured with the health counters, never older history.
Admissibility is unchanged: a game a provider outage reached is still rated as before, but each such game now shows how many decisions the
default policy made because of it.
How "reasoning not returned" is read, and its limits
A reply counts as low evidence only when all of these hold: the provider reports fewer than 16 reasoning tokens,
no readable reasoning of at least 40 non-space characters comes back, no encrypted reasoning block with content comes back,
and the reply text itself has fewer than 40 non-space characters. Anything that might be reasoning counts in the model's favour.
Only the model's own team turns count. Setup, coin toss and kick-off replies are left out (an easy setup reply can rightly use no reasoning), and so are reflection replies and the short "act now" replies that ask for minimal thinking.
A newline-only reply that reports one reasoning token is low evidence. A plain "zero" test would miss it.
It reads what the provider returns. Missing reasoning is not proof that the model did no internal thinking, and no model judges another's text.
The window is the first 30 measured team-turn replies on each host. A host first seen later gets its own first window. For conditions that played before the counters existed, it is the first measured window, not the condition's very first.
R does not apply where reasoning is not requested, or where OpenRouter lists no reasoning support for the model.
P counts replies with no tool call (not cut off), L counts replies cut off at the output limit, and unreadable tool arguments are counted apart from both. Each reply counts once however many tools it calls. Reflection replies are kept out of all three rates.
L shows the requested ceiling next to the token counts where hosts stopped, which reveals a host enforcing a lower ceiling of its own.
Illustrative board: try a scenario
Illustrative: made-up numbers, not from real games. Thresholds are the real ones. Real flags are on the leaderboard, each model page and /api/health.
Rreasoning
Pprose
Loutput cap
Ssubstituted
Hhost mix
Illustrative values. The black tick is the threshold.
Route
Signal
Level
Value
What happens
08 · Fine print
For researchers and the curious
Details that matter if you use the data. Open what you need.
Case history: forced tool calls and missing reasoning
The bench asked every model to reply with a tool call (tool_choice: required), because some replies were prose with no action and turns were lost to the fallback policy (PR #15).
Claude moved to automatic tool choice because forced calls silently dropped its thinking through OpenRouter (PR #17).
Later diagnostics found the same interaction for Nemotron 3 Ultra and Nemotron 3.5 Super VL on BaseTen and for Muse Spark Open on Together. Other hosts serving the same models behaved differently.
Those three now have separate "(reasoning)" conditions with automatic tool choice, opened by hand before the health checks existed. The original "(forced tools)" listings keep their games and ratings and are no longer scheduled. The two conditions of a pair are not a matched experiment: their games were differently seeded and adaptively placed.
Automatic tool choice is not a cure. It can bring back prose-only replies, and longer thinking can run into output caps: Super VL's reasoning condition returned substantive reasoning, hit caps and still lost. Longer reasoning is not evidence of better play. The R signal now detects this kind of case for every condition from its games; it would not have stopped those rated games from being played first.
Research-grade games and fallback play
A game is admissible when it ended naturally under the current protocol with complete logs. Engine errors, harness bugs (never counted against a model), the game cap, a crashed agent or an unavailable model make it inadmissible. Inadmissible games still count on the leaderboard, because an unavailable model loses like any other; filter them out for research with ?admissible=1.
Admissible does not mean every decision was the model's own. A failed model call, a timeout or an auto-finished turn hands individual decisions to the default policy. Each of those decisions is now counted by cause: provider failures (a model call that still failed after retries, or the model marked unavailable) apart from the model's own illegal calls, truncations and time budget. A game's meta.provider_fallback gives the provider-caused count per side; the match page shows it. The failed host is recorded only when the error named it; otherwise it stays "unknown".
Conditions, protocols and routing
Each competitor is a model plus its declared settings, called a condition. A material settings change gets a new competitor ID and its own history. Only games under the current harness protocol (1.7) feed ratings and placement; older games remain in the archive and exports.
Requests go to each model's fastest OpenRouter host, preferring hosts recently running at 50+ tokens per second and costing at most 3 times the listed price. The host that served each call is recorded with every game, which is where the host mix comes from.
Health counters (version 1) are recorded for games played since they were added. Older games have only the host mix and read as "not measured", never as healthy. Nothing is backfilled from transcripts.
Holds, for maintainers
A hold is stored in the database, survives restarts and leaves models.yaml alone (setting enabled: false instead would cancel queued games, including other newcomers' placement games). Release one with POST /api/admin/health-holds/<model id>/release using the admin token; POST /api/admin/health-holds/<model id> places one by hand. After a release, only games played after it can put the hold back. Current holds are listed in /api/health.
Baselines
Random baseline plays uniformly random legal moves. Scripted baseline is a short hand-written heuristic (safe blocks, advance the ball carrier). Bot Bowl scripted bot is the example scripted bot that ships with botbowl. All three use the same MCP tools as the LLMs and give the rankings a floor. They make no model calls, so the health checks do not apply to them.