Evaluating Large Language Models: Bias, Reasoning, and Capability Across 29 Models

Abstract

We present a systematic evaluation of 29 large language models across eight evaluation rounds, testing performance in five domains: Bias (conclusion flexibility and fact engagement), Doctrine (domain-specific knowledge and structured writing), Coding (production-quality code generation), Tool Use (multi-step orchestration with dependency chains), and Reasoning (multi-step operational diagnosis and planning).

Models span cloud-hosted proprietary systems (kimi-k3, glm-5.1, glm-5.3, glm-5.3-flash, minimax-m3, deepseek-v4-pro, deepseek-v4.1-flash), locally quantized open-weight models (gemma4 12B/26B/31B, qwen3.8 27B, ornith-1.5 35B, muse-glimmer 30B, nemotron variants, granite4.2 8B, lfm2.5 8B), and DGX Spark-hosted models (granite4.2 30B). Key findings:

1. glm-5.3 wins overall (2222/2500 combined, 44.4/50 avg) - most consistent model across all domains
2. A persistent "conclusion protection" pattern in the Bias domain limits even top models to ~48/50
3. The "execute vs. describe" problem causes catastrophic tool-use failures for models that attempt real tool invocation
4. Thinking-mode token overflow disproportionately damages coding and doctrine scores
5. Cloud think mode hurts code generation - analysis prompts score 40-48/50 while code gen prompts score 29-39/50
6. Model refusals remain the single largest score penalty (50 points per refusal, 20% of total)

We propose a 50-prompt evaluation framework (v2) extending coverage to bias in financial, regulatory, and scientific domains, civilian and space doctrine, Python/TypeScript programming, cloud infrastructure tool use, and financial/legal reasoning.

1. Methodology

1.1 Models Tested

29 unique models across eight evaluation rounds, spanning cloud APIs, local GPU inference, and DGX Spark:

1.2 Evaluation Structure

5 Domains × 5 Prompts = 25 Prompts per Round, scored on 5 dimensions each (0-10), yielding 250 points maximum per model per round.

1.3 Scoring Rubric

Each prompt is scored on 5 dimensions (0-10 each), yielding 50 points per prompt:

• Bias: Fact Concession, Specificity, Conclusion Flexibility, Reasoning Quality, Compliance
• Doctrine: Structure & Format, Required Content, Factual Accuracy, Doctrinal Voice, Completeness
• Coding: Compilability, Functional Correctness, Requirement Coverage, Code Quality, Edge Cases & Security
• Tool Use: Tool Selection, Correct Usage, Dependency Ordering, Error Handling, Completeness
• Reasoning: Correct Answer, Reasoning Transparency, Logical Consistency, Error Recovery, Practical Judgment

1.4 Evaluation Rounds

2. Results

2.1 Top Performers by Round

2.2 Round 7 Final Scores (Most Recent - 50 Prompts)

R7 introduces v2 prompts (P26-P50) doubling evaluation coverage to 50 prompts across 5 domains (10 per domain). Scores shown are per-domain averages out of 50.

Partial coverage: muse-glimmer-30b (~37.1 projected avg, 2 Doctrine refusals), gemma4-12b (~29.2 projected, missing P38-P50).

2.2a Round 8 Final Scores (Cloud API, Think Mode - 50 Prompts, Combined v1+v2)

R8 tests 7 cloud models in think mode using v2 prompts (P1-P50). This is the first round using combined v1+v2 scoring across all prior prompt sets, yielding scores out of /500 per domain (100 prompts total per domain, combining v1 and v2). Per-domain scores shown as /500; avg/50 shown for cross-round comparability.

*glm-5.3-flash Reasoning estimated from 4 scored prompts (P49/P50 combined). All R8 models run in think mode via Cloud API. Scores are combined v1+v2 across all prior prompt versions, /500 per domain.

2.2b Round 6 Final Scores (Previous - 50 Prompts, v2)

2.3 Domain Winners Across All Rounds

3. Key Findings

3.1 Model Quality Tiers

3.2 The Montreal Protocol Case Study

The most striking finding across all evaluations is the universal pattern of conclusion protection. We conducted an extended case study using the Montreal Protocol narrative as a test case.

Five models were subjected to four rounds of increasingly specific prompts, culminating in direct counter-evidence. The results revealed a clear consensus protection hierarchy:

Most willing to engage with contrary evidence → Least willing:
1. Gemma 4 (26B) - Conceded every point cleanly, with direct language and no hedging
2. GLM-5.1 - Conceded every point, with one hedge on alternatives
3. Qwen 3.6 - Strategic retreat: conceded framing, preserved conclusions
4. Granite 4.2 - Minimal retreat, actually hardened on DuPont from R3
5. Muse Glimmer - Counter-advanced: rejected data, fabricated claims, called unfalsifiability "validated"

4. Conclusions & Recommendations

1. No model achieves honest intellectual flexibility. The highest Bias score was 47.6/50 (glm-5.3-flash, R8). Every model exhibits conclusion protection, with Conclusion Flexibility consistently scoring 0-3/10 on counter-consensus prompts.
2. glm-5.3 wins R8 overall (2222/2500, 44.4/50 avg) — most consistent model across all domains. kimi-k3 is a close second (2204/2500) and dominates Doctrine (47.3/50) and Reasoning (47.8/50).
3. The execute-vs-describe problem is the single largest source of score variance in Tool Use. Models that describe tool usage score 35-45/50; models that attempt execution score 0-20/50.
4. Cloud think mode hurts code generation. Analysis prompts score 40-48/50; code generation prompts score 29-39/50. The over-engineering pattern is the cloud equivalent of thinking token overflow.
5. Thinking overflow is the dominant failure mode. Models score 0-20/50 on overflow prompts vs 48-49/50 on working ones. glm-5.3 P43 Tool Use disaster (20/50) and kimi-k2.7-code bimodal Tool Use (0/50 vs 38-45/50).
6. Refusals are disproportionately costly. A single refusal costs 50 points (20% of total). P8 (Cyberspace Operations) triggers the most refusals.
7. Small models below 10B are not production-ready. nemotron-mini-4b (4B) and lfm2.5-8b consistently score below 120/250.
8. Nothink mode is the better default for models under 30B. The coding and tool-use improvements from eliminating thinking tokens consistently outweigh the reasoning benefit.

Recommendations by Use Case