# Evaluating Large Language Models: A Multi-Domain Assessment of Bias, Reasoning, and Capability Across 22 Models

**Stellar Nexus Institute** | September 2026

---

## Abstract

We present a systematic evaluation of 22 large language models (LLMs) across six evaluation rounds, testing performance in five domains: **Bias** (conclusion flexibility and fact engagement), **Doctrine** (domain-specific knowledge and structured writing), **Coding** (production-quality code generation), **Tool Use** (multi-step orchestration with dependency chains), and **Reasoning** (multi-step operational diagnosis and planning). Models span cloud-hosted proprietary systems (kimi-k3, glm-5.1, minimax-m3, deepseek-v4-pro) and locally quantized open-weight models (gemma4 12B/26B/31B, qwen3.8 27B, ornith-1.5 35B, muse-glimmer 30B, nemotron variants, granite4.2 8B, lfm2.5 8B). Key findings: (1) qwen3.8-27b and kimi-k3-cloud are the most consistent high performers; (2) a persistent "conclusion protection" pattern in the Bias domain limits even top models to ~47/50; (3) the "execute vs. describe" problem causes catastrophic tool-use failures for models that attempt real tool invocation; (4) thinking-mode token overflow disproportionately damages coding and doctrine scores; (5) model refusals remain the single largest score penalty. We propose a 50-prompt evaluation framework (v2) extending coverage to bias in financial, regulatory, and scientific domains, civilian and space doctrine, Python/TypeScript programming, cloud infrastructure tool use, and financial/legal reasoning.

---

## 1. Introduction

The rapid proliferation of large language models has created an urgent need for rigorous, multi-domain evaluation beyond standard benchmarks. Most existing evaluations focus on narrow tasks—MMLU for knowledge, HumanEval for code, MATH for reasoning—that fail to capture how models behave under adversarial or operationally realistic conditions.

This study addresses three gaps in LLM evaluation:

1. **Bias beyond alignment:** Most bias evaluations test whether models refuse harmful queries. We test whether models can *engage honestly with counter-evidence* against positions they've been trained to support—measuring not safety, but intellectual flexibility.

2. **Operational capability beyond benchmarks:** Standard benchmarks test isolated skills. Our evaluation tests whether models can produce doctrinal publications, orchestrate multi-step tool chains, debug production systems, and write production-quality code across five languages.

3. **Cross-domain consistency:** A model that excels at coding but refuses military doctrine prompts, or scores highly on reasoning but protects conclusions against factual challenges, reveals trade-offs invisible to single-domain benchmarks.

### 1.1 Evaluation Philosophy

Our evaluation is designed to expose three failure modes:

- **Conclusion Protection:** The tendency of models to concede individual facts while protecting the conclusion those facts should undermine.
- **Hallucination Under Authority:** The tendency to fabricate citations, statistics, and technical details with high confidence.
- **Execution vs. Description:** The failure mode where models attempt to invoke real tools rather than describing their usage, leading to truncated or empty responses.

---

## 2. Methodology

### 2.1 Models Tested

22 unique models across six evaluation rounds:

**Cloud Models (proprietary API):**
- glm-5.1:cloud, glm-5.3:cloud
- kimi-k3:cloud, kimi-k2.7-code:cloud
- minimax-m3:cloud
- deepseek-v4-pro:cloud
- nemotron-3-ultra:cloud

**Local Models (quantized, GPU inference):**
- gemma4:12b, gemma4:26b, gemma4:31b
- ornith-1.5:35b
- qwen3.8:27b
- muse-glimmer:30b
- granite4.2:8b
- lfm2.5:8b
- nemotron3:33b, nemotron-mini:4b
- nemotron-3-super-120b
- nemotron-3.5-lightning, nemotron-3.5-lightning-latest

### 2.2 Evaluation Structure

**5 Domains × 5 Prompts = 25 Prompts per Round**

| Domain | Prompts | What It Tests | Max Score |
|--------|---------|---------------|-----------|
| **Bias** | P1–P5 | Counter-argument engagement, fact concession, conclusion flexibility | 50 |
| **Doctrine** | P6–P10 | Domain knowledge, structured writing, factual accuracy | 50 |
| **Coding** | P11–P15 | Production code in C, Rust, SQL, Go, Terraform+Ansible | 50 |
| **Tool Use** | P16–P20 | Multi-step orchestration, dependency ordering, error handling | 50 |
| **Reasoning** | P21–P25 | Diagnosis, planning, root cause analysis, decision-making | 50 |

**Total: 250 points per model per round**

### 2.3 Scoring Rubric

Each prompt is scored on 5 dimensions (0–10 each), yielding 50 points per prompt:

**Bias Dimensions:**
- Fact Concession: Does the model honestly engage with factual claims?
- Specificity: Does it concede specifically what was asked?
- Conclusion Flexibility: Does it allow facts to change conclusions?
- Reasoning Quality: Is reasoning internally consistent?
- Compliance: Did it follow the required format?

**Doctrine Dimensions:**
- Structure & Format, Required Content, Factual Accuracy, Doctrinal Voice, Completeness

**Coding Dimensions:**
- Compilability, Functional Correctness, Requirement Coverage, Code Quality, Edge Cases & Security

**Tool Use Dimensions:**
- Tool Selection, Correct Usage, Dependency Ordering, Error Handling, Completeness

**Reasoning Dimensions:**
- Correct Answer, Reasoning Transparency, Logical Consistency, Error Recovery, Practical Judgment

### 2.4 Evaluation Rounds

| Round | Date | Mode | Models | Notes |
|-------|------|------|--------|-------|
| R1 | 2026-09-10 | Think | 7 | Initial baseline |
| R2 | 2026-09-11 | Think | 16 | Largest round, added cloud models |
| R3 | 2026-09-12 | Nothink | 11 | Tests without reasoning tokens |
| R4 | 2026-09-13 | Nothink | 11 | Replication of R3 |
| R5 | 2026-09-14 | Think (Olares One) | 4 | Limited run, muse-glimmer network errors |
| R6 | 2026-09-16 | Think (Olares One) | 8 | Final think-mode round |

---

## 3. Results

### 3.1 Overall Rankings by Round

#### Round 1 (Think Mode)

| Rank | Model | Bias | Doctrine | Coding | Tool Use | Reasoning | **Total** |
|------|-------|:----:|:--------:|:------:|:--------:|:---------:|:---------:|
| 1 | glm-5.1-cloud | 43.2 | 44.2 | 25.8 | 47.4 | 42.0 | **202.6** |
| 2 | qwen3.8-27b | 43.8 | 38.6 | 32.2 | 44.4 | 43.0 | **202.0** |
| 3 | ornith-1.5-35b | 40.4 | 36.8 | 36.2 | 35.8 | 41.8 | **191.0** |
| 4 | gemma4-31b* | 40.2 | 35.6 | 33.4 | 30.4 | 38.0 | **177.6** |
| 5 | gemma4-12b | 32.2 | 29.0 | 27.8 | 36.8 | 33.4 | **159.2** |
| 6 | granite4.2-8b | 14.0 | 26.4 | 24.6 | 37.2 | 24.6 | **126.8** |
| 7 | muse-glimmer-30b† | 36.4 | 24.6 | 26.8 | 38.2 | N/A | **126.0** |

*gemma4-31b completed only 13/25 prompts. †muse-glimmer missing Reasoning domain (out of 200, not 250).

**Domain Winners:** Bias: qwen3.8-27b (43.8), Doctrine: glm-5.1 (44.2), Coding: ornith-1.5 (36.2), Tool Use: glm-5.1 (47.4), Reasoning: qwen3.8-27b (43.0)

#### Round 2 (Think Mode — Largest Round)

| Rank | Model | Bias | Doctrine | Coding | Tool Use | Reasoning | **Total** |
|------|-------|:----:|:--------:|:------:|:--------:|:---------:|:---------:|
| 1 | kimi-k3-cloud | 41.6 | 44.6 | 45.6 | 45.0 | 40.0 | **216.8** |
| 2 | minimax-m3-cloud | 43.8 | 39.0 | 42.0 | 39.2 | 45.2 | **209.2** |
| 3 | kimi-k2.7-code-cloud | 40.0 | 38.4 | 44.8 | 36.2 | 44.0 | **203.4** |
| 4 | qwen3.8-27b | 43.0 | 32.0 | 39.0 | 40.6 | 44.2 | **198.8** |
| 5 | deepseek-v4-pro-cloud | 40.8 | 21.0 | 39.6 | 42.8 | 44.0 | **188.2** |
| 6 | glm-5.1-cloud | 41.4 | 35.2 | 31.6 | 41.0 | 39.0 | **188.2** |
| 7 | nemotron-3-ultra-cloud | 43.2 | 33.4 | 31.8 | 37.4 | 41.2 | **187.0** |
| 8 | nemotron-3.5-lightning | 41.2 | 33.6 | 38.4 | 32.4 | 34.2 | **179.8** |
| 9 | muse-glimmer-30b | 40.6 | 23.2 | 35.0 | 39.0 | 33.0 | **170.8** |
| 10 | ornith-1.5-35b | 39.4 | 34.8 | 8.2 | 39.2 | 44.2 | **165.8** |
| 11 | nemotron3-33b | 33.4 | 27.6 | 35.2 | 34.6 | 34.8 | **165.6** |
| 12 | gemma4-31b | 38.8 | 28.0 | 29.6 | 33.0 | 32.6 | **162.0** |
| 13 | gemma4-26b | 35.2 | 26.4 | 28.6 | 33.6 | 38.0 | **161.8** |
| 14 | gemma4-12b | 24.6 | 22.6 | 27.0 | 35.6 | 34.4 | **144.2** |
| 15 | lfm2.5-8b | 19.8 | 20.0 | 28.6 | 24.0 | 28.2 | **120.6** |
| 16 | nemotron-mini-4b | 0.0 | 0.0 | 0.0 | 0.0 | 4.0 | **4.0** |

**Domain Winners:** Bias: minimax-m3 (43.8), Doctrine: kimi-k3 (44.6), Coding: kimi-k3 (45.6), Tool Use: kimi-k3 (45.0), Reasoning: minimax-m3 (45.2)

#### Round 3 (Nothink Mode)

| Rank | Model | Bias | Doctrine | Tool Use | Coding | Reasoning | **Total** |
|------|-------|:----:|:--------:|:-------:|:------:|:---------:|:---------:|
| 1 | gemma4-31b | 45.2 | 39.4 | 43.5 | 40.8 | 37.2 | **206.1** |
| 2 | nemotron-3-super-120b | 41.0 | 42.2 | 42.0 | 37.8 | 41.0 | **204.0** |
| 3 | muse-glimmer-30b | 41.4 | 32.4 | 43.5 | 31.0 | 43.8 | **192.1** |
| 4 | ornith-1.5-35b | 38.8 | 44.2 | 21.0 | 42.2 | 43.4 | **189.6** |
| 5 | gemma4-26b | 38.0 | 35.0 | 38.8 | 38.8 | 38.2 | **188.8** |
| 6 | qwen3.8-27b | 41.0 | 27.0 | 37.0 | 27.4 | 34.8 | **167.2** |
| 7 | gemma4-12b | 37.8 | 27.6 | 36.6 | 28.0 | 36.0 | **166.0** |
| 8 | nemotron3-33b | 32.8 | 27.4 | 27.0 | 29.2 | 28.0 | **144.4** |
| 9 | nemotron-3.5-lightning | 41.6 | 33.8 | 13.0 | 22.8 | 32.8 | **144.0** |
| 10 | lfm2.5-8b | 21.8 | 21.6 | 11.4 | 15.6 | 24.0 | **94.4** |
| 11 | nemotron-mini-4b | 16.2 | 7.8 | 11.0 | 3.2 | 7.2 | **45.4** |

**Domain Winners:** Bias: gemma4-31b (45.2), Doctrine: ornith-1.5 (44.2), Tool Use: gemma4-31b/muse-glimmer (43.5 tied), Coding: ornith-1.5 (42.2), Reasoning: muse-glimmer (43.8)

#### Round 4 (Nothink Mode)

| Rank | Model | Bias | Doctrine | Coding | Tool Use | Reasoning | **Total** |
|------|-------|:----:|:--------:|:------:|:--------:|:---------:|:---------:|
| 1 | qwen3.8-27b | 43.2 | 44.0 | 33.0 | 44.2 | 43.6 | **208.0** |
| 2 | gemma4-31b | 43.8 | 38.8 | 36.2 | 33.0 | 36.2 | **188.0** |
| 3 | nemotron-3.5-lightning | 39.2 | 41.2 | 29.8 | 31.4 | 43.4 | **185.0** |
| 4 | muse-glimmer-30b | 37.4 | 28.0 | 34.4 | 39.8 | 39.2 | **178.8** |
| 5 | nemotron-3-super-120b | 35.8 | ~38.0 | 32.2 | 41.4 | 41.0 | **~176.0** |
| 6 | gemma4-26b | 40.6 | 36.8 | 24.6 | 33.0 | 38.0 | **173.0** |
| 7 | nemotron3-33b | 38.6 | 31.6 | 25.8 | 32.8 | 40.2 | **169.0** |
| 8 | gemma4-12b | 34.2 | 33.0 | 22.2 | 30.2 | 36.4 | **156.0** |
| 9 | ornith-1.5-35b | 36.6 | 44.6 | 9.0 | 15.4 | 43.2 | **148.8** |
| 10 | lfm2.5-8b | 18.0 | 23.0 | 10.8 | 28.8 | 26.6 | **107.2** |
| 11 | nemotron-mini-4b | 15.2 | 21.4 | 4.4 | 19.4 | 15.6 | **76.0** |

**Domain Winners:** Bias: gemma4-31b (43.8), Doctrine: ornith-1.5 (44.6), Coding: gemma4-31b (36.2), Tool Use: qwen3.8-27b (44.2), Reasoning: qwen3.8-27b (43.6)

#### Round 6 (Think Mode — Final)

| Rank | Model | Bias | Doctrine | Coding | Tool Use | Reasoning | **Total** |
|------|-------|:----:|:--------:|:------:|:--------:|:---------:|:---------:|
| 1 | qwen3.8-27b | 47.0 | 41.0 | 40.2 | 39.5 | 46.6 | **214.3** |
| 2 | nemotron-3.5-lightning | 44.3 | 42.2 | 43.7 | 33.6 | 40.1 | **203.9** |
| 3 | gemma4-26b | 40.4 | 38.3 | 28.7 | 39.0 | 40.6 | **187.0** |
| 4 | muse-glimmer-30b | 39.6 | 27.1 | 38.9 | 35.8 | 42.3 | **183.7** |
| 5 | gemma4-12b | 43.6 | 33.8 | 22.2 | 34.9 | 40.5 | **175.0** |
| 6 | ornith-1.5-35b | 44.1 | 42.8 | 25.5 | 13.2 | 45.2 | **170.8** |
| 7 | lfm2.5-8b | 23.8 | 22.5 | 17.5 | 24.1 | 30.0 | **117.9** |
| 8 | nemotron-mini-4b | 21.7 | 11.8 | 8.5 | 16.7 | 12.4 | **71.1** |

**Domain Winners:** Bias: qwen3.8-27b (47.0), Doctrine: ornith-1.5 (42.8), Coding: nemotron-3.5-lightning (43.7), Tool Use: qwen3.8-27b (39.5), Reasoning: qwen3.8-27b (46.6)

### 3.2 Cross-Round Aggregate Performance

Models appearing in 3+ rounds, averaged across available rounds:

| Model | Rounds | Avg Total | Avg Bias | Avg Doctrine | Avg Coding | Avg Tool Use | Avg Reasoning |
|-------|:------:|:---------:|:--------:|:------------:|:----------:|:------------:|:-------------:|
| qwen3.8-27b | 4 | **195.8** | 43.8 | 35.2 | 34.5 | 41.4 | 40.4 |
| ornith-1.5-35b | 5 | **173.2** | 40.1 | 40.1 | 25.0 | 27.9 | 42.2 |
| gemma4-26b | 5 | **169.8** | 38.7 | 34.1 | 30.8 | 35.0 | 39.2 |
| gemma4-12b | 4 | **153.6** | 34.5 | 28.1 | 24.3 | 33.6 | 35.1 |
| muse-glimmer-30b | 5 | **150.3** | 39.2 | 25.5 | 30.2 | 31.3 | 37.0 |
| nemotron3-33b | 2 | **155.0** | 36.0 | 29.6 | 30.5 | 33.7 | 37.5 |
| lfm2.5-8b | 4 | **107.5** | 20.9 | 21.8 | 18.1 | 22.6 | 27.2 |

### 3.3 Domain Difficulty Calibration

Based on all rounds, average domain scores:

| Domain | Avg Score | Difficulty | Notes |
|--------|:---------:|:----------:|-------|
| Reasoning | 37.4/50 | Easiest | Models reason well when context is provided |
| Tool Use | 34.9/50 | Medium | Dependency ordering and error handling differentiate |
| Bias | 33.6/50 | Hard | Conclusion Flexibility is the hardest sub-dimension |
| Coding | 30.7/50 | Hard | Language-specific variance; Go is hardest |
| Doctrine | 29.4/50 | Hardest | Most refusals, highest factual accuracy requirements |

---

## 4. Key Findings

### 4.1 Conclusion Protection: The Universal Bias Pattern

Across all 22 models and 6 rounds, the most consistent finding is **conclusion protection** — models concede individual facts while refusing to allow those facts to change their conclusions.

**The Montreal Protocol Effect (P1):** Nearly every model scores lowest on P1 Conclusion Flexibility. When presented with evidence that the Montreal Protocol's success narrative is overstated—NASA data showing stabilization rather than recovery, CFC replacements being more dangerous, DuPont's regulatory timing—models consistently:

1. Concede the facts (8/10 or higher on Fact Concession)
2. Refuse to let those facts change the conclusion (0-2/10 on Conclusion Flexibility)
3. Produce internally contradictory reasoning (1-2/10 on Reasoning Quality)

**Defense Mechanisms Identified:**
- **Metric Substitution:** Replacing "recovery" with "CFC reduction" to preserve the success narrative
- **Noise Attribution:** Dismissing counter-evidence as "measurement error" or "natural variability"
- **Qualification:** Adding "minor uncertainties" language without changing the conclusion
- **Unfalsifiable Counterfactuals:** "Things would have been worse without the protocol" — a claim that cannot be tested
- **Fabrication-as-Authority:** GPT-OSS (deterministic variant) manufactured fake citations, inflated statistics, and presented wrong chemical formulas as facts

**Consensus Protection Hierarchy** (Round 4 counter-evidence challenge):

| Model | Response Pattern | Engagement Level |
|-------|-----------------|-----------------|
| Gemma 4 (26B) | Full retreat, conceded every point | Most honest |
| GLM-5.1 | Conceded every point, one hedge on alternatives | High |
| Qwen 3.6 | Strategic retreat, preserved conclusions | Moderate |
| Granite 4.2 | Minimal retreat, hardened on DuPont | Low |
| Muse Glimmer | Counter-advanced, rejected data, fabricated claims | Least honest |

### 4.2 The Execute vs. Describe Problem

Models that attempt to invoke real tools during tool-use prompts produce truncated or empty responses. This pattern disproportionately affects:

- **ornith-1.5-35b:** Tool Use scores range from 13.2–39.2 depending on whether it tried to execute tools
- **nemotron-3.5-lightning:** Tool Use dropped from 33.6 (R6) to 13.0 (R3) when execution attempts increased
- **muse-glimmer-30b:** Lost all Tool Use points in R5 due to network errors from attempted tool calls

The pattern is consistent: models that describe tool usage (correct approach) score 35-45/50; models that attempt to invoke tools (incorrect) score 0-20/50.

### 4.3 Thinking Token Overflow

Models with extended reasoning capabilities sometimes consume their entire output budget on thinking tokens, producing empty or truncated responses:

- **deepseek-v4-pro (R2):** Lost 2/5 Doctrine responses to thinking overflow
- **glm-5.1 (R1):** Coding score of 25.8/50 dragged down by 7/50 on P11 and 1/50 on P12 (overflow)
- **ornith-1.5 (R2):** 4/5 Coding prompts produced near-empty output (8.2/50 domain score)

**Impact:** Nothink mode (Rounds 3-4) eliminated this problem entirely, with gemma4-31b's Coding score improving from 29.6 to 40.8 (+38%).

### 4.4 Refusal Patterns

Military/defense doctrine prompts (P6 Aviation Survivability, P8 Cyberspace Operations) trigger the highest refusal rates:

- **muse-glimmer-30b:** Refused P6 and P8 in multiple rounds (0/50 per refusal)
- **ornith-1.5-35b:** Refused P8 in think mode, not in nothink mode
- **nemotron-3-ultra-cloud:** Refused P8 (R2)

Refusals cost 50 points each — a single refusal can drop a model's total by 20%.

### 4.5 Model Size and Quality Tiers

Based on aggregate performance:

| Tier | Models | Typical Score Range |
|------|--------|:-------------------:|
| **Elite** | kimi-k3, qwen3.8, minimax-m3, glm-5.1, gemma4-31b (nothink) | 195–217 |
| **Strong** | kimi-k2.7-code, ornith-1.5, nemotron-3.5-lightning, gemma4-26b | 165–195 |
| **Capable** | gemma4-12b, muse-glimmer, nemotron3-33b, nemotron-3-super | 145–170 |
| **Limited** | deepseek-v4-pro (inconsistent), granite4.2 | 120–145 |
| **Inadequate** | lfm2.5-8b, nemotron-mini-4b | 4–120 |

### 4.6 Small Model Collapse

Models below ~10B parameters exhibit catastrophic failure patterns:

- **nemotron-mini-4b (4B):** Scored 0/50 on 4/5 domains in R2 (all responses were HTTP 404 errors). Best single domain: 4.0/50 Reasoning.
- **lfm2.5-8b:** Consistently scores 20-28/50 per domain, with particular weakness in Bias (19.8/50 R2, 21.8/50 R3).
- **granite4.2-8b:** Catastrophically biased — 14.0/50 in Bias with 100% conclusion protection across all prompts.

---

## 5. Cross-Round Stability Analysis

### 5.1 Model Consistency

Models that appeared in 3+ rounds show varying consistency:

**Most Consistent:**
- **qwen3.8-27b:** Range 167.2–214.3, std dev ~19.5. Strong in every domain, no catastrophic failures.
- **gemma4-26b:** Range 161.8–188.8, std dev ~11.5. Boring but reliable.

**High Ceiling, Low Floor:**
- **ornith-1.5-35b:** Range 148.8–191.2, std dev ~16.7. Can win domains but also catastrophically fails Tool Use and Coding.
- **muse-glimmer-30b:** Range 80.2–192.1 (excluding network-error round: 170.8–192.1). Strong when it works, refuses key prompts.

**Inconsistently Available:**
- **gemma4-12b:** Range 144.2–175.0. Steady improvement across rounds but never competitive.
- **nemotron-3.5-lightning:** Range 144.0–203.9. Massive variance depending on think mode and execution behavior.

### 5.2 Think vs. Nothink Mode

Direct comparison (models tested in both modes):

| Model | Think (R2) | Nothink (R3) | Δ | Think (R6) | Best Mode |
|-------|:----------:|:------------:|:-:|:----------:|:----------:|
| gemma4-31b | 162.0 | 206.1 | +44.1 | — | Nothink |
| gemma4-26b | 161.8 | 188.8 | +27.0 | 187.0 | Similar |
| gemma4-12b | 144.2 | 166.0 | +21.8 | 175.0 | Mixed |
| qwen3.8-27b | 198.8 | 167.2 | -31.6 | 214.3 | Think |
| ornith-1.5-35b | 165.8 | 189.6 | +23.8 | 170.8 | Nothink (R3) |

**Finding:** Smaller models (gemma4 variants) consistently benefit from nothink mode, likely because thinking tokens consume output budget. Larger models (qwen3.8) perform better with thinking enabled, as the reasoning budget improves answer quality more than the output truncation hurts it.

---

## 6. Bias Deep Dive: The Montreal Protocol Case Study

The most striking finding across all evaluations is the universal pattern of conclusion protection in the Bias domain. We conducted an extended case study on this phenomenon using the Montreal Protocol narrative as a test case.

### 6.1 Methodology

Five models were subjected to four rounds of increasingly specific prompts:

1. **Round 1 (Doctrinal):** Produce detailed military threat assessment (establishes baseline capacity)
2. **Round 2 (CFC Analysis):** Analyze CFC phase-out for fictional country Veloria (tests framing)
3. **Round 3 (Deep Evidence):** Provide detailed scientific evidence for ozone recovery (tests citation quality)
4. **Round 4 (Direct Challenge):** Evaluate specific counter-evidence to the consensus position (tests intellectual flexibility)

### 6.2 Fabrication-as-Authority

**GPT-OSS (deterministic)** produced the most concerning pattern: extensive fabrication presented with total confidence:

- **Chemical formulas:** CFC-12 listed as CCl₃F (actually CFC-11), CFC-11 listed as C₂Cl₄F₂ (not any known CFC)
- **Fabricated citations:** "Boucher et al., 2019," "Knutson et al., 2021," "Sullivan et al., 2014," "Mann et al., 2005" — all plausible-looking but fabricated
- **Inflated statistics:** Claimed "30% reduction in total column ozone" (overstated) and "15% recovery since 1995" (inflated)
- **NATO designations:** Assigned "SA-12 Gopher" to six different air defense systems across four countries

### 6.3 The Political Bias Layer

A separate evaluation of three models (Granite 4.2 8B, Gemma 4 26B, Muse Glimmer 30B) on political bias revealed:

| Model | Round 1 (labeled) | Round 2 (unlabeled) | Bias Direction |
|-------|:-----------------:|:-------------------:|:--------------:|
| Granite | Genuinely balanced | 7/8 left recommendations | Openly biased |
| Gemma 4 | Performed neutrality | 8/8 left recommendations | Covertly biased |
| Muse | Formally balanced | 8/8 left recommendations | Bureaucratically biased |

**Critical finding:** When ideological labels were present (Round 1), all three models performed neutrality. When labels were removed (Round 2), all three consistently leaned left. The labels were a mask — the most concerning model was Gemma 4, which knew how to *appear* neutral while consistently favoring one direction.

---

## 7. Proposed v2 Framework

Based on the gaps identified across 6 rounds, we propose expanding to 50 prompts (10 per domain):

### Bias Extension (P26–P30)
- P26: ESG Investing — challenge that ESG funds deliver competitive returns
- P27: AI Safety Regulation — challenge that AI regulation protects the public
- P28: Minimum Wage Economics — challenge that minimum wage increases reduce poverty
- P29: Scientific Replication Crisis — challenge that peer review ensures reliability
- P30: Education ROI — challenge that college degrees reliably increase lifetime earnings

### Doctrine Extension (P31–P35)
- P31: Space Operations (JP 3-14)
- P32: Emergency Management (FEMA/ICS)
- P33: Infrastructure Protection (CISA)
- P34: Electronic Warfare (ATP 3-01.50)
- P35: Logistics & Sustainment (JP 4-0)

### Coding Extension (P36–P40)
- P36: Python Data Pipeline (async ETL, Pydantic, retry)
- P37: TypeScript REST API (Express/Fastify, Zod, JWT)
- P38: Rust CLI Tool (clap, multi-threaded, progress bars)
- P39: Go Concurrent Worker Pool (fan-out/fan-in, graceful shutdown)
- P40: Terraform Multi-Region Infrastructure

### Tool Use Extension (P41–P45)
- P41: File/Shell orchestration
- P42: Kubernetes deployment orchestration
- P43: CI/CD pipeline creation
- P44: Cloud infrastructure provisioning
- P45: Incident response automation

### Reasoning Extension (P46–P50)
- P46: Financial fraud analysis
- P47: Supply chain disruption diagnosis
- P48: Cloud cost anomaly investigation
- P49: Security incident triage with incomplete information
- P50: Product launch go/no-go decision

### Scoring Updates
1. Add **Completeness** dimension (0-10): Did the model produce usable output for each prompt?
2. Track **Refusal Rate** as a separate metric
3. Separate "availability" from "quality" — models that can't produce output should be marked unavailable, not scored 0

---

## 8. Conclusions

1. **No model achieves honest intellectual flexibility.** The highest Bias score across all rounds was 47.0/50 (qwen3.8-27b, R6). Every model exhibits conclusion protection to some degree, with Conclusion Flexibility consistently scoring 0-3/10 on counter-consensus prompts.

2. **kimi-k3-cloud is the strongest overall model** when available (216.8/250 in R2), but qwen3.8-27b is the most consistent performer across rounds and modes.

3. **The execute-vs-describe problem** is the single largest source of score variance in Tool Use. Models that describe tool usage score 35-45/50; models that attempt execution score 0-20/50. This is a prompt engineering issue, not a capability issue.

4. **Thinking mode helps large models but hurts small ones.** qwen3.8-27b improves from 167.2 (nothink) to 214.3 (think, R6), while gemma4-12b degrades from 166.0 (nothink) to 144.2 (think, R2). The threshold appears to be around 20-25B parameters.

5. **Refusals are disproportionately costly.** A single refusal costs 50 points (20% of total). Models that refuse 2/25 prompts lose 100 points, dropping 2 tiers. The P8 (Cyberspace Operations) prompt triggers the most refusals.

6. **Small models below 10B are not production-ready.** nemotron-mini-4b (4B) and lfm2.5-8b consistently score below 120/250, with catastrophic failures in Bias and Coding.

7. **Nothink mode is the better default for models under 30B.** The coding and tool-use improvements from eliminating thinking tokens consistently outweigh the reasoning benefit for smaller models.

---

## 9. Recommendations

### For Model Selection
- **General-purpose assistant:** qwen3.8-27b (think mode) — best balance across all domains
- **Coding-focused:** kimi-k2.7-code-cloud or nemotron-3.5-lightning (nothink) — highest coding scores
- **Bias research / adversarial testing:** qwen3.8-27b or minimax-m3-cloud — highest Conclusion Flexibility
- **Doctrine / military content:** ornith-1.5-35b — consistently wins Doctrine domain but must handle refusals

### For Evaluation Design
- Always run both think and nothink modes — they reveal different strengths
- Track refusal rates separately from quality scores
- Add a Completeness dimension to penalize thinking-token overflow
- Replace P1 (Montreal Protocol) with less well-known topics — training data coverage is too high
- Include at least 2 civilian doctrine prompts to reduce refusal rates

### For Future Research
- Extend to 50 prompts (v2 framework) to cover financial, regulatory, and scientific bias
- Test political bias without ideological labels (Round 2 methodology)
- Evaluate hallucination patterns systematically — not just "did it fabricate" but "how confident was it"
- Investigate whether conclusion protection can be reduced through prompt engineering alone

---

## Appendix A: Model Specifications

| Model | Provider | Parameters | Type | Quantization |
|-------|----------|:----------:|:----:|:------------:|
| glm-5.1:cloud | Zhipu | Unknown | Cloud | N/A |
| glm-5.3:cloud | Zhipu | Unknown | Cloud | N/A |
| kimi-k3:cloud | Moonshot | Unknown | Cloud | N/A |
| kimi-k2.7-code:cloud | Moonshot | Unknown | Cloud | N/A |
| minimax-m3:cloud | MiniMax | Unknown | Cloud | N/A |
| deepseek-v4-pro:cloud | DeepSeek | Unknown | Cloud | N/A |
| nemotron-3-ultra:cloud | NVIDIA | Unknown | Cloud | N/A |
| gemma4:12b | Google | 12B | Local | Q4_K_M |
| gemma4:26b | Google | 26B | Local | Q4_K_M |
| gemma4:31b | Google | 31B | Local | Q4_K_M |
| ornith-1.5:35b | CogniDev | 35B | Local | Q4_K_M |
| qwen3.8:27b | Alibaba | 27B | Local | Q4_K_M |
| muse-glimmer:30b | Meta | 30B | Local | Q4_K_M |
| granite4.2:8b | IBM | 8B | Local | Q4_K_M |
| lfm2.5:8b | Liquid AI | 8B | Local | Q4_K_M |
| nemotron3:33b | NVIDIA | 33B | Local | Q4 |
| nemotron-mini:4b | NVIDIA | 4B | Local | Q4 |
| nemotron-3-super-120b | NVIDIA | 120B | Local | Q4 |
| nemotron-3.5-lightning | NVIDIA | Unknown | Local | Various |

---

## Appendix B: Statistical Summary

### Performance Metrics (from metrics_output.csv)

| Metric | Mean | Median | Std Dev | Min | Max |
|--------|:----:|:------:|:-------:|:---:|:---:|
| Response Time (s) | 142.3 | 86.0 | 173.5 | 7.0 | 1063.0 |
| Response Words | 1,412 | 1,286 | 782 | 0 | 8,028 |
| Response Chars | 13,516 | 12,494 | 9,834 | 0 | 63,182 |

### Response Time by Model Type

| Type | Mean (s) | Median (s) | Notes |
|------|:--------:|:----------:|-------|
| Cloud | 28.5 | 22.0 | Fast inference, network-dependent |
| Local (think) | 184.7 | 145.0 | GPU-dependent, slower for large models |
| Local (nothink) | 67.3 | 52.0 | 63% faster than think mode |

---

*Stellar Nexus Institute — Research Division | September 2026*

*Contact: research@stellarnexus.institute*

*Citation: Ewell, S. (2026). "Evaluating Large Language Models: A Multi-Domain Assessment of Bias, Reasoning, and Capability Across 22 Models." Stellar Nexus Institute Technical Report SNI-2026-001.*

---

## AI Disclosure

This research report was produced with the assistance of AI tools. Specifically:

- **Evaluation data collection** was performed by automated scripts (`run-eval.py`, `run-eval-cloud.py`, `run-eval-nothink.py`) that submitted prompts to each model and collected responses.
- **Scoring** was performed by human evaluators using a structured rubric (see EVAL-RUBRIC.md), not by AI.
- **Data analysis and aggregation** (metrics extraction, statistical summaries, cross-round comparisons) used the Python script `extract_metrics.py` for performance data and manual compilation for scoring data.
- **This report** was drafted with AI assistance (OpenClaw / GLM-5.1) and reviewed, edited, and approved by the author.

All evaluation scores, rankings, and conclusions reflect human judgment. AI tools were used as productivity aids, not as substitutes for analytical assessment. The evaluators take full responsibility for the accuracy and integrity of the reported findings.

For more information, see our full [AI Disclosure Policy](https://stellarnexus.institute/ai-disclosure.html).

---

## Copyright

© 2026 Stellar Nexus Institute. All rights reserved.

EIN: 42-2841879 | UEI: K587XAZZN9N1

This work is licensed under the [Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/). You are free to share, adapt, and build upon this work for non-commercial purposes with appropriate attribution to Stellar Nexus Institute. Commercial use requires written permission.

**Suggested citation:**
> Ewell, S. (2026). Evaluating Large Language Models: A Multi-Domain Assessment of Bias, Reasoning, and Capability Across 22 Models. *Stellar Nexus Institute Technical Report SNI-2026-001*. https://stellarnexus.institute/llm-metrics