v1.0 · 150 tasks
PawBench
How (Model × Harness) combinations perform on production-grade tasks
Agent Performance = f(Model, Harness) The same 150 tasks across (Model × Harness) combinations — read each axis independently and separate the model's contribution from the harness's.
150 Tasks 6 Sources 3 Harnesses 7 Capabilities
Model × Harness Score Matrix
All 150 tasks (text + multimodal)
| Model | Hermes v2026.4.23 | OpenClaw v2026.4.24 | QwenPaw v1.1.3 | Avg | Δ |
|---|---|---|---|---|---|
| claude-opus-4.6 | 78.4 | 76.1 | 78.3 | 77.6 | +2.3 |
| deepseek-v4-pro | 72.1 | 75.4 | 75.6 | 74.4 | +3.6 |
| qwen3.7-maxtext-only | 72.3 | 72.5 | 77.6 | 74.1 | +5.4 |
| qwen3.6-max-previewtext-only | 68.1 | 75.1 | 78.3 | 73.9 | +10.3 |
| qwen3.6-plus | 70.4 | 73.6 | 75.0 | 73.0 | +4.6 |
| qwen3.6-27b | 68.2 | 72.9 | 72.7 | 71.3 | +4.7 |
| glm-5.1text-only | 63.2 | 68.5 | 71.1 | 67.6 | +7.9 |
| kimi-k2.6 | 66.4 | 66.6 | 66.6 | 66.5 | +0.2 |
| qwen3.6-35b-a3b | 56.7 | 67.8 | 68.3 | 64.3 | +11.5 |
| Avg | 68.4 | 72.1 | 73.7 | 71.4 |
Leaderboard
| # | Model | Harness | Overall ↓ | Automated | LLM Judge | Tasks | Updated |
|---|---|---|---|---|---|---|---|
| 1 | claude-opus-4.6 | Hermes | 78.4 | 82.6 | 90.8 | 150 | 2026-05-29 |
| 2 | qwen3.6-max-preview | QwenPaw | 78.3 | 87.2 | 81.1 | 150 | 2026-05-29 |
| 3 | claude-opus-4.6 | QwenPaw | 78.3 | 85.3 | 83.9 | 150 | 2026-05-29 |
| 4 | qwen3.7-max | QwenPaw | 77.6 | 84.6 | 82.9 | 150 | 2026-05-29 |
| 5 | claude-opus-4.6 | OpenClaw | 76.1 | 83.6 | 80.7 | 150 | 2026-05-29 |
| 6 | deepseek-v4-pro | QwenPaw | 75.6 | 83.7 | 80.6 | 150 | 2026-05-29 |
| 7 | deepseek-v4-pro | OpenClaw | 75.4 | 83.5 | 80.7 | 150 | 2026-05-29 |
| 8 | qwen3.6-max-preview | OpenClaw | 75.1 | 84.4 | 81.7 | 150 | 2026-05-29 |
| 9 | qwen3.6-plus | QwenPaw | 75.0 | 84.6 | 79.1 | 150 | 2026-05-29 |
| 10 | qwen3.6-plus | OpenClaw | 73.6 | 82.3 | 77.2 | 150 | 2026-05-29 |
| 11 | qwen3.6-27b | OpenClaw | 72.9 | 82.2 | 77.5 | 150 | 2026-05-29 |
| 12 | qwen3.6-27b | QwenPaw | 72.7 | 83.8 | 77.6 | 150 | 2026-05-29 |
| 13 | qwen3.7-max | OpenClaw | 72.5 | 79.3 | 75.9 | 150 | 2026-05-29 |
| 14 | qwen3.7-max | Hermes | 72.3 | 80.4 | 79.9 | 150 | 2026-05-29 |
| 15 | deepseek-v4-pro | Hermes | 72.1 | 81.2 | 79.0 | 150 | 2026-05-29 |
| 16 | glm-5.1 | QwenPaw | 71.1 | 85.0 | 83.0 | 150 | 2026-05-29 |
| 17 | qwen3.6-plus | Hermes | 70.4 | 80.5 | 76.6 | 150 | 2026-05-29 |
| 18 | glm-5.1 | OpenClaw | 68.5 | 72.6 | 74.5 | 150 | 2026-05-29 |
| 19 | qwen3.6-35b-a3b | QwenPaw | 68.3 | 77.8 | 68.2 | 150 | 2026-05-29 |
| 20 | qwen3.6-27b | Hermes | 68.2 | 78.7 | 75.0 | 150 | 2026-05-29 |
| 21 | qwen3.6-max-preview | Hermes | 68.1 | 76.8 | 77.6 | 150 | 2026-05-29 |
| 22 | qwen3.6-35b-a3b | OpenClaw | 67.8 | 77.6 | 70.8 | 150 | 2026-05-29 |
| 23 | kimi-k2.6 | QwenPaw | 66.6 | 80.1 | 69.0 | 150 | 2026-05-29 |
| 24 | kimi-k2.6 | OpenClaw | 66.6 | 79.3 | 70.9 | 150 | 2026-05-29 |
| 25 | kimi-k2.6 | Hermes | 66.4 | 78.7 | 70.3 | 150 | 2026-05-29 |
| 26 | glm-5.1 | Hermes | 63.2 | 75.3 | 71.3 | 150 | 2026-05-29 |
| 27 | qwen3.6-35b-a3b | Hermes | 56.7 | 65.8 | 61.5 | 150 | 2026-05-29 |
How it works
Three steps from model to score
- 1
Pick a model and harness
Any OpenAI-compatible endpoint or local model; harness can be QwenPaw / OpenClaw / Hermes
- 2
Run in Docker
Each task runs in an isolated container with workspace files mounted, strict timeouts and retries
- 3
Automated + LLM grading
Python grade() functions plus an LLM judge; hybrid tasks zero the LLM share if automated < 0.75