Reproducible Benchmark Comparisons
Third-party comparison platforms often stop at metadata. We're tracking the gaps — and building toward open, reproducible evals for Chinese open-source coding models.
The Data Gap Today
Platforms like BenchLM let you compare models side-by-side, but for many Chinese open-source pairs — e.g., DeepSeek Coder 2.0 vs LongCat-2.0 — only metadata (context window, parameter count) is available. Actual benchmark scores show "coming soon" or are missing entirely.
Independent evaluators (Vellum, LLM-Stats, AceCloud, Atlas Cloud) publish positioning tables and selected scores, but rarely run the same harness across all models with published configs. Developers searching for "LongCat vs DeepSeek SWE-bench" often land on release notes, not reproducible numbers.
What We Track (Public Reports)
Until independent runs are published, we aggregate publicly reported scores with source attribution:
| Benchmark | LongCat-2.0 | DeepSeek V4-Pro | Qwen3.6 Plus | Kimi K2.6 | GLM-5.1 |
|---|---|---|---|---|---|
| SWE-bench Pro | 59.5 | — | — | — | 58.4 |
| SWE-Bench Verified | — | 80.6 | 78.8 | — | — |
| Terminal-Bench 2.0 | — | — | 61.6% | 66.7% | — |
| LiveCodeBench | — | 93.5 | — | — | — |
| Context window | 1M | 1M | 1M | 256K | — |
"—" indicates no comparable public score found for that suite. Different benchmarks measure different task types — do not rank models on a single cell.
Planned Reproducible Evals
We aim to publish runs with:
- Fixed harness versions — pinned SWE-bench, Terminal-Bench, and LiveCodeBench commits
- API configs documented — temperature, max tokens, tool schemas, retry policy
- Cost per run — tokens consumed and USD spent alongside scores
- Raw logs — where licensing permits, link to failure cases for debugging
Priority pairs: LongCat-2.0 vs DeepSeek V4-Pro on agentic coding suites; LongCat-2.0 vs Kimi K2.6 on long-session terminal tasks; LongCat-2.0 vs GLM-5.1 on web-agent scenarios.