Databricks Benchmarks AI Coding Agents on 10-Plus-Language Codebase, Finds $1.28 GLM 5.2 Near Top Tier
Updated
Updated · Databricks · Jul 8
Databricks Benchmarks AI Coding Agents on 10-Plus-Language Codebase, Finds $1.28 GLM 5.2 Near Top Tier
3 articles · Updated · Databricks · Jul 8
Summary
Databricks said its internal benchmark of AI coding agents on real engineering tasks found three capability tiers and showed no single vendor dominates the cost-performance frontier.
GLM 5.2 emerged as a standout open model, statistically tied with Opus 4.8 on quality while costing $1.28 per task versus $1.94, prompting Databricks to consider it a daily-driver option.
Task-level costs often diverged from token pricing: Sonnet 5 was about 1.7x cheaper per token than Opus 4.8 but cost $2.09 per task versus $1.94 and scored 81% against 87%.
Harness design also materially changed efficiency, with Pi cutting cost by more than 2x in some runs by sending about 3x less context per turn while preserving quality.
The benchmark was built from recent human-written pull requests across a multi-million-line, 10-plus-language codebase, with sealed git history and test-based grading to avoid leakage and overfitting.