Agentic NL2SQL Just Hit 91.7% on Real Enterprise Schemas

Natural-language-to-SQL has been stuck in a familiar trap: dazzling benchmark scores, disappointing production results. Earlier this month, a team at DevRev published a new arXiv paper that tested on 900 execution-verified, real-world enterprise queries — and a cost-aware agentic architecture answered 91.7% correctly in a single generation, 54.6 percentage points above the next-best baseline. What decided the outcome wasn't a bigger model. It was the architecture. Together with a second paper published the same week, it draws the clearest picture yet of what it takes for AI-powered data querying to survive real enterprise databases.

Why Real Enterprise Databases Are So Hard

A quick definition first: NL2SQL (also called Text-to-SQL) is the task of automatically translating natural-language questions into SQL queries, so business users can pull data without writing code. It's the engine underneath every "ask your data" product.

The catch: almost every impressive number to date came from clean academic benchmarks. On Spider 1.0 (2023-era results), leading models reached 86.6% execution accuracy. Swap in Spider 2.0 — enterprise warehouses with thousand-column tables, multiple SQL dialects, and graph-like table linkage — and the best code agent manages just 21.3%, while GPT-4o drops to 10.1%. When dbt Labs re-ran its own benchmark this April, top 2026 models writing SQL directly scored 64.5% on realistic questions. One in three answers was wrong — and the wrong ones looked exactly like the right ones.

Where 91.7% Comes From: Single Generation, Three Engineering Components

Back to the new paper (arXiv 2609.04641). It makes two contributions. First, the DevRev NL2SQL benchmark: 900 execution-verified queries over nested-type and link-graph schemas, plus a "Semantic Depth Score" (SDS) that measures analytical reasoning depth. Second, an architecture built for exactly this regime.

The key phrase is cost-aware single generation. Many agent systems buy accuracy by generating many candidate SQLs and voting, at the cost of multiplied inference spend. This architecture goes the other way: one generation, with the budget spent on three engineering components —

1. Schema selection: narrow down the relevant tables and link paths inside a graph-like nested schema, so the model doesn't get lost in hundreds of tables;

1. Metadata retrieval: pull column meanings, primary keys, and example values on demand instead of stuffing everything into context;

1. Error repair: fix failures at the point of execution validation rather than starting over.

On its own benchmark, the system reaches 91.7% answer correctness; on the public Spider 2.0 Snowflake dataset, it stays competitive with leading systems at a single-generation operating point. Same budget, different architecture — a completely different return.

The Biggest Lever Is Still Semantic Assets

The dbt Labs benchmark update from April tells the same story from another angle. On an enterprise insurance database, they compared raw SQL generation against routing through a semantic layer — where metric definitions, dimensions, and join paths are declared up front. Within the semantic layer's coverage, both frontier models answered 100% correctly. Outside it, the semantic layer scored 0% — not a calculation error, but a coverage boundary.

The fix is telling: they had an LLM write three additional dbt models that pre-fold the deep join hops. With coverage extended, the semantic layer hit 98.2% (Sonnet 4.6) and 100% (GPT-5.3 Codex), versus 84.1%–90.0% for raw SQL. A cross-test made the economics explicit: cranking reasoning effort from minimum to maximum left text-to-SQL wobbling between 50% and 65%, while the semantic-layer path held at 87%–100%. Budget spent on reasoning tokens returns almost nothing; budget spent on semantic modeling returns thirty to forty points.

This matches what we argued before: a semantic layer isn't documentation — it's a compiler. See our earlier pieces on the semantic-layer data agent architecture and the enterprise Text-to-SQL reality check.

Being Right Isn't Enough: Numerical Discipline Is Fragile

Accuracy is the entry ticket, not the finish line. A second arXiv paper the same week (2609.04806) is a cold shower for anyone fine-tuning domain models: researchers tuned a base instruction model into a "financial domain LLM" and a "numeracy-enhanced" variant, then measured how often they fabricated numbers in financial summarization. The result defies intuition — the base model's overt hallucination rate was 5.4%; after domain fine-tuning it jumped to 82.5%; the numeracy-enhanced version hit 98% (per the paper's abstract). The mechanism: template injection — the model slots memorized "typical values" into familiar report formats regardless of the input. Fine-tuning improved numerical competence while destroying numerical restraint.

The regulatory environment is tightening too. On September 7, China's Supreme People's Court issued its first judicial framework on AI-related disputes — 24 articles covering, among other things, liability for AI hallucination. For enterprises, a fast but untraceable querying system is no longer just an engineering risk; it's a compliance risk.

The playbook for trustworthy data querying is therefore clear: semantic assets as the foundation, deterministic validation as the backstop, auditable reasoning chains, and explicit abstention when the system doesn't know. We expanded on agent auditability in this article.

Closing Thoughts

Read together, these two papers tell a plain story: the ceiling of AI-powered data querying was never in model parameters — it's in whether an enterprise has organized its own data into machine-readable, verifiable structure. That's exactly what OntiCards data cards do: they turn table structures, column meanings, business definitions, and relationships into a "data dictionary + map + navigation" for agents. Our querying engine addresses data through cards, produces traceable results, and runs fully on-premise so your data never leaves your domain.

If you'd like to try natural-language querying against your own enterprise database, reach out at hello@onticards.com to request a test account.

References

• A Cost-Aware Agentic Architecture for NL-to-SQL over Nested Enterprise Schemas, with a New Benchmark — arXiv:2609.04641

• When Financial Fine-tuning Fails: A Three-Level Detectability Analysis of Numerical Hallucination in Domain-Adapted Language Models — arXiv:2609.04806

• Text-to-SQL Accuracy: Why Benchmarks Lie in Production — Particula Tech

• Semantic Layer vs Text-to-SQL: When Each Wins, and Why Mature Teams Use Both — Colrows

• Writing Context Into Your Data: AI-Ready Data, Semantic Layers, Knowledge Graphs, Ontologies — Bits, Bytes and Neural Networks

• Supreme People's Court issues judicial opinion on AI disputes (as covered in AI weekly) — Yesky