Natural Language to SQL: Why Accuracy Depends on Your Semantic Layer

Natural language to SQL accuracy is the metric that determines whether your text-to-SQL investment pays off or becomes an expensive demo. Without semantic grounding, accuracy hovers around 30-40% on real enterprise schemas. With a semantic layer that exposes governed metrics, joins, and column descriptions, accuracy on production workloads exceeds 90%.

Natural language to SQL accuracy is the metric that determines whether your text-to-SQL investment pays off or becomes an expensive demo. The promise is compelling: business users ask questions in plain English, and an AI system generates the correct SQL query. The reality is that accuracy rates without semantic grounding hover around 30-40% on real-world enterprise schemas — a number that makes the technology unusable for anything beyond toy demonstrations.

dbt Labs published benchmark results showing that semantic layer integration improves text-to-SQL accuracy by 2-3x on complex enterprise queries. Google's research echoes this finding, showing 66% accuracy degradation when LLMs query raw tables versus semantically enriched schemas. The data is clear: your semantic layer is not optional infrastructure — it is the difference between a working product and a failed experiment.

Why Raw Schema Text-to-SQL Fails at Enterprise Scale

Text-to-SQL works remarkably well on simple schemas. Give an LLM a database with five tables and clear column names, and it will generate correct queries 80%+ of the time. The problem emerges when you point that same LLM at an enterprise data warehouse with 1,000+ tables, cryptic column names like dim_cust_attr_7, and business logic encoded in views three layers deep.

• •Schema complexity. Enterprise warehouses average 500-2,000 tables. Without guidance, the LLM must guess which of 15 tables containing 'customer' data is the right one for any given question.
• •Naming conventions. Column names like rev_net_adj_q or mrr_churn_gross_ex_pause encode business logic that the LLM cannot decode from the name alone.
• •Join complexity. Real queries often require 5-10 table joins with specific join conditions. Missing one condition or joining on the wrong key produces silently wrong results.
• •Business logic in SQL. Metric definitions often involve CASE statements, date range filters, exclusion lists, and aggregation rules that vary by context. 'Monthly active users' might have three different definitions depending on the product surface.
• •Temporal semantics. Questions like 'what was last quarter's revenue' require knowing your fiscal calendar, timezone conventions, and whether 'last quarter' means the most recently completed quarter or the trailing 90 days.

The Semantic Layer Advantage: Benchmark Data

Recent benchmarks from multiple sources consistently show the same pattern: semantic layers dramatically improve text-to-SQL accuracy.

The improvement is most dramatic on complex queries — exactly the queries that business users actually need answered. Simple queries like 'how many orders yesterday' work fine without a semantic layer. It is the nuanced questions — 'what is our net revenue retention by cohort excluding trials' — where the semantic layer becomes essential.

What a Semantic Layer Actually Provides to the LLM

A semantic layer translates between human business concepts and physical database schemas. For text-to-SQL specifically, it provides four critical capabilities:

1. Metric definitions. Instead of the LLM guessing how to calculate 'net revenue,' the semantic layer provides the exact formula: SUM(amount) WHERE type = 'recognized' AND status != 'refunded' FROM finance.transactions. This eliminates the most common and most dangerous type of error.

2. Entity resolution. When a user says 'customer,' the semantic layer maps that to dim_customers in the warehouse, specifies the primary key, identifies the correct join paths to other entities, and notes any required filters (like is_test = false).

3. Relationship graphs. The semantic layer encodes which tables can be joined, on which keys, and in what order. This prevents the LLM from generating Cartesian products or joining on the wrong foreign key.

4. Contextual constraints. Business rules like 'always filter deleted records,' 'exclude internal test accounts,' and 'use fiscal quarters not calendar quarters' are encoded once and applied automatically to every query.

Architecture: Integrating Semantic Layers with Text-to-SQL

There are three primary architecture patterns for connecting a semantic layer to a text-to-SQL system:

Pattern A: Semantic-first query generation. The LLM generates queries against semantic model definitions (metrics, dimensions, entities) rather than raw SQL. The semantic layer then compiles these into physical SQL. This is the approach used by dbt Semantic Layer and similar tools. It has the highest accuracy but requires a comprehensive semantic model.

Pattern B: Context-enriched SQL generation. The LLM generates SQL directly, but its context window is enriched with semantic definitions, table descriptions, and relationship metadata. This is more flexible but slightly less accurate because the LLM can still deviate from the provided context.

Pattern C: Hybrid with validation. The LLM generates SQL with context enrichment, then a validation layer checks the generated query against semantic rules before execution. Queries that reference ambiguous terms or violate business rules are flagged for revision. Data Workers implements this pattern through MCP-native agents that provide context enrichment and post-generation validation.

Building an Effective Semantic Layer for Text-to-SQL

A semantic layer is only as good as its coverage. Here is how to build one that actually improves accuracy:

• •Start with the top 50 questions. Survey your business users and identify the 50 most common analytical questions. Define semantic models that cover these questions first. This typically addresses 80% of text-to-SQL requests.
• •Define metrics, not just tables. A table description says 'this table contains order data.' A metric definition says 'total orders = COUNT(*) FROM orders WHERE status = completed AND is_test = false AND order_date >= [start_date].' The metric definition is what the LLM actually needs.
• •Encode join paths explicitly. Do not rely on foreign key constraints in your warehouse — many warehouses have them disabled for performance. Explicitly define which tables join on which keys and in what order.
• •Version your semantic definitions. Business logic changes quarterly. Your semantic layer should be version-controlled, reviewed, and deployed like code. Data Workers integrates with Git-based workflows through its documentation framework.
• •Test with real questions. Build a test suite of 100+ natural language questions with known correct SQL. Run these against your text-to-SQL system regularly to catch accuracy regressions.

Common Pitfalls in Text-to-SQL Deployments

Even with a semantic layer, text-to-SQL deployments can fail. The most common pitfalls include:

The Role of MCP in Text-to-SQL Architecture

The Model Context Protocol (MCP) is emerging as the standard interface between AI models and data tools. For text-to-SQL, MCP provides a structured way to deliver semantic context to any LLM without building custom integrations for each model provider.

Data Workers' 15 MCP-native agents serve as the bridge between your semantic layer and your LLM. When a user asks a natural language question, the agents retrieve relevant metric definitions, table relationships, quality signals, and governance rules — then deliver them to the LLM in a structured format that maximizes query accuracy. Because MCP is an open protocol, this works with Claude, GPT, Gemini, and any future MCP-compatible model.

This approach means you define your semantic layer once and it works across every AI tool in your stack — from text-to-SQL interfaces to automated reporting to ad-hoc analysis agents.

Measuring Text-to-SQL Accuracy in Production

Accuracy benchmarks on public datasets do not reflect real-world performance. You need to measure accuracy against your own data, with your own questions, using your own semantic definitions. Key metrics to track:

• •Execution success rate. What percentage of generated queries execute without errors? Target: 95%+.
• •Result correctness rate. Of queries that execute, what percentage return the correct answer? This requires human validation on a sample. Target: 85%+ for production use.
• •Semantic resolution accuracy. What percentage of business terms are correctly mapped to their canonical definitions? Target: 95%+.
• •User satisfaction score. Do users trust and use the system regularly? Track daily active users and repeat usage as proxies for accuracy perception.

Getting Started: From Raw SQL to Semantic Text-to-SQL

The path from a raw text-to-SQL prototype to a production-grade semantic system is straightforward but requires discipline. Start by auditing your most common queries, define semantic models for your top metrics and entities, implement a validation layer that catches ambiguous or violating queries, and measure accuracy continuously.

Data Workers accelerates this journey by auto-discovering semantic relationships from your existing catalog, query logs, and documentation. With 85+ integrations across the modern data stack, it connects to your warehouse, catalog, and quality tools to build a context layer that makes text-to-SQL actually work. Explore the documentation to learn how, or book a demo to see it in action against your own schema.

Ready to make text-to-SQL actually accurate? Book a demo to see how Data Workers' semantic context layer transforms natural language query accuracy on your enterprise schema.