MCP Server for Databricks: AI Agents Meet the Lakehouse

An MCP server for Databricks connects AI agents to the lakehouse platform — Unity Catalog, SQL warehouses, Delta tables, jobs, and notebooks — through the Model Context Protocol. It lets agents query, manage, and orchestrate Databricks resources in natural language without custom API wrappers or hard-coded credentials.

An MCP server for Databricks connects AI agents to the lakehouse platform, enabling natural language access to Unity Catalog, SQL warehouses, Delta tables, jobs, and notebooks. Databricks is the data platform of choice for organizations that need both analytics and machine learning on a unified architecture, and MCP integration makes it accessible to AI agents without custom API wrappers. This guide covers the Databricks-specific MCP server implementation: authentication with Unity Catalog, querying through SQL warehouses, Delta table operations, and job management.

The Databricks ecosystem presents unique MCP integration opportunities. Unity Catalog provides a governance layer with fine-grained permissions that map naturally to MCP tool boundaries. SQL warehouses offer serverless compute for agent-initiated queries. Delta Lake provides ACID transactions and time travel that agents can leverage for data debugging. And the Databricks Jobs API enables agents to trigger and monitor pipeline runs.

Databricks MCP Server Architecture

A Databricks MCP server differs from a generic database MCP server in one important way: it interacts with multiple Databricks services, not just a SQL endpoint. The server needs to connect to Unity Catalog (for metadata), SQL warehouses (for queries), the Jobs API (for pipeline management), and the Workspace API (for notebook access).

Authentication: Unity Catalog and Service Principals

Databricks authentication for MCP servers should use service principals rather than personal access tokens. Service principals provide machine-identity authentication that is independent of individual users, supports fine-grained Unity Catalog permissions, and can be managed through your identity provider.

Create a dedicated service principal for MCP access: configure it in your Databricks account admin console, generate an OAuth client secret, and grant it the minimum required Unity Catalog privileges. For a read-only MCP server, the service principal needs USE CATALOG, USE SCHEMA, and SELECT on the catalogs and schemas you want to expose.

Store the OAuth credentials in your secret manager and inject them into the MCP server at runtime. Never embed Databricks credentials in MCP server code or configuration files. The MCP server should authenticate using the OAuth client credentials flow, obtaining short-lived tokens that are automatically refreshed.

Querying Through SQL Warehouses

Databricks SQL warehouses are the optimal compute layer for MCP-initiated queries. They support serverless scaling (no cold start for pre-warmed warehouses), automatic suspension (no cost when idle), and workload isolation (MCP queries do not compete with other workloads for compute).

When implementing the query tool, use the Databricks SQL Connector for Python (databricks-sql-connector). Submit queries asynchronously using the Statement Execution API for long-running queries, and poll for results. Return results as structured tables with column names and types — AI agents produce better analysis when they receive typed, labeled data.

For cost management, implement query governance in the MCP server. Estimate scan size before execution using query dry-runs where available. Set a per-query cost ceiling. Route queries to different warehouse sizes based on estimated complexity — simple metadata queries go to a small warehouse, analytical queries go to a medium or large warehouse.

Delta Lake Integration: Time Travel and Versioning

Delta Lake's time travel capability is uniquely powerful in an MCP context. AI agents can use time travel to debug data quality issues by comparing current data against historical versions:

Version comparison. Expose a tool that compares table state between two versions or timestamps: 'Show me what changed in the customers table between yesterday and today.' The tool executes two version-specific queries and returns the diff — new rows, modified rows, and deleted rows.

Rollback analysis. Before an agent suggests rolling back a table to a previous version, it can preview the impact: 'If I restore the orders table to version 42, which rows would change?' This preview capability prevents accidental data loss from hasty rollbacks.

Audit trail. The DESCRIBE HISTORY command returns a complete audit log of table changes: who changed what, when, and how (append, overwrite, merge). Expose this as an MCP tool for compliance and debugging.

Unity Catalog: Governance-Aware MCP Tools

Unity Catalog is Databricks' governance layer, and it integrates naturally with MCP security boundaries. Instead of implementing access control in your MCP server, leverage Unity Catalog's built-in permissions:

• •Catalog-level access. The MCP service principal can only see catalogs it has been granted access to. This means agents cannot discover or query data outside their authorized scope — the governance boundary is enforced by Databricks, not by your MCP server code
• •Column-level security. Unity Catalog supports column masking and row filtering. If PII columns are masked for the MCP service principal, agents receive masked data without needing any logic in the MCP server
• •Lineage integration. Unity Catalog tracks lineage automatically for all queries. MCP-initiated queries appear in the lineage graph, making it easy to audit which data was accessed and how it flowed through transformations
• •Tags and classifications. Expose Unity Catalog tags (PII, sensitive, deprecated) in your MCP metadata tools. Agents can use these tags to make informed decisions about which data to query and how to handle results

Job Management: Agent-Triggered Pipelines

Enabling AI agents to trigger Databricks jobs requires careful permission scoping. Create job-specific permissions: the MCP service principal should be able to run specific jobs (not all jobs), view run status, and cancel runs it initiated. Never grant the ability to create or modify job definitions — that remains a human-controlled operation.

The job management tools enable powerful agent workflows: 'The data quality check on the orders table detected a schema drift. Trigger the remediation job to fix the staging models and rerun the affected pipeline.' The agent initiates the job, monitors its progress, and reports the result — all without human intervention for pre-approved remediation patterns.

Production Deployment on Databricks

For production deployment, run the MCP server as a Databricks job or on your existing Kubernetes infrastructure. If running on Databricks, use a serverless job with the MCP server as the entry point. If running externally, ensure network connectivity to the Databricks workspace (VPC peering or private link).

Data Workers provides a pre-built Databricks MCP integration as part of its 15-agent platform. The integration covers Unity Catalog discovery, SQL warehouse querying, Delta Lake time travel, and job management — with built-in caching, rate limiting, and audit logging. For teams already using Data Workers, Databricks connectivity is configured through the platform's unified connection manager. For more on integration patterns, see the Data Workers documentation.

A Databricks MCP server unlocks the full lakehouse for AI agents — from data discovery through Unity Catalog to query execution on SQL warehouses to pipeline management through the Jobs API. The key is leveraging Databricks' built-in governance (Unity Catalog permissions, column masking, lineage tracking) rather than reimplementing access control in your MCP server. Start with read-only catalog and query tools, validate the security model, and expand to job management and Delta Lake operations. Visit the Data Workers blog for more Databricks integration content, or book a demo to see the integration in action.