Agentic Data Engineering on AWS: How AI Agents Run Glue, Redshift and S3 Tables
What agentic data engineering means on AWS, what AWS ships for it today, and how Data Workers runs the whole loop from alert to verified fix
Agentic data engineering on AWS means AI agents that do the operational work of a data team across AWS services: they read a failed Glue run or a broken Redshift number, trace the cause to where it started, draft or apply the fix inside the permissions you set, verify the result downstream and leave a record.
Data Workers is the agentic data platform built for exactly that job. It runs on top of Glue, Redshift, S3 Tables, Lake Formation and the tools next to them, with 20+ specialist agents, one context graph and an approval flow you control per domain.
Key Takeaways
- •Agentic data engineering is about owning outcomes, not writing snippets. An agent that writes a query is an assistant. An agent that closes an incident across services is the agentic part.
- •AWS ships strong building blocks for it: the AWS MCP Server, Bedrock AgentCore, the SageMaker Data Agent and AI features inside Glue, Redshift and SageMaker Catalog.
- •AWS's own assistants help a person inside one service, and the person reviews and runs the result. That is a sensible design for AWS, and it leaves the cross-service loop to your team.
- •Data Workers runs that loop end to end: detect, diagnose, fix, review, verify and remember, from Aurora and DMS through Glue and S3 Tables to Redshift, dbt and the dashboard.
- •Your data stays in your AWS accounts, IAM and Lake Formation stay the lock, and every Data Workers call shows up in CloudTrail like any other principal's.
What agentic data engineering on AWS actually means
Most AWS data teams run the same shape of estate. Operational databases replicate through AWS DMS or zero-ETL into S3. Glue jobs turn the landing zone into Apache Iceberg tables, more of them in S3 Tables, registered in the Glue Data Catalog and governed by Lake Formation. Redshift serves the warehouse, dbt builds the models, and Airflow on MWAA or Step Functions runs the schedule. Athena and EMR take the ad hoc and heavy work, and Amazon Quick dashboards sit on top.
Every one of those services works well on its own. The engineering hours go into the seams between them: a column renamed upstream that turns into nulls in a Glue job, a Lake Formation request that sits in a queue for two days, a Redshift cost spike nobody can tie to a model. Agentic data engineering moves that seam work to agents. The bar is simple: the agent should finish the job, not hand back a suggestion.
We break the work into five jobs, and a real agentic setup covers all of them:
- •Context. Know how the estate fits together: lineage from source to dashboard, owners, definitions and recent incidents.
- •Detection and diagnosis. Turn a failed check or an alarm into a named root cause, in whichever system it lives in.
- •Change. Write the fix where the problem started, whether that is a Glue script, a dbt model, a grant or a schema.
- •Control. Route each change to the right approver, at the autonomy level set for that domain, with the blast radius in front of them.
- •Proof. Verify the downstream number is right again and keep a receipt an auditor can read.
What AWS ships for agentic data engineering today
AWS has moved fast on agent building blocks this year. Here is what its own documentation describes for data teams.
| Area | What AWS ships | What it does for a data team |
|---|---|---|
| Agent access to AWS | AWS MCP Server (GA May 2026) and Agent Toolkit | Lets a coding agent call any AWS API through one tool, gated by IAM and logged in CloudTrail |
| Open-source MCP servers | awslabs Redshift, Data Processing (Glue, EMR, Athena) and S3 Tables servers | Read-only by default; Data Processing and S3 Tables allow writes only with --allow-write |
| Agent runtime | Bedrock AgentCore: Runtime, Gateway, Memory, Identity, Policy, Evaluations | A kit for building and governing your own agents |
| Notebook and SQL help | SageMaker Data Agent in SageMaker Unified Studio | Generates code, SQL and plans, and diagnoses errors in notebooks |
| Warehouse help | Amazon Q generative SQL in Redshift query editor v2 | Writes SQL from a prompt for a person to review and run |
| Quality | Glue Data Quality with AI rule recommendations | Recommends DQDL rules that a person reviews and saves |
| Spark | Spark troubleshooting and upgrade agents for EMR | Finds a root cause and recommends code changes |
| Reference design | Agentic Data Operations Platform (ADOP), an aws-samples reference architecture (Aug 2026) | Shows sub-agents generating ETL, quality checks and DAGs with human-in-the-loop approval |
That is a rich toolbox. Notice the pattern, though. Each AI feature helps one person inside one service, and AWS's agent offerings are kits you assemble. ADOP is a reference architecture you clone and run yourself. AWS sells building blocks, and keeping its assistants on the "recommend, a person runs it" side of the line is the right call for a cloud provider that serves every kind of customer.
Where the AWS-native path stops
The gap shows up the first time an incident crosses a service boundary. Take a rename in Aurora that DMS carries into S3. The Glue job still maps the old column, so every new row lands with a null. The job succeeds, a dbt model on Redshift files the night's revenue under "unknown", and a Glue Data Quality rule fails at dawn.
AWS raised a clear signal. What happens next is your team's work: open CloudWatch, the Glue console, the dbt project and the Airflow UI, find the stale mapping, fix the script, rerun the job for the affected partitions, rebuild the models and check the dashboard. No AWS assistant owns that sequence, because it spans four services and a repo AWS doesn't host.
Building it yourself on AgentCore is possible. You would need a context graph across AWS and non-AWS systems, specialist agents for schemas, pipelines, access and cost, a verification loop, per-domain autonomy, an approval inbox and a receipt format. That is a product. It is the product Data Workers ships.
How Data Workers runs agentic data engineering on AWS
Data Workers is built around the loop AWS leaves open. The Autonomous Data Conductor owns the outcome: it takes a Glue Data Quality failure, a CloudWatch alarm or a Security Hub finding as one input, directs the right specialist agents and does not close the incident until the downstream checks pass.
- •One context graph. Data Context Wizard joins the Glue Data Catalog, Lake Formation, Redshift and DMS (over the AWS API or MCP servers) with your dbt project, Airflow and BI, plus Snowflake or BigQuery if you run them.
- •Specialist agents. The Data Agents Swarm brings 20+ agents, including the Schema Evolution agent, the Pipeline Building agent, the Incident Debugging agent, the Data Access Governance agent and the Cost Savings and Cleanup agent. Every agent is an MCP server.
- •Your coding agent as the way in. Engineers keep working in Claude Code, Codex, Cursor or Kiro, and call Data Workers from there.
- •Graded autonomy. Each domain runs at its own level: observe, propose, act reversibly or autonomous. Failed-load fixes can run at a higher level than schema changes.
- •Receipts. Every change records why it was made, what it touched downstream, who approved it and how to undo it. CloudTrail tells you what was called; the receipt tells you why.
In the Aurora rename above, the Schema Evolution agent flags the contract change from the source team's pull request before the Glue job runs. If it slips through, the Conductor ties the nulls to the stale mapping, the Pipeline agent drafts the script fix with its blast radius attached, and after approval the rerun, the dbt rebuild and the verification happen as one tracked piece of work.
Agentic data engineering on AWS by service
| AWS service | Typical toil | What Data Workers does |
|---|---|---|
| AWS Glue | Failed or silently wrong jobs, stale mappings, rule upkeep | Diagnoses the run, drafts the script fix, reruns behind approval and verifies. See AI agents for AWS Glue |
| Amazon Redshift | Broken models, slow queries, cost spikes | Traces wrong numbers to their source and drafts cost fixes for their owners. See AI agents for Amazon Redshift |
| S3 Tables and Iceberg | Schema drift, table upkeep, partition backfills | Catches schema changes before they land and runs scoped backfills. See AI agents for S3 Tables and Iceberg |
| Lake Formation | A queue of access requests | Proposes least-privilege, time-boxed grants for Lake Formation to apply. See Lake Formation access request automation |
| MWAA and Step Functions | Reruns and backfills after failures | Ties the failure to its cause and runs the recovery as one approved change. See autonomous data pipelines on AWS |
How to start
Start where the failures repeat. Create read-only IAM roles, connect Data Workers to your dbt project and orchestrator, give your team's client the AWS MCP Server for the Glue Data Catalog, Lake Formation, DMS and Redshift, and put one domain in propose mode. Failed Glue loads and Data Quality failures make a good first domain because they end the same way every time. When your team approves the proposals as-is for a few weeks, move that domain up to reversible actions. The step-by-step climb is in From AWS to an autonomous data platform, and the full service-by-service comparison is in Data Workers on AWS.
Frequently Asked Questions
What is agentic data engineering on AWS?
It is the use of AI agents to run data engineering work across AWS services end to end: detecting a problem, finding its cause, making the fix where it started, verifying the result and recording what happened. It goes beyond assistants that write a query or a notebook cell for a person to run.
Does AWS have an agentic data engineering product?
AWS ships building blocks and assistants: the AWS MCP Server, Bedrock AgentCore, the SageMaker Data Agent, Amazon Q generative SQL and AI features in Glue Data Quality and SageMaker Catalog. It also publishes ADOP, a reference architecture you deploy yourself. A packaged agent that owns data operations across services is the job Data Workers does.
Does Data Workers replace Glue, Redshift or Lake Formation?
No. Every AWS service keeps its job, and IAM and Lake Formation stay the enforcement point. Data Workers runs the operations across them.
Does our data leave our AWS accounts?
No. Your data stays in your accounts. Data Workers stores metadata and PII-redacted facts about your data, such as lineage, owners and incident history, and the hosted Conductor sees workflow metadata only.
Can Data Workers use Amazon Bedrock models?
Yes. Data Workers supports Bedrock as a model provider, so you can run it on models you already use there, with no markup on model spend.
Sources
- •AWS MCP Server general availability (May 6, 2026)
- •awslabs MCP servers, including the Redshift, Data Processing and S3 Tables servers
- •What is Amazon Bedrock AgentCore
- •SageMaker Data Agent
- •Amazon Q generative SQL in Redshift query editor v2
- •Glue Data Quality rule recommendations (Sep 22, 2026)
- •Agentic Data Operations Platform (ADOP) (Aug 21, 2026)