AI Agents for AWS Glue: Fix Failed Jobs and Data Quality Failures Automatically
What Glue's own AI features do, what the Data Processing MCP server gives coding agents, and how Data Workers fixes Glue failures end to end
AI agents for AWS Glue should do more than explain a stack trace. The useful ones find out why a Glue job failed or wrote bad rows, fix the cause where it started, rerun only the affected partitions, rebuild what reads them and prove the data is right again.
AWS has added AI to Glue this year, mostly in the form of recommendations a person reviews. Data Workers is the agent layer that acts on those signals and owns the fix from the failed run to the verified result.
Key Takeaways
- •Glue Data Quality now recommends rules using generative AI through Amazon Bedrock, and a person reviews, adjusts and saves them.
- •The awslabs Data Processing MCP server lets coding agents inspect Glue jobs, Athena and EMR, read-only unless
--allow-writeis set. - •The costliest Glue failures are the ones that succeed: a job that writes nulls after an upstream rename never trips a retry.
- •Data Workers agents trace Glue failures across DMS, S3 Tables, Redshift and dbt, draft the script fix behind approval, rerun scoped partitions and verify the outcome.
- •Your Glue jobs, rulesets and Data Catalog stay exactly where they are.
What AWS ships for Glue today
| Feature | What it does | Status |
|---|---|---|
| Glue 6.0 | Spark 4.1 and Apache Iceberg v3 | GA |
| Glue Data Quality | DQDL rules, anomaly detection, and AI rule recommendations through Bedrock that you review and save | GA (recommendations Sep 2026) |
| Glue Data Catalog | Technical metadata, crawlers, statistics and Iceberg v3 optimization | GA |
| Data Processing MCP server (awslabs) | Glue, EMR and Athena tools for coding agents; read-only unless --allow-write | Available |
| SageMaker Unified Studio | Visual ETL, notebooks and the SageMaker Data Agent for writing and debugging code | GA |
Glue is a mature, reliable engine, and the new AI features make rule authoring faster. They stop at the point where a person takes over: the recommendation is reviewed, the failed run is investigated, the fix is written and the rerun is scheduled by your team.
The four Glue failures that eat engineering time
1. The job that fails loudly. Out-of-memory errors, missing partitions, permissions on a new bucket. These are visible in CloudWatch and usually quick, but they still need someone to read the log, change the config and rerun.
2. The job that succeeds and writes bad data. A source column is renamed in Aurora, DMS carries it into S3, and the Glue mapping still reads the old name. Every new row gets a null. The run is green. This is the most expensive kind, because the damage spreads before anyone notices.
3. The data quality failure with no owner. A Glue Data Quality ruleset fails at dawn and the result is published. Who looks at it, and what do they do? Too often the answer is a ticket that waits.
4. The upgrade and drift work. Moving jobs to a new Glue version, keeping mappings in step with sources and retiring jobs nobody uses. Necessary, never urgent, always behind.
How Data Workers agents handle Glue
Data Workers connects to the Glue Data Catalog, Glue job runs, Lake Formation and DMS over the AWS API or MCP servers, and to your dbt project, orchestrator and BI. The context graph knows which Glue job writes which table and who reads it.
- •Diagnose. The Incident Debugging agent reads the failed run or the failed ruleset and traces the cause: a config limit, a stale mapping, an upstream schema change in the DMS task.
- •Fix. The Pipeline Building agent drafts the Glue script or job configuration change as a diff, with the blast radius attached, for the job owner to approve.
- •Prevent. The Schema Evolution agent catches the upstream rename in the source team's pull request and maps its impact through the Glue job, the target table and everything downstream, before the next run.
- •Rerun and rebuild. After approval, the affected partitions are rerun and the downstream dbt models rebuilt through your existing DAG.
- •Verify. Data Workers reruns the Glue Data Quality ruleset, diffs row counts and values, checks the dbt tests and records a tamper-evident receipt.
- •Strengthen the rules. The Quality Monitoring agent turns recurring incidents into new checks, so your rulesets get better from real failures.
Glue Data Quality and Data Workers together
Keep your Glue Data Quality rulesets. They are a strong signal. Data Workers is what happens after the signal.
| Step | Glue Data Quality | Data Workers |
|---|---|---|
| Write rules | DQDL rules, AI recommendations you review and save | Writes and repairs checks and dbt tests, and adds rules from past incidents |
| Detect | Evaluates rules and anomalies on each run | Takes the failed result as a signal, alongside freshness and volume signals |
| Diagnose | Reports which rule failed | Traces why, across DMS, Glue, S3 Tables, Redshift and dbt |
| Fix | Left to your team | Drafts the fix where the problem started, behind approval |
| Verify | The ruleset passes on its next run | Reruns the ruleset, diffs values and checks downstream tests before closing |
Getting started with Glue
Give Data Workers a read-only IAM role for the Glue Data Catalog and job runs, connect your dbt project and orchestrator, and put failed Glue loads in propose mode. Each failure arrives with its root cause, the proposed fix or rerun and its blast radius. When your team approves those proposals as-is for a few weeks, move the domain to reversible actions. If you also use SageMaker Catalog alongside Glue, see You're on AWS Glue and SageMaker Catalog. For pipeline recovery beyond Glue, read autonomous data pipelines on AWS.
Frequently Asked Questions
Can AI fix failed AWS Glue jobs automatically?
Yes, within the autonomy you allow. Data Workers diagnoses the failed or silently wrong run, drafts the fix to the script or configuration, and after approval reruns the affected partitions and verifies the result. Domains you move to the act-reversibly level can apply undoable fixes without waiting.
Does Glue Data Quality fix data issues on its own?
Glue Data Quality evaluates rules, detects anomalies and recommends rules with generative AI. Resolving a failure is left to your team. Data Workers takes the failed result and runs the fix and the verification.
How does Data Workers catch Glue jobs that succeed but write bad data?
It watches freshness, volume and quality signals on the tables Glue writes, and the Schema Evolution agent flags upstream schema changes before the job runs. Because the context graph links the source, the job and the target table, a null spike points straight at the stale mapping.
Does Data Workers work with the Data Processing MCP server?
Yes. Your engineers can keep the awslabs Data Processing MCP server in their coding agent for direct Glue access, and add Data Workers for context, approvals and cross-service fixes.
Sources
- •AWS Glue documentation history
- •Glue Data Quality rule recommendations (Sep 22, 2026)
- •Glue Data Catalog Iceberg V3 support (Oct 1, 2026)
- •awslabs MCP servers