Enhancing AWS Glue Data Quality Pipelines with Context-Aware Rule Discovery for Great Expectations

How DataBuck SDK helps data teams discover business-relevant validation rules and run them within existing AWS Glue pipelines.

Author : Sri Harsha Tanamala

Data quality checks are an essential part of any reliable data pipeline. AWS Glue and Great Expectations (GX) give data engineers the tools to validate datasets, detect errors, and prevent unreliable data from reaching downstream systems.

But there’s a challenge that validation frameworks alone don’t fully solve: How do you know which rules to write in the first place?

A dataset might pass standard checks for null values, duplicates, and data types while still violating important business requirements. A banking transaction, for example, could contain valid values in every column but have a closing balance that doesn’t match the transaction amount and fees.

Finding these issues requires more than profiling. It requires an understanding of how the data is supposed to behave.

Discovering Rules Instead of Writing Them All Manually

The DataBuck SDK from FirstEigen helps address this gap by discovering data quality rules using dataset characteristics and business context.

Rather than relying only on statistical patterns, DataBuck can use domain information and policy documents to propose validation rules, including checks that involve relationships between multiple columns.

These discovered rules can then be reviewed, adjusted, and converted into Great Expectations validations.

This allows teams to retain their existing GX workflows while reducing the effort involved in identifying and implementing relevant checks.

How It Works with AWS Glue

The integration follows a straightforward workflow:

Load the data: AWS Glue reads the dataset from Amazon S3 into a Spark DataFrame.

Discover rules: DataBuck SDK analyzes the data alongside relevant business context and policy documents.

Review the rules: Engineers can inspect, modify, or remove the proposed rules before execution.

Validate with GX: The selected rules are converted into Great Expectations validations and executed within the Glue job.

Review results: Validation reports are saved to Amazon S3, providing visibility into failed checks and quality-gate decisions.

The process can run within a single AWS Glue job, with an optional review stage between discovery and validation.

Importantly, DataBuck complements existing AWS Glue Data Quality capabilities rather than replacing them. The approach is particularly useful for teams already using Great Expectations that want to incorporate business-specific checks without manually defining every rule.

What Did the Test Show?
In the example documented on AWS Builder Center, the integration was tested using a banking transactions dataset containing 10,000 records.

DataBuck discovered 96 validation rules, including column-level checks, cross-column business rules, and a dataset-level uniqueness check.

The results were:

96 rules discovered from the dataset and its business context.
74 rules passed validation.
22 rules failed, identifying data quality issues.
6 rules breached the quality gate, indicating conditions that could prevent unreliable data from moving downstream.
Some of the discovered checks evaluated whether transaction balances were consistent with transaction amounts and fees, whether sanctions matches aligned with customer risk ratings, and whether transaction identifiers were unique.

These are examples of checks where business meaning matters as much as the values themselves.

The results demonstrate the integration’s ability to discover and execute different types of rules. They do not, by themselves, establish that every discovered rule is appropriate for production. Human review remains important.

Making Data Quality More Relevant to the Business

Data quality isn’t simply about running more checks. It’s about running checks that reflect how the business operates.

Great Expectations provides a flexible validation framework. AWS Glue provides the processing environment. DataBuck SDK adds context-aware rule discovery to help teams identify what should be validated.

Together, they offer a practical approach to moving from manually maintained rules toward a more automated, business-aware validation process.

For teams already working with AWS Glue and Great Expectations, this integration can be introduced without replacing their existing data quality framework.

Explore the Full Implementation

The complete technical walkthrough, including AWS Glue configuration, DataBuck SDK integration, Great Expectations conversion, sample results, and source code, is available on AWS Builder Center.

Read the full article on AWS Builder Center

To learn more about context-aware data quality, automated rule discovery, and the DataBuck SDK, visit FirstEigen.

This article summarizes a technical implementation originally published on AWS Builder Center by Sri Harsha Tanamala, Data Engineer Consultant at FirstEigen. FirstEigen develops DataBuck.