Introduction
Data validation code in Python has a way of decaying: business rules get buried in nested if statements, validation logic tangles with error handling, and adding a new check means spelunking through procedural functions to find where it belongs. Mature frameworks exist for this, but there is real value in building something minimal by hand — a small domain-specific language (DSL) for data validation, a vocabulary of functions and classes that express rules the way data professionals actually think about them. The complete code for this tutorial is available on GitHub.
Why build a DSL?
Consider validating customer data the usual way:
def validate_customers(df):
errors = ()
if df('customer_id').duplicated().any():
errors.append("Duplicate IDs")
if (df('age') < 0).any():
errors.append("Negative ages")
if not df('email').str.contains('@').all():
errors.append("Invalid emails")
return errorsThis hardcodes the validation logic, mixes business rules with error handling, and becomes unmaintainable as rules grow. A DSL separates those concerns and creates reusable validation components — rules that read like business requirements:
# Traditional approach
if df('age').min() < 0 or df('age').max() > 120:
raise ValueError("Invalid ages found")
# DSL approach
validator.add_rule(Rule("Valid ages", between('age', 0, 120), "Ages must be 0-120"))The DSL approach separates what is being validated (business rules) from how violations are handled (error reporting). That makes the logic testable, reusable, and readable even by non-programmers.
Creating a sample dataset
A realistic e-commerce customer dataset with deliberate quality problems makes a good test bed:
import pandas as pd
customers = pd.DataFrame({
'customer_id': (101, 102, 103, 103, 105),
'email': ('john@gmail.com', 'invalid-email', '', 'sarah@yahoo.com', 'mike@domain.co'),
'age': (25, -5, 35, 200, 28),
'total_spent': (250.50, 1200.00, 0.00, -50.00, 899.99),
'join_date': ('2023-01-15', '2023-13-45', '2023-02-20', '2023-02-20', '')
}) # Note: 2023-13-45 is an intentionally malformed date.The data contains duplicate customer IDs, invalid email formats, impossible ages, negative spend amounts and malformed dates — exactly the issues a validator should catch.
Defining the Rule class
class Rule:
def __init__(self, name, condition, error_msg):
self.name = name
self.condition = condition
self.error_msg = error_msg
def check(self, df):
# The condition function returns True for VALID rows.
# We use ~ (bitwise NOT) to select the rows that VIOLATE the condition.
violations = df(~self.condition(df))
if not violations.empty:
return {
'rule': self.name,
'message': self.error_msg,
'violations': len(violations),
'sample_rows': violations.head(3).index.tolist()
}
return NoneThe design keeps validation logic and error reporting apart: the condition function focuses solely on the business rule, while the Rule class handles violation details consistently.
Managing collections of rules
class DataValidator:
def __init__(self):
self.rules = ()
def add_rule(self, rule):
self.rules.append(rule)
return self # Enables method chaining
def validate(self, df):
results = ()
for rule in self.rules:
violation = rule.check(df)
if violation:
results.append(violation)
return resultsThe add_rule method returns self to enable method chaining, and validate executes all rules independently while collecting violation reports — so one failed rule never prevents the others from running.
Creating readable conditions
A rule’s condition can be any function that takes a DataFrame and returns a boolean Series. Plain lambdas work but read poorly, so helper functions provide a readable validation vocabulary:
def not_null(column):
return lambda df: df(column).notna()
def unique_values(column):
return lambda df: ~df.duplicated(subset=(column), keep=False)
def between(column, min_val, max_val):
return lambda df: df(column).between(min_val, max_val)Each helper returns a lambda built on pandas boolean operations: not_null uses notna(), unique_values uses duplicated(..., keep=False) to flag every duplicate occurrence, and between wraps pandas’ range check. Pattern matching uses regular expressions directly:
import re
def matches_pattern(column, pattern):
return lambda df: df(column).str.match(pattern, na=False)The na=False parameter ensures missing values are treated as validation failures rather than matches — usually the right behaviour for required fields.
Building the validator
validator = DataValidator()
validator.add_rule(Rule(
"Unique customer IDs",
unique_values('customer_id'),
"Customer IDs must be unique across all records"
))
validator.add_rule(Rule(
"Valid email format",
matches_pattern('email', r'^(^@s)+@(^@s)+.(^@s)+$'),
"Email addresses must contain @ symbol and domain"
))
validator.add_rule(Rule(
"Reasonable customer age",
between('age', 13, 120),
"Customer age must be between 13 and 120 years"
))
validator.add_rule(Rule(
"Non-negative spending",
lambda df: df('total_spent') >= 0,
"Total spending amount cannot be negative"
))The email pattern requires at least one character before and after the @ sign plus a domain extension; the between helper sets sensible age limits; and the final rule uses an inline lambda to require non-negative total_spent values. Each rule reads almost like a business requirement. The validator then runs against any DataFrame with matching columns:
issues = validator.validate(customers)
for issue in issues:
print(f"❌ Rule: {issue('rule')}")
print(f"Problem: {issue('message')}")
print(f"Affected rows: {issue('sample_rows')}")
print()The output pinpoints specific problems and their locations, which makes debugging simple. For the sample data:
Validation Results:
❌ Rule: Unique customer IDs
Problem: Customer IDs must be unique across all records
Violations: 2
Affected rows: (2, 3)
❌ Rule: Valid email format
Problem: Email addresses must contain @ symbol and domain
Violations: 3
Affected rows: (1, 2, 4)
❌ Rule: Reasonable customer age
Problem: Customer age must be between 13 and 120 years
Violations: 2
Affected rows: (1, 3)
❌ Rule: Non-negative spending
Problem: Total spending amount cannot be negative
Violations: 1
Affected rows: (3)Adding cross-column validation
Real business rules often span columns. Custom lambdas handle the logic:
def high_spender_email_required(df):
high_spenders = df('total_spent') > 500
has_valid_email = df('email').str.contains('@', na=False)
# Passes if: (Not a high spender) OR (Has a valid email)
return ~high_spenders | has_valid_email
validator.add_rule(Rule(
"High Spenders Need Valid Email",
high_spender_email_required,
"Customers spending over $500 must have valid email addresses"
))The expression ~high_spenders | has_valid_email reads as “either not a high spender, or has a valid email” — high-spending customers must have valid contact details, while low spenders pass regardless.
Handling date validation
def valid_date_format(column, date_format="%Y-%m-%d"):
def check_dates(df):
# pd.to_datetime with errors="coerce" turns invalid dates into NaT (Not a Time)
parsed_dates = pd.to_datetime(df(column), format=date_format, errors="coerce")
# A row is valid if the original value is not null AND the parsed date is not NaT
return df(column).notna() & parsed_dates.notna()
return check_dates
validator.add_rule(Rule(
"Valid Join Dates",
valid_date_format('join_date'),
"Join dates must follow YYYY-MM-DD format"
))Validation passes only when the original value is non-null and the parsed date is valid (not NaT). Using errors="coerce" in pd.to_datetime removes the need for try-except blocks by converting malformed strings to NaT automatically.
Validating inside pipelines
def validate_dataframe(validator):
def decorator(func):
def wrapper(df, *args, **kwargs):
issues = validator.validate(df)
if issues:
error_details = (f"{issue('rule')}: {issue('violations')} violations" for issue in issues)
raise ValueError(f"Data validation failed: {'; '.join(error_details)}")
return func(df, *args, **kwargs)
return wrapper
return decorator
# Note: 'customer_validator' needs to be defined globally or passed in a real implementation
# Assuming 'customer_validator' is the instance we built earlier
# @validate_dataframe(customer_validator)
def process_customer_data(df):
return df.groupby('age').agg({'total_spent': 'sum'})This decorator validates data before processing begins, stopping corrupted data from spreading through a pipeline and producing descriptive errors that name the failed rules. (As noted in the snippet, customer_validator must be accessible to the decorator.)
Extending the vocabulary
# Statistical outlier detection
def within_standard_deviations(column, std_devs=3):
# Valid if absolute difference from mean is within N standard deviations
return lambda df: abs(df(column) - df(column).mean()) <= std_devs * df(column).std()
# Referential integrity across datasets
def foreign_key_exists(column, reference_df, reference_column):
# Valid if value in column is present in the reference_column of the reference_df
return lambda df: df(column).isin(reference_df(reference_column))
# Custom business logic
def profit_margin_reasonable(df):
# Ensures 0 <= margin <= 1
margin = (df('revenue') - df('cost')) / df('revenue')
return (margin >= 0) & (margin <= 1)New validation rules slot in as composable functions returning boolean Series. A complete usage example, assuming the helpers live in a module called data_quality_dsl:
import pandas as pd
from data_quality_dsl import DataValidator, Rule, unique_values, between, matches_pattern
# Sample data
df = pd.DataFrame({
'user_id': (1, 2, 2, 3),
'email': ('user@test.com', 'invalid', 'user@real.com', ''),
'age': (25, -5, 30, 150)
})
# Build validator
validator = DataValidator()
validator.add_rule(Rule("Unique users", unique_values('user_id'), "User IDs must be unique"))
validator.add_rule(Rule("Valid emails", matches_pattern('email', r'^(^@)+@(^@)+.(^@)+$'), "Invalid email format"))
validator.add_rule(Rule("Reasonable ages", between('age', 0, 120), "Age must be 0-120"))
# Run validation
issues = validator.validate(df)
for issue in issues:
print(f"❌ {issue('rule')}: {issue('violations')} violations")Conclusion
This DSL works because it matches how data professionals think about validation: rules express business logic as readable requirements while pandas provides the performance underneath. Separation of concerns keeps the logic testable and maintainable, there are no dependencies beyond pandas, and no learning curve for anyone who already knows pandas operations. It is a starting point, deliberately simple, that can grow into something fancier.
Limitations and what to watch
A hand-rolled DSL trades power for clarity. It has no schema inference, no data-drift detection, no HTML reports and no integration ecosystem — capabilities that dedicated libraries such as Great Expectations and Pandera provide out of the box, at the cost of heavier dependencies and steeper learning curves. Performance on very large DataFrames depends entirely on how the condition lambdas are written, and regex-based email validation is famously approximate. For teams already automating data workflows with AI agents, validation gates like these pair naturally with the governance patterns covered in AI agents need guardrails.
