Data quality problems are everywhere: missing values where there should be none, dates in the wrong format, duplicate records that slip through, outliers that distort analysis, and text fields with inconsistent spelling. Left unchecked, these issues break analytics and pipelines and can lead to wrong business decisions. Manual validation is slow and error-prone, because the same problems have to be investigated repeatedly across datasets and subtle issues are easy to miss. The five checks below cover the most common data-quality problems and can each be automated with a small Python script built on the pandas library. The code samples are concise, illustrative starting points rather than production-ready tools.
1. Analysing missing data
The problem: a dataset that should be complete instead contains empty cells, null values, empty strings, and placeholder text such as “N/A” or “Unknown”. Some columns are mostly empty, others have only a few gaps, and the first step is simply to measure the extent of the problem.
What it checks: it scans every column for missing data in all its forms, treats common placeholder strings as missing, and calculates a completeness score per column so the worst offenders are easy to spot.
import pandas as pd
import numpy as np
def missing_data_report(df, placeholders=("N/A", "Unknown", "", "null")):
df = df.replace(list(placeholders), np.nan)
report = pd.DataFrame({
"missing": df.isna().sum(),
"missing_pct": (df.isna().mean() * 100).round(2),
})
report["completeness_pct"] = (100 - report["missing_pct"]).round(2)
return report.sort_values("missing_pct", ascending=False)2. Validating data types
The problem: columns that should hold numeric IDs contain stray text, date fields mix dates with random strings, and email columns include values that are not valid addresses. These mismatches cause scripts to crash or calculations to go wrong.
What it checks: given a simple schema describing the expected type for each column, it attempts to coerce each column to that type and flags the values that fail, along with a violation count per column.
def validate_types(df, schema):
# schema example: {"age": "int", "signup": "datetime", "price": "float"}
issues = {}
for col, expected in schema.items():
if col not in df.columns:
issues[col] = "column missing"
continue
if expected in ("int", "float"):
coerced = pd.to_numeric(df[col], errors="coerce")
elif expected == "datetime":
coerced = pd.to_datetime(df[col], errors="coerce")
else:
continue
bad = coerced.isna() & df[col].notna()
if bad.any():
issues[col] = f"{int(bad.sum())} value(s) not {expected}"
return issues3. Detecting duplicate records
The problem: the same entity appears more than once, either as an exact duplicate or as a near-duplicate with small differences in spelling or spacing. Both inflate counts and skew aggregates.
What it checks: it surfaces exact duplicates directly, and for fuzzy matches it compares text fields using a string-similarity measure such as Levenshtein distance to catch records that are nearly, but not exactly, identical.
def find_duplicates(df, subset=None):
# subset lets the check focus on the columns that define identity
mask = df.duplicated(subset=subset, keep=False)
return df[mask].sort_values(by=subset or list(df.columns))4. Detecting outliers
The problem: a value is negative when it should be positive, or several orders of magnitude larger than the rest. Outliers skew statistics, break models, and are hard to find by eye in large datasets.
What it checks: it flags statistical outliers using methods such as the interquartile range (IQR) or z-score, optionally combined with domain rules for values that are simply impossible. The IQR method below is a robust default that does not assume a normal distribution.
def iqr_outliers(df, column):
q1, q3 = df[column].quantile([0.25, 0.75])
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
mask = (df[column] < lower) | (df[column] > upper)
return df[mask]5. Checking cross-field consistency
The problem: individual fields look valid, but their relationships do not hold. An end date precedes a start date, or a stored total does not match quantity multiplied by unit price.
What it checks: it encodes business rules that span multiple columns, recomputes derived values, and reports the rows that violate each rule. The example rules below should be replaced with the ones that apply to a given dataset.
def check_consistency(df):
bad_dates = df[df["end_date"] < df["start_date"]]
recomputed = (df["quantity"] * df["unit_price"]).round(2)
bad_totals = df[recomputed != df["total"].round(2)]
return {"date_order": bad_dates, "totals": bad_totals}Caveats and good practice
Automated checks find suspicious values; they do not decide what to do with them. A flagged outlier may be a genuine extreme rather than an error, a “missing” field may be legitimately blank, and a type violation may reveal a column that simply needs a clearer schema. The right response is to review what each check surfaces, encode domain knowledge into the rules, and run the checks on a sample before applying them to a full pipeline. For broader validation, these scripts pair well with dedicated tooling; see the related round-up of Python data-validation libraries and the wider discipline of building reliable data pipelines.
Wrapping up
Catching data-quality issues early, before they reach analyses or production systems, is far cheaper than fixing them afterward. A practical way to start is to automate the check that addresses the most pressing problem, configure its rules for the specific dataset, validate the setup on a sample, and then integrate it into the pipeline so that issues are flagged automatically. Clean data is the foundation of reliable analytics, and systematic validation means less time spent firefighting later.
