Introduction
Many data scientists have spent hours debugging a model, only to discover the problem was not the algorithm but a single erroneous null value buried tens of thousands of rows deep. Kaggle competitions and course exercises create the impression that data arrives as clean, well-labeled CSV files with no class-imbalance problems. Production data rarely looks like that. This article walks through a real-life data project to explore four practical habits for handling messy, real-world datasets — and why practicing on imperfect data is the fastest way to build genuinely useful skills.
The NoBroker data project: a practical test of real-world chaos
The project uses data from NoBroker, an Indian property-technology company that connects owners and tenants in a broker-free marketplace.
The goal is to predict how many interactions a property listing will receive within a given time frame. The project ships with three datasets: property_data_set.csv (property details such as type, location, amenities, size and rent), property_photos.tsv (listing photos), and property_interactions.csv (interaction timestamps for each property).
Clean interview data vs. real production data: a reality check
Interview datasets are polished, balanced and boring. Actual production data is another matter: missing values, duplicate rows, inconsistent formats and silent errors that wait until Friday afternoon to break a pipeline. The NoBroker property dataset — 28,888 properties across three tables — looks fine at first glance. Dig deeper and it reveals 11,022 missing photo URLs, corrupted JSON strings with rogue backslashes, and more. That gap is the line between clean and messy: clean data teaches model-building, while production data teaches survival.
Four exercises follow, one for each of the most common failure modes.
Practice #1: Handling missing data
Missing data is not just an annoyance; it is a decision point. Delete the row? Fill it with the mean? Flag it as unknown? The right answer depends on why the data is missing and how much loss is tolerable.
The NoBroker dataset had three kinds of missingness. The photo_urls column was missing 11,022 values out of 28,888 rows — 38% of the dataset. The following code shows the check and its output.
Deleting those rows would destroy valuable property records. Instead, the missing photos were treated as a count of zero, and the pipeline moved on.
def correction(x):
if x is np.nan or x == 'NaN':
return 0 # Missing photos = 0 photos
else:
return len(json.loads(x.replace('\', '').replace('{title','{"title')))
pics('photo_count') = pics('photo_urls').apply(correction)For numeric columns such as total_floor (23 missing) and categorical columns such as building_type (38 missing), the strategy was imputation: numeric gaps filled with the mean, categorical gaps with the mode.
for col in x_remain_withNull.columns:
x_remain(col) = x_remain_withNull(col).fillna(x_remain_withNull(col).mean())
for col in x_cat_withNull.columns:
x_cat(col) = x_cat_withNull(col).fillna(x_cat_withNull(col).mode()(0))The first rule: never delete without asking why. Missing values often have a pattern — here, the absent photo URLs were not random.
Practice #2: Detecting outliers
An outlier is not always an error, but it is always suspicious. An 800-year-old property with 21 bathrooms and 40,000 square feet is either a dream venue or a data-entry mistake. The NoBroker dataset was full of such red flags: box plots revealed property ages above 100, sizes above 10,000 square feet, and deposit amounts above 3.5 million. A few were legitimate luxury properties; most were entry errors.
df_num.plot(kind='box', subplots=True, figsize=(22,10))
plt.show()Here is the output.
The fix was interquartile-range (IQR) based outlier removal — a simple statistical rule that flags values far outside the typical spread. A helper function removes the flagged rows:
def remove_outlier(df_in, col_name):
q1 = df_in(col_name).quantile(0.25)
q3 = df_in(col_name).quantile(0.75)
iqr = q3 - q1
fence_low = q1 - 2 * iqr
fence_high = q3 + 2 * iqr
df_out = df_in.loc((df_in(col_name) <= fence_high) & (df_in(col_name) >= fence_low))
return df_out # Note: Multiplier changed from 1.5 to 2 to match implementation.The function is then applied to the numeric columns.
df = dataset.copy()
for col in df_num.columns:
if col in ('gym', 'lift', 'swimming_pool', 'request_day_within_3d', 'request_day_within_7d'):
continue # Skip binary and target columns
df = remove_outlier(df, col)
print(f"Before: {dataset.shape(0)} rows")
print(f"After: {df.shape(0)} rows")
print(f"Removed: {dataset.shape(0) - df.shape(0)} rows ({((dataset.shape(0) - df.shape(0)) / dataset.shape(0) * 100):.1f}% reduction)")Here is the output.
After outlier removal, the dataset shrank from 17,386 rows to 15,170 — a loss of about 12.7% in exchange for a far better-behaved training set. For the target variable request_day_within_3d, capping was used instead of removal: values above 10 were clipped to 10 so extreme cases could not distort predictions. The code below compares results before and after.
def capping_for_3days(x):
num = 10
return num if x > num else x
df('request_day_within_3d_capping') = df('request_day_within_3d').apply(capping_for_3days)
before_count = (df('request_day_within_3d') > 10).sum()
after_count = (df('request_day_within_3d_capping') > 10).sum()
total_rows = len(df)
change_count = before_count - after_count
percent_change = (change_count / total_rows) * 100
print(f"Before capping (>10): {before_count}")
print(f"After capping (>10): {after_count}")
print(f"Reduced by: {change_count} ({percent_change:.2f}% of total rows affected)")The outcome:
A cleaner distribution, better model performance and fewer debugging sessions.
Practice #3: Dealing with duplicates and inconsistencies
Duplicates are easy — df.drop_duplicates() handles a repeated row. Inconsistencies are harder: a JSON string mangled by three different systems requires detective work. The photo_urls column should have contained valid JSON arrays; instead it held malformed strings with missing quotes, escaped backslashes and random trailing characters.
text_before = pics('photo_urls')(0)
print('Before Correction: nn', text_before)A first pass at repair:
The full fix required multiple string replacements to correct the formatting before parsing.
text_after = text_before.replace('\', '').replace('{title', '{"title').replace(')"', ')').replace('),"', ')","')
parsed_json = json.loads(text_after)Here is the output.
After the repairs, the JSON was valid and parsable. It is not the most elegant string manipulation, but it works — and the pattern generalizes. Inconsistent formats appear everywhere: dates saved as strings, typos in categorical values, numeric IDs stored as floats. The remedy is always standardization.
Practice #4: Data type validation and schema checking
Everything starts at load time. Discovering later that dates are strings or numbers are objects wastes hours. In the NoBroker project, types were validated during the CSV read itself, using the appropriate pandas parameters.
data = pd.read_csv('property_data_set.csv')
print(data('activation_date').dtype)
data = pd.read_csv('property_data_set.csv',
parse_dates=('activation_date'),
infer_datetime_format=True,
dayfirst=True)
print(data('activation_date').dtype)Here is the output.
The same validation was applied to the interactions dataset.
interaction = pd.read_csv('property_interactions.csv',
parse_dates=('request_date'),
infer_datetime_format=True,
dayfirst=True)This was necessary, not merely good practice: the project computes date-time differences between activation and request dates, and if the dates load as strings, the following code raises an error.
num_req('request_day') = (num_req('request_date') - num_req('activation_date')) / np.timedelta64(1, 'D')Schema checks ensure the structure does not silently change. In production, data also drifts — its distribution shifts over time. Drift can be simulated by perturbing input ratios and checking whether the model or its validation logic detects and responds to the change.
Documenting the cleaning steps
Three months later, nobody remembers why request_day_within_3d was capped at 10. Six months later, a teammate breaks the pipeline by removing an outlier filter. A year later, the model is in production and no one understands why it fails. Documentation is not optional — it is the difference between a reproducible pipeline and a script that works until it doesn’t. The NoBroker project documented every change in code comments and structured notebook sections with explanations and a table of contents.
# Assignment
# Read and Explore All Datasets
# Data Engineering
Handling Pics Data
Number of Interactions Within 3 Days
Number of Interactions Within 7 Days
Merge Data
# Exploratory Data Analysis and Processing
# Feature Engineering
Remove Outliers
One-Hot Encoding
MinMaxScaler
Classical Machine Learning
Predicting Interactions Within 3 Days
Deep Learning
# Try to correct the first Json
# Try to replace corrupted values then convert to json
# Function to correct corrupted json and get count of photosVersion control matters just as much: track changes to cleaning logic, save intermediate datasets, and keep a changelog of what was tried. The goal is not perfection; it is clarity. A decision that cannot be explained cannot be defended when the model fails.
Final thoughts
Clean data is a myth. The strongest data scientists are not the ones who avoid messy datasets but the ones who know how to tame them: they surface missing values before training, catch outliers before they skew predictions, check schemas before joining tables, and write everything down so the next person does not start from scratch. Foundational habits like these — alongside basic statistical fluency — transfer to every project, because real impact comes from building something functional out of imperfect data.
Limitations and what to watch
A few caveats for anyone applying these techniques directly. IQR-based outlier removal is a blunt instrument: it can silently discard rare but legitimate cases (genuine luxury listings, in this example), so removed rows should be inspected before being thrown away. Mean and mode imputation preserve row counts but flatten variance and can bias models when data is not missing at random — more careful approaches model the missingness itself. Ad-hoc string surgery on malformed JSON works for one-off projects but is fragile in production, where a proper parser or upstream fix is safer. And thresholds used here (caps at 10, specific IQR multipliers) are project-specific choices, not universal constants; every dataset deserves its own justification for each cleaning decision.
