5 Useful DIY Python Functions for JSON Parsing and Processing

Image by author

Introduction

Working with JSON in Python often turns out to be harder than expected, and basic json.loads() only takes the job so far. API responses, configuration files and data exports frequently arrive deeply nested, inconsistently structured, or missing fields that code assumes will be present.

The five functions below cover tasks that recur constantly in web scraping, API integration and data processing: safely reading nested values, flattening nested objects, deep-merging configurations, filtering fields against a schema, and converting between nested JSON and flat dot-notation keys. The full code is available on GitHub.

1. Safely extracting nested values

JSON objects often sit several levels deep, and reaching into them directly is where a KeyError tends to appear. The following helper walks a path one key at a time and returns a fallback instead of raising when part of the path is missing:

def get_nested_value(data, path, default=None):
    """
    Safely extract nested values from JSON using dot notation.

    Args:
        data: Dictionary or JSON object
        path: Dot-separated string like "user.profile.email"
        default: Value to return if path doesn't exist

    Returns:
        The value at the path, or default if not found
    """
    keys = path.split('.')
    current = data

    for key in keys:
        if isinstance(current, dict):
            current = current.get(key)
            if current is None:
                return default
        elif isinstance(current, list):
            try:
                index = int(key)
                current = current(index)
            except (ValueError, IndexError):
                return default
        else:
            return default

    return current

Example:

# Sample JSON data
user_data = {
    "user": {
        "id": 123,
        "profile": {
            "name": "Allie",
            "email": "allie@example.com",
            "settings": {
                "theme": "dark",
                "notifications": True
            }
        },
        "posts": (
            {"id": 1, "title": "First Post"},
            {"id": 2, "title": "Second Post"}
        )
    }
}

# Extract values
email = get_nested_value(user_data, "user.profile.email")
theme = get_nested_value(user_data, "user.profile.settings.theme")
first_post = get_nested_value(user_data, "user.posts.0.title")
missing = get_nested_value(user_data, "user.profile.age", default=25)

print(f"Email: {email}")
print(f"Theme: {theme}")
print(f"First post: {first_post}")
print(f"Age (default): {missing}")

Output:

Email: allie@example.com
Theme: dark
First post: First Post
Age (default): 25

The function splits the path string on dots and steps through the data structure key by key. At each level it checks whether the current value is a dictionary or a list. For dictionaries it uses .get(key), which returns None for missing keys rather than raising; for lists it tries to convert the key into an integer index. The default parameter supplies the fallback when part of the path does not exist, which keeps code from crashing on incomplete or inconsistent API data. This pattern is especially useful for responses where some fields are optional or appear only under certain conditions.

2. Flattening nested JSON into a single level

Machine learning pipelines, CSV exports and database inserts generally expect flat data, while API responses and configuration files tend to be nested. Converting nested objects into flat key-value pairs is therefore a common step:

def flatten_json(data, parent_key='', separator="_"):
    """
    Flatten nested JSON into a single-level dictionary.

    Args:
        data: Nested dictionary or JSON object
        parent_key: Prefix for keys (used in recursion)
        separator: String to join nested keys

    Returns:
        Flattened dictionary with concatenated keys
    """
    items = ()

    if isinstance(data, dict):
        for key, value in data.items():
            new_key = f"{parent_key}{separator}{key}" if parent_key else key

            if isinstance(value, dict):
                # Recursively flatten nested dicts
                items.extend(flatten_json(value, new_key, separator).items())
            elif isinstance(value, list):
                # Flatten lists with indexed keys
                for i, item in enumerate(value):
                    list_key = f"{new_key}{separator}{i}"
                    if isinstance(item, (dict, list)):
                        items.extend(flatten_json(item, list_key, separator).items())
                    else:
                        items.append((list_key, item))
            else:
                items.append((new_key, value))
    else:
        items.append((parent_key, data))

    return dict(items)

Complex nested JSON:

# Complex nested JSON
product_data = {
    "product": {
        "id": 456,
        "name": "Laptop",
        "specs": {
            "cpu": "Intel i7",
            "ram": "16GB",
            "storage": {
                "type": "SSD",
                "capacity": "512GB"
            }
        },
        "reviews": (
            {"rating": 5, "comment": "Excellent"},
            {"rating": 4, "comment": "Good value"}
        )
    }
}

flattened = flatten_json(product_data)

for key, value in flattened.items():
    print(f"{key}: {value}")

Output:

product_id: 456
product_name: Laptop
product_specs_cpu: Intel i7
product_specs_ram: 16GB
product_specs_storage_type: SSD
product_specs_storage_capacity: 512GB
product_reviews_0_rating: 5
product_reviews_0_comment: Excellent
product_reviews_1_rating: 4
product_reviews_1_comment: Good value

The function recurses to an arbitrary depth. For a dictionary it processes each key-value pair, joining the parent key and the current key with a separator; for a list it uses the index as part of the key, preserving order and structure. A flattened key such as reviews_0_rating shows the rating of the first review. The separator parameter controls the output format — dots for dot notation, underscores for snake_case, or slashes for path-like keys — which makes it convenient when turning JSON into DataFrame columns or CSV rows that each need a unique name.

3. Deep-merging multiple JSON objects

Configuration management often means combining several JSON sources: default settings, environment-specific overrides, user preferences and more. A plain dict.update() only handles the top level, so a recursive deep merge is needed to combine nested dictionaries rather than replace them wholesale:

def deep_merge_json(base, override):
    """
    Deep merge two JSON objects, with override taking precedence.

    Args:
        base: Base dictionary
        override: Dictionary with values to override/add

    Returns:
        New dictionary with merged values
    """
    result = base.copy()

    for key, value in override.items():
        if key in result and isinstance(result(key), dict) and isinstance(value, dict):
            # Recursively merge nested dictionaries
            result(key) = deep_merge_json(result(key), value)
        else:
            # Override or add the value
            result(key) = value

    return result

Default configuration:

import json

# Default configuration
default_config = {
    "database": {
        "host": "localhost",
        "port": 5432,
        "timeout": 30,
        "pool": {
            "min": 2,
            "max": 10
        }
    },
    "cache": {
        "enabled": True,
        "ttl": 300
    },
    "logging": {
        "level": "INFO"
    }
}

# Production overrides
prod_config = {
    "database": {
        "host": "prod-db.example.com",
        "pool": {
            "min": 5,
            "max": 50
        }
    },
    "cache": {
        "ttl": 600
    },
    "monitoring": {
        "enabled": True
    }
}

merged = deep_merge_json(default_config, prod_config)

print(json.dumps(merged, indent=2))

Production overrides:

{
  "database": {
    "host": "prod-db.example.com",
    "port": 5432,
    "timeout": 30,
    "pool": {
      "min": 5,
      "max": 50
    }
  },
  "cache": {
    "enabled": true,
    "ttl": 600
  },
  "logging": {
    "level": "INFO"
  },
  "monitoring": {
    "enabled": true
  }
}

Where both objects contain a dictionary under the same key, the function merges those dictionaries instead of overwriting, preserving values that were not explicitly changed. For example database.port and database.timeout keep their defaults while database.host is overridden, and nested pool settings such as min and max are both updated. New keys that are absent from the base, such as a monitoring section set to True in the production overrides, are added. Merges can also be chained to build a layered configuration:

final_config = deep_merge_json(
    deep_merge_json(default_config, prod_config),
    user_preferences
)

This layered default-then-environment-then-runtime pattern is common in application configuration.

4. Filtering JSON by schema or whitelist

APIs frequently return more data than is needed, which bloats responses and can expose sensitive fields in logs. The following function keeps only the fields named in a schema:

def filter_json(data, schema):
    """
    Filter JSON to keep only fields specified in schema.

    Args:
        data: Dictionary or JSON object to filter
        schema: Dictionary defining which fields to keep
                Use True to keep a field, nested dict for nested filtering

    Returns:
        Filtered dictionary containing only specified fields
    """
    if not isinstance(data, dict) or not isinstance(schema, dict):
        return data

    result = {}

    for key, value in schema.items():
        if key not in data:
            continue

        if value is True:
            # Keep this field as-is
            result(key) = data(key)
        elif isinstance(value, dict):
            # Recursively filter nested object
            if isinstance(data(key), dict):
                filtered_nested = filter_json(data(key), value)
                if filtered_nested:
                    result(key) = filtered_nested
            elif isinstance(data(key), list):
                # Filter each item in the list
                filtered_list = ()
                for item in data(key):
                    if isinstance(item, dict):
                        filtered_item = filter_json(item, value)
                        if filtered_item:
                            filtered_list.append(filtered_item)
                    else:
                        filtered_list.append(item)
                if filtered_list:
                    result(key) = filtered_list

    return result

Sample API response:

import json
# Sample API response
api_response = {
    "user": {
        "id": 789,
        "username": "Cayla",
        "email": "cayla@example.com",
        "password_hash": "secret123",
        "profile": {
            "name": "Cayla Smith",
            "bio": "Software developer",
            "avatar_url": "https://example.com/avatar.jpg",
            "private_notes": "Internal notes"
        },
        "posts": (
            {
                "id": 1,
                "title": "Hello World",
                "content": "My first post",
                "views": 100,
                "internal_score": 0.85
            },
            {
                "id": 2,
                "title": "Python Tips",
                "content": "Some tips",
                "views": 250,
                "internal_score": 0.92
            }
        )
    },
    "metadata": {
        "request_id": "abc123",
        "server": "web-01"
    }
}

# Schema defining what to keep
public_schema = {
    "user": {
        "id": True,
        "username": True,
        "profile": {
            "name": True,
            "avatar_url": True
        },
        "posts": {
            "id": True,
            "title": True,
            "views": True
        }
    }
}

filtered = filter_json(api_response, public_schema)

print(json.dumps(filtered, indent=2))

Output:

{
  "user": {
    "id": 789,
    "username": "Cayla",
    "profile": {
      "name": "Cayla Smith",
      "avatar_url": "https://example.com/avatar.jpg"
    },
    "posts": (
      {
        "id": 1,
        "title": "Hello World",
        "views": 100
      },
      {
        "id": 2,
        "title": "Python Tips",
        "views": 250
      }
    )
  }
}

The posts array is filtered so that each post retains only id, title and views, while content and internal_score are dropped. Sensitive fields such as password_hash and private_notes never reach the output, which makes the function useful for logging or for cleaning data before sending it to a frontend. Different schemas can serve different use cases — a minimal schema for list views, a detailed one for single items, and an admin schema that includes everything.

5. Converting between JSON and dot notation

Some systems store flat key-value pairs, yet nested JSON is easier to work with in code. A pair of functions can convert in both directions.

JSON to dot notation

def json_to_dot_notation(data, parent_key=''):
    """
    Convert nested JSON to flat dot-notation dictionary.

    Args:
        data: Nested dictionary
        parent_key: Prefix for keys (used in recursion)

    Returns:
        Flat dictionary with dot-notation keys
    """
    items = {}

    if isinstance(data, dict):
        for key, value in data.items():
            new_key = f"{parent_key}.{key}" if parent_key else key

            if isinstance(value, dict):
                items.update(json_to_dot_notation(value, new_key))
            else:
                items(new_key) = value
    else:
        items(parent_key) = data

    return items

Dot notation back to JSON

def dot_notation_to_json(flat_data):
    """
    Convert flat dot-notation dictionary to nested JSON.

    Args:
        flat_data: Dictionary with dot-notation keys

    Returns:
        Nested dictionary
    """
    result = {}

    for key, value in flat_data.items():
        parts = key.split('.')
        current = result

        for i, part in enumerate(parts(:-1)):
            if part not in current:
                current(part) = {}
            current = current(part)

        current(parts(-1)) = value

    return result

Testing the round trip:

import json
# Original nested JSON
config = {
    "app": {
        "name": "MyApp",
        "version": "1.0.0"
    },
    "database": {
        "host": "localhost",
        "credentials": {
            "username": "admin",
            "password": "secret"
        }
    },
    "features": {
        "analytics": True,
        "notifications": False
    }
}

# Convert to dot notation (for environment variables)
flat = json_to_dot_notation(config)
print("Flat format:")
for key, value in flat.items():
    print(f"  {key} = {value}")

print("n" + "="*50 + "n")

# Convert back to nested JSON
nested = dot_notation_to_json(flat)

print("Nested format:")
print(json.dumps(nested, indent=2))

Output:

Flat format:
  app.name = MyApp
  app.version = 1.0.0
  database.host = localhost
  database.credentials.username = admin
  database.credentials.password = secret
  features.analytics = True
  features.notifications = False

==================================================

Nested format:
{
  "app": {
    "name": "MyApp",
    "version": "1.0.0"
  },
  "database": {
    "host": "localhost",
    "credentials": {
      "username": "admin",
      "password": "secret"
    }
  },
  "features": {
    "analytics": true,
    "notifications": false
  }
}

The json_to_dot_notation function flattens the structure, splitting each key on dots, while dot_notation_to_json rebuilds the nesting by creating intermediate dictionaries as needed — the loop handles every part except the last to create the nesting levels, then assigns the value to the final key. This keeps configuration readable while still fitting a flat key-value store.

Limitations and what to watch

These helpers are deliberately lightweight, so a few caveats are worth keeping in mind. The path and flattening functions assume dot separators, which will behave unexpectedly if a key itself contains a dot; choosing a separator that cannot appear in the data avoids this. Flattening large or deeply nested objects can produce very wide records and, for pathological inputs, hit Python’s recursion limit. The deep-merge function combines dictionaries but replaces lists rather than merging them, which may or may not be the desired behaviour. And schema filtering only removes fields that are named, so any new sensitive field added upstream will pass through until the schema is updated. Validating input with a library such as Pydantic or JSON Schema is worth considering for production systems.

Wrapping up

JSON processing goes well beyond a basic json.loads() call. Most projects eventually need tools to navigate nested structures, reshape them, merge configurations, filter fields and convert between formats, and these same patterns transfer to XML, YAML or custom formats. A reasonable starting point is the safe-access function to prevent KeyError exceptions, adding the others as needs arise.

Bala Priya C is a developer and technical writer from India whose interests span mathematics, programming, data science, DevOps and natural language processing. She writes tutorials, how-to guides and opinion pieces to share knowledge with the developer community.