Redacting PII in Nested JSON Payloads

Within the broader Structured vs Unstructured Data Sync stage of the PII Extraction & Redaction Pipelines architecture, one shape of data resists both the regex overlay and the coordinate-aware PDF extractors: the arbitrarily nested JSON blob. SaaS webhooks, CRM export envelopes, event-bus messages, and API responses arrive as dictionaries inside lists inside dictionaries, with personal data buried at unpredictable depths. When a Data Subject Request (DSR) demands that a value be removed or masked, you cannot flatten the document and hope — flattening destroys the schema the downstream consumer still expects. The engineers who hit this are the integration and privacy teams reconciling structured system-of-record fields against semi-structured payloads, the same reconciliation problem solved for documents in syncing structured CRM data with unstructured PDFs. This page builds a recursive, path-aware redactor that walks the whole tree, masks the values a policy names, preserves every type and every position, and satisfies the data-minimization duty of GDPR Art. 5(1)© without breaking the contract the payload’s consumer relies on.

The redactor is deterministic and structure-preserving: it descends dicts and lists, accumulates a key path at each step, tests that path against an allow/deny policy, and replaces only the matched leaf while leaving the container shape intact.

Prerequisites #

• Python 3.11+ — for tuple[...] / list[...] generics and stable insertion-ordered dicts, which keep the rebuilt payload byte-comparable to the input.
• Pydantic v2 — pip install "pydantic>=2.6". The redaction policy is a validated model, so a typo in a key pattern is rejected at load time rather than silently redacting nothing.
• Standard library only for the walk — json for round-tripping and copy are enough; the recursion needs no third-party graph library. This keeps the redactor safe to run inside a queue worker with a minimal dependency surface.
• A JSON payload that parses to native Python — dict, list, str, int, float, bool, and None. The walker assumes json.loads output; if you hold raw bytes, parse first so the structure the policy targets is real Python containers, not a string.

The redactor owns only the masking decision on a payload whose PII locations are already known from the discovery stage. It does not detect PII; it enforces a policy that names where PII lives by path, complementing the probabilistic detection in NLP-based entity recognition.

Step-by-step implementation #

Step 1 — Model the redaction policy with path patterns #

A path is the sequence of keys and array indices from the document root to a value: ("profile", "addresses", 0, "line1"). A policy names paths with dotted patterns, where * matches exactly one segment (any key or any array index) and ** matches any depth. deny_paths marks what to mask; allow_paths carves out exceptions that survive even when a broader deny would catch them — an explicit allow always wins, so an audit field can be preserved inside an otherwise-redacted subtree.

from pydantic import BaseModel, ConfigDict, Field, field_validator


class RedactionPolicy(BaseModel):
    """A path-addressed allow/deny policy for masking nested values."""

    model_config = ConfigDict(frozen=True, extra="forbid")

    deny_paths: tuple[str, ...] = Field(default_factory=tuple)
    allow_paths: tuple[str, ...] = Field(default_factory=tuple)
    placeholder: str = "[REDACTED]"
    max_depth: int = Field(default=64, gt=0)

    @field_validator("deny_paths", "allow_paths")
    @classmethod
    def _non_empty_segments(cls, patterns: tuple[str, ...]) -> tuple[str, ...]:
        for p in patterns:
            if not p or any(seg == "" for seg in p.split(".")):
                raise ValueError(f"malformed path pattern: {p!r}")
        return patterns

Freezing the policy (frozen=True) means the same object drives every worker in a fleet identically, and extra="forbid" stops a mistyped field name (deny_path instead of deny_paths) from loading a policy that quietly redacts nothing — a silent under-redaction is a disclosure under GDPR Art. 5(1)(f).

Step 2 — Match a concrete path against a pattern #

Matching is a small recursive wildcard engine. The concrete path is normalized to strings (array indices become their decimal form) so one pattern language covers both keys and indices. ** matches zero or more segments, which is what lets a single **.ssn catch a social-security number wherever it appears in the tree.

def _match(pattern: tuple[str, ...], path: tuple[str, ...]) -> bool:
    """Glob-style match: '*' is one segment, '**' is any number of segments."""
    if not pattern:
        return not path
    head, rest = pattern[0], pattern[1:]
    if head == "**":
        # '**' consumes nothing, or consumes one segment and retries.
        return _match(rest, path) or (bool(path) and _match(pattern, path[1:]))
    if not path:
        return False
    if head == "*" or head == path[0]:
        return _match(rest, path[1:])
    return False


def should_redact(policy: RedactionPolicy, path: tuple[object, ...]) -> bool:
    """A path is redacted when a deny pattern matches and no allow pattern does."""
    normalized = tuple(str(seg) for seg in path)
    if any(_match(tuple(a.split(".")), normalized) for a in policy.allow_paths):
        return False
    return any(_match(tuple(d.split(".")), normalized) for d in policy.deny_paths)

Evaluating allow_paths first makes the exception unambiguous: a value under **.audit_id is kept even if it sits inside a denied profile.** subtree, so non-repudiation metadata survives redaction intact.

Step 3 — Replace values without breaking their type #

A redactor that turns every masked value into the string "[REDACTED]" breaks any consumer that expects an integer age or a boolean flag — and a schema break downstream is as disruptive as a missed redaction. Type-preserving replacement keeps the JSON schema valid: strings become the placeholder, numbers collapse to a neutral zero, booleans to False, and null stays null. Because bool is a subclass of int in Python, it must be checked first.

def mask_value(value: object, policy: RedactionPolicy) -> object:
    """Replace a scalar with a type-preserving redaction marker."""
    if isinstance(value, bool):
        return False
    if isinstance(value, str):
        return policy.placeholder
    if isinstance(value, int):
        return 0
    if isinstance(value, float):
        return 0.0
    if value is None:
        return None
    # A dict/list caught directly by a deny path: drop the whole subtree.
    return policy.placeholder

Redacting a whole container (when a deny pattern names the container’s own path rather than its leaves) collapses it to the placeholder string, which is the correct behavior for “remove this entire payment_methods array” — the array is gone, not emptied of individual fields.

Step 4 — Walk the tree recursively, preserving structure #

The core is a recursive redact that rebuilds the payload rather than mutating it, so the caller’s original object is never altered and a cycle cannot corrupt a half-mutated structure. It threads the growing path, a depth counter bounded by policy.max_depth, and a seen set of container object ids to detect reference cycles.

def redact(
    node: object,
    policy: RedactionPolicy,
    path: tuple[object, ...] = (),
    depth: int = 0,
    seen: frozenset[int] = frozenset(),
) -> object:
    """Return a structurally identical copy with policy-matched values masked."""
    if depth > policy.max_depth:
        raise RecursionError(f"payload deeper than max_depth={policy.max_depth}")

    if isinstance(node, (dict, list)):
        if id(node) in seen:
            raise ValueError(f"cyclic reference at path {path!r}")
        seen = seen | {id(node)}

    if isinstance(node, dict):
        return {
            key: _process(value, policy, path + (key,), depth, seen)
            for key, value in node.items()
        }
    if isinstance(node, list):
        return [
            _process(item, policy, path + (index,), depth, seen)
            for index, item in enumerate(node)
        ]
    return node  # scalar root


def _process(
    value: object,
    policy: RedactionPolicy,
    child_path: tuple[object, ...],
    depth: int,
    seen: frozenset[int],
) -> object:
    """Mask a matched value; otherwise recurse into containers or keep scalars."""
    if should_redact(policy, child_path):
        return mask_value(value, policy)
    if isinstance(value, (dict, list)):
        return redact(value, policy, child_path, depth + 1, seen)
    return value

Dict comprehension preserves key insertion order and list comprehension preserves array length and position, so a consumer indexing addresses[1] still finds the second address — now with its line1 masked but its zip intact. Testing should_redact in _process before recursing is what lets a policy mask either a leaf or an entire subtree with the same pattern language.

Configuration reference #

Verification #

Correctness means three things at once: the named values are masked, the untouched values and the overall shape survive, and non-string types keep their type. This test exercises a deep object, an array of objects, an allow-override, and an integer field.

def test_nested_redaction_preserves_shape_and_types():
    payload = {
        "subject_id": "s-1001",
        "email": "ada@example.com",
        "age": 34,
        "profile": {
            "ssn": "123-45-6789",
            "audit_id": "a-77",
            "addresses": [
                {"line1": "1 Alpha St", "zip": "94016"},
                {"line1": "2 Beta Ave", "zip": "94017"},
            ],
        },
    }
    policy = RedactionPolicy(
        deny_paths=("email", "age", "profile.**", "profile.addresses.*.line1"),
        allow_paths=("**.audit_id", "profile.addresses.*.zip"),
    )
    out = redact(payload, policy)

    # Named PII is masked, types preserved.
    assert out["email"] == "[REDACTED]"
    assert out["age"] == 0                       # int stays int
    assert out["profile"]["ssn"] == "[REDACTED]"

    # Structure and allow-overrides survive intact.
    assert out["subject_id"] == "s-1001"          # never denied
    assert out["profile"]["audit_id"] == "a-77"   # allow wins over profile.**
    assert len(out["profile"]["addresses"]) == 2  # array length preserved
    assert out["profile"]["addresses"][0]["line1"] == "[REDACTED]"
    assert out["profile"]["addresses"][1]["zip"] == "94017"  # allow-kept

    # Original object is untouched (functional rebuild).
    assert payload["email"] == "ada@example.com"

Expect every denied path masked, both audit_id and each zip preserved by the allow rules despite the sweeping profile.** deny, both array elements present in order, and the caller’s original payload unchanged because redact rebuilds rather than mutates.

Troubleshooting #

Cyclic references crash or loop forever. Root cause: a payload assembled in memory (not from json.loads) contains a container that references an ancestor, so naive recursion never terminates. Fix: the seen id-set raises a ValueError on revisit; catch it, route the payload to quarantine, and never redact a structure JSON itself cannot represent. Genuine JSON is acyclic, so a cycle signals an upstream bug worth surfacing.

Huge payloads exhaust the stack or memory. Root cause: a deeply nested or very large document blows the recursion limit or doubles memory during the rebuild. Fix: the max_depth guard converts runaway depth into a clean RecursionError; for multi-megabyte documents, stream them with an incremental JSON parser and redact per top-level record instead of loading the whole array, mirroring how large exports are chunked in paginating large SaaS exports for DSR.

Non-string PII slips through unmasked. Root cause: a policy written for strings ignores a numeric account number or a boolean consent flag, or a masker that only handles str leaves other types visible. Fix: mask_value covers every JSON scalar type and preserves it; confirm the value’s path is actually denied, remembering that bool must be tested before int so a flag is not masked to 0.

A wildcard matches too much or too little. Root cause: * was used where ** was meant (or the reverse), so profile.*.ssn misses a nested profile.contacts.work.ssn. Fix: use ** when depth is variable and * for exactly one intervening segment; unit-test each pattern with _match against representative paths before deploying the policy.

An allow-override does not take effect. Root cause: the allow pattern does not normalize array indices the same way the path does, so profile.addresses.0.zip fails to match a path whose index is the integer 0. Fix: rely on the string normalization in should_redact, express index positions with * unless a single index is truly intended, and assert the override in a test the way the verification step does.

Frequently Asked Questions #

Related #

• Structured vs Unstructured Data Sync — the parent stage that reconciles a subject’s identity across storage worlds before redaction.
• Syncing Structured CRM Data with Unstructured PDFs — the sibling workflow that maps document evidence back to a structured record.
• NLP-Based Entity Recognition — the detector that finds PII in free text where no key path names it.
• Paginating Large SaaS Exports for DSR — streaming very large payloads so redaction runs per record instead of loading the whole export.
• PII Extraction & Redaction Pipelines — the end-to-end architecture this JSON redactor plugs into.