Validating Detected PII with Checksums

Within the broader Regex Pattern Libraries for PII Detection specification — part of the PII Extraction & Redaction Pipelines architecture — this page closes the gap a purely structural regex leaves open: a nine-digit sequence, a sixteen-digit run, or a well-formed IBAN-shaped string can match the grammar perfectly and still not be the identifier you think it is. Regex answers does this look like a card number; a checksum answers is this arithmetically a valid card number. The engineers who need this are running a Data Subject Request (DSR) pipeline where a raw regex layer, built as in Building a Regex Library for Email and SSN Detection, produces high-recall but noisy hits, and where both error directions carry a compliance cost. A false negative — a real identifier that goes undetected — leaks personal data and breaches the confidentiality duty of GDPR Art. 5(1)(f). A false positive — an invoice number redacted as an SSN, or a legitimate order ID masked out of an export — corrupts a record the subject was entitled to receive under GDPR Art. 15 and undermines the accuracy principle of GDPR Art. 5(1)(d). The value of a checksum layer is that it cuts false positives without lowering recall: it only ever rejects candidates that are arithmetically impossible, so a genuine identifier is never dropped by the check.

The pattern is a validation layer placed immediately after the regex pass: each hit is dispatched to a type-specific checksum, and only arithmetically valid candidates are confirmed as PII while the rest are discarded as false positives.

Prerequisites #

The code targets Python 3.11+ and uses only the standard library — no third-party dependency is needed for the checksum math itself:

• pydantic (v2) — to wrap the validator layer in a strict, auditable config using v2 idioms (model_config = ConfigDict(...), field_validator, model_validate). Never v1 (class Config, @validator, .dict()).
• re and string (stdlib) — normalization of matched values before validation.

The layer assumes it receives raw hits from an upstream regex pass — each hit a dict (or dataclass) carrying at least a type, the matched value, and its offsets — exactly the {type, start, end, value, confidence} shape emitted by the regex pattern library. Checksums validate structure, not entitlement: a Luhn-valid number is a syntactically real card, not proof it belongs to the requesting subject. Contextual disambiguation is still the job of the Confidence Scoring & Thresholds stage; the checksum layer’s role is narrowly to strip the impossible candidates before they reach it.

Step-by-Step Implementation #

Step 1 — Luhn (mod-10) for payment card numbers #

The Luhn algorithm is the check digit built into every payment card PAN. Doubling every second digit from the right, summing the digits of those products with the untouched digits, and requiring the total to be a multiple of ten catches all single-digit errors and most transpositions — which is exactly the noise a bare \d{13,19} regex produces.

def luhn(number: str) -> bool:
    """Return True if the digit string satisfies the Luhn mod-10 check."""
    digits = [int(c) for c in number if c.isdigit()]
    if len(digits) < 13:
        return False
    checksum, parity = 0, len(digits) % 2
    for i, d in enumerate(digits):
        if i % 2 == parity:
            d *= 2
            if d > 9:
                d -= 9
        checksum += d
    return checksum % 10 == 0

Step 2 — IBAN mod-97 for international bank accounts #

An IBAN carries a two-digit check value validated by ISO 7064 mod-97. Move the first four characters to the end, replace each letter with its position value (A=10 … Z=35), interpret the result as one large integer, and require value % 97 == 1. Python’s arbitrary-precision integers make the “large integer” step trivial.

import string


def iban_valid(iban: str) -> bool:
    """Validate an IBAN via ISO 7064 mod-97 (structure only, not account existence)."""
    s = "".join(iban.split()).upper()
    if not (15 <= len(s) <= 34) or not s[:2].isalpha() or not s[2:4].isdigit():
        return False
    rearranged = s[4:] + s[:4]
    digits = "".join(
        str(string.ascii_uppercase.index(c) + 10) if c.isalpha() else c
        for c in rearranged
    )
    if not digits.isdigit():
        return False
    return int(digits) % 97 == 1

Step 3 — ABA checksum for US routing numbers #

A US bank routing number is nine digits with a position-weighted checksum. Weighting the digits 3, 7, 1, 3, 7, 1, 3, 7, 1 and requiring the weighted sum to be a multiple of ten rejects the overwhelming majority of nine-digit sequences that merely look like routing numbers.

def aba_valid(routing: str) -> bool:
    """Validate a nine-digit ABA routing number by its weighted checksum."""
    d = [int(c) for c in routing if c.isdigit()]
    if len(d) != 9:
        return False
    weights = (3, 7, 1, 3, 7, 1, 3, 7, 1)
    total = sum(w * n for w, n in zip(weights, d))
    return total % 10 == 0

Step 4 — Structural plausibility for US SSNs #

An SSN has no arithmetic check digit, but its structure excludes ranges that were never issued: an area of 000, 666, or 900–999; a group of 00; or a serial of 0000 are all invalid regardless of context. These structural rules are the SSN equivalent of a checksum — they cannot confirm a real SSN, but they cheaply reject impossible ones that a \d{3}-\d{2}-\d{4} pattern would otherwise pass.

def ssn_plausible(ssn: str) -> bool:
    """Reject US SSNs in never-issued area, group, or serial ranges."""
    digits = "".join(c for c in ssn if c.isdigit())
    if len(digits) != 9:
        return False
    area, group, serial = int(digits[:3]), int(digits[3:5]), int(digits[5:])
    if area == 0 or area == 666 or area >= 900:
        return False
    if group == 0 or serial == 0:
        return False
    return True

Step 5 — A validator layer that filters raw regex hits #

The individual checks compose into a single dispatch: map each pii_type to its validator, run every raw hit through the matching check, and keep only the ones that pass. A hit whose type has no registered validator is not silently dropped — it passes through untouched so an unchecked identifier is never lost, in line with the fail-closed posture the parent library applies at its margins.

from collections.abc import Callable

VALIDATORS: dict[str, Callable[[str], bool]] = {
    "card_pan": luhn,
    "iban": iban_valid,
    "aba_routing": aba_valid,
    "us_ssn": ssn_plausible,
}


def filter_hits(hits: list[dict]) -> tuple[list[dict], list[dict]]:
    """Split raw regex hits into checksum-confirmed and rejected false positives."""
    confirmed, rejected = [], []
    for hit in hits:
        validator = VALIDATORS.get(hit["type"])
        if validator is None:
            confirmed.append(hit)             # no check available: keep, do not lose
            continue
        if validator(hit["value"]):
            confirmed.append({**hit, "confidence": 1.0, "checksum": "pass"})
        else:
            rejected.append({**hit, "confidence": 0.0, "checksum": "fail"})
    return confirmed, rejected

Step 6 — Wrap the layer in a Pydantic v2 policy #

Because the decision to drop a candidate is itself a compliance action, the layer’s behaviour is governed by a validated, immutable policy. fail_action decides whether a checksum failure is dropped outright or diverted to human review — dropping is safe for high-arithmetic-strength checks like Luhn and mod-97, while a bare structural check like the SSN rule may warrant review before a legitimate record is discarded.

from typing import Literal
from pydantic import BaseModel, ConfigDict, field_validator


class ChecksumPolicy(BaseModel):
    model_config = ConfigDict(extra="forbid", frozen=True)

    enabled_types: frozenset[str]
    fail_action: Literal["drop", "review"] = "drop"

    @field_validator("enabled_types")
    @classmethod
    def known_types_only(cls, v: frozenset[str]) -> frozenset[str]:
        unknown = v - VALIDATORS.keys()
        if unknown:
            raise ValueError(f"No checksum validator registered for: {sorted(unknown)}")
        return v


def apply_policy(hits: list[dict], policy: ChecksumPolicy) -> tuple[list[dict], list[dict]]:
    """Run only the policy-enabled validators; unlisted types pass through."""
    scoped = [h for h in hits if h["type"] in policy.enabled_types]
    passthrough = [h for h in hits if h["type"] not in policy.enabled_types]
    confirmed, rejected = filter_hits(scoped)
    return confirmed + passthrough, rejected

Setting extra="forbid" and frozen=True makes the policy a stable audit artifact, and known_types_only fails the build if someone enables a type with no validator behind it — turning a silent no-op into a loud error, the accountability posture GDPR Art. 5(2) expects.

Configuration Reference #

Verification #

Validate the checksum layer against known-good and known-bad vectors, so a regression in the arithmetic is caught before it either leaks a real identifier or discards a legitimate record. The vectors below are published test values, not live personal data.

import pytest


@pytest.mark.parametrize("value, ok", [
    ("4111111111111111", True),   # canonical Luhn-valid test PAN
    ("4111111111111112", False),  # last digit broken -> Luhn fail
    ("49927398716", False),       # too short for a PAN
])
def test_luhn(value: str, ok: bool):
    assert luhn(value) is ok


@pytest.mark.parametrize("value, ok", [
    ("GB82 WEST 1234 5698 7654 32", True),   # ISO test IBAN
    ("GB82 WEST 1234 5698 7654 33", False),  # broken check digits
])
def test_iban(value: str, ok: bool):
    assert iban_valid(value) is ok


@pytest.mark.parametrize("value, ok", [
    ("021000021", True),    # valid weighted checksum
    ("021000022", False),   # checksum fails
])
def test_aba(value: str, ok: bool):
    assert aba_valid(value) is ok


@pytest.mark.parametrize("value, ok", [
    ("078-05-1120", True),   # structurally plausible
    ("000-12-3456", False),  # invalid area 000
    ("666-45-6789", False),  # invalid area 666
    ("123-00-6789", False),  # invalid group 00
])
def test_ssn(value: str, ok: bool):
    assert ssn_plausible(value) is ok


def test_layer_splits_hits():
    hits = [
        {"type": "card_pan", "value": "4111111111111111"},
        {"type": "us_ssn", "value": "000-12-3456"},
        {"type": "person", "value": "Jordan Vega"},  # no validator -> passthrough
    ]
    confirmed, rejected = filter_hits(hits)
    assert {h["value"] for h in confirmed} == {"4111111111111111", "Jordan Vega"}
    assert [h["value"] for h in rejected] == ["000-12-3456"]

Expected output: the Luhn-valid PAN and the un-validatable person name are confirmed; the invalid-area SSN is rejected as a false positive; every checksum function agrees with its published vectors. Wire the same vectors into CI so a change to the arithmetic fails the build rather than silently shifting the pipeline’s precision.

Troubleshooting #

Formatted values fail a checksum that should pass. Root cause: separators, spaces, or a surrounding label were passed into the validator unstripped. Fix: normalize before validating — the functions above already discard non-digits, but confirm the regex value is the identifier alone and not SSN: 078-05-1120; enable normalize so 4111 1111 1111 1111 and 4111111111111111 validate identically.

Real identifiers are being dropped (over-strict SSN check). Root cause: treating a structural-only check as if it had checksum strength and dropping on failure. Fix: an SSN structural rule rejects only never-issued ranges — if legitimate values are still being lost, set fail_action="review" for us_ssn so a human confirms before a record protected under GDPR Art. 5(1)(d) is discarded; reserve drop for arithmetic checks like Luhn and mod-97.

A whole class of identifier stops being redacted. Root cause: a type was enabled in the policy but its regex emits a type string that does not match a VALIDATORS key, so nothing routes to the check — or worse, hits are silently dropped. Fix: rely on known_types_only to fail the build on an unregistered type, and confirm passthrough_unknown keeps un-validatable hits rather than discarding them, so a missing validator never becomes a silent leak.

IBANs from one country always fail. Root cause: the mod-97 check is correct but the upstream regex captured a truncated or over-long span, so the length guard rejects it before the arithmetic runs. Fix: verify the IBAN pattern captures the full country-specific length (they range 15–34 characters); the checksum cannot rescue a mis-bounded match, which is an offset-alignment concern shared with Structured vs Unstructured Data Sync.

Checksum passes but the value still is not really PII. Root cause: expecting arithmetic validity to confirm entitlement. A Luhn-valid test PAN or a structurally valid SSN is a syntactically real identifier, not proof it is the subject’s live data. Fix: treat the checksum layer as precision-raising only, and route residual ambiguity to Confidence Scoring & Thresholds for contextual disambiguation rather than auto-redacting on checksum alone.

Frequently Asked Questions #

Related #

• Regex Pattern Libraries for PII Detection — the parent specification whose raw structural matches this checksum layer filters.
• Building a Regex Library for Email and SSN Detection — the concrete high-recall patterns that produce the hits validated here.
• Confidence Scoring & Thresholds for PII Detection — where checksum-confirmed and residual-ambiguous matches are scored and routed.
• Structured vs Unstructured Data Sync — keeping validated offsets aligned across CRM rows and free-text attachments.
• PII Extraction & Redaction Pipelines — the parent architecture this precision-raising stage feeds.