Regex Pattern Libraries for PII Detection

Within the broader PII Extraction & Redaction Pipelines architecture, a regex pattern library is the deterministic first pass — the layer that must catch every structurally unambiguous identifier before any probabilistic model is allowed to spend compute. When a Data Subject Request (DSR) demands that you locate and transform every occurrence of a person’s data inside a statutory window, an email address, a national insurance number, or a card PAN is not a “maybe”: it is a fixed grammar you either match or leak. The gap this page closes is the distance between a folder of ad-hoc re.compile() calls scattered through connector code and a versioned, validated, ReDoS-safe pattern library whose every match carries a type, an offset, a confidence, and an audit record a supervisory authority can inspect under GDPR Art. 30.

Naive regex handling fails in three predictable ways that each map to a compliance defect. Unvalidated patterns ship a malformed expression that silently matches nothing, so a whole class of identifier goes undetected and unredacted. Unbounded patterns backtrack catastrophically on a hostile document and stall a worker past the deadline. And matches emitted without a stable confidence and offset cannot be routed, reviewed, or reproduced when a regulator asks how a field was found. The pattern library treats regex as a governed compliance control, not a convenience — deterministic where determinism is possible, and explicit about where it must hand off to the NLP-Based Entity Recognition stage for contextual disambiguation.

Phase 1: Pattern Schema Definition & Pre-Deployment Validation #

A production pattern library cannot rely on strings compiled inline at call sites. Each pattern is a versioned record with a declared type, boundary policy, and confidence baseline, validated at definition time so a malformed expression fails the build rather than silently matching nothing in production. Using Pydantic v2 in strict mode, the library compiles every pattern during validation — a syntax error becomes a loud ValidationError, never a quiet gap in coverage. This mirrors the validation discipline applied at ingestion in the Schema Validation Rules stage, narrowed here to the pattern contract.

import re
from pydantic import BaseModel, ConfigDict, field_validator
from typing import Optional

class RegexPattern(BaseModel):
    model_config = ConfigDict(strict=True, frozen=True, extra="forbid")

    pii_type: str
    pattern: str
    flags: int = re.IGNORECASE | re.MULTILINE
    boundary_enforced: bool = True
    min_confidence: float = 0.85
    description: Optional[str] = None

    @field_validator("pattern")
    @classmethod
    def validate_compilable(cls, v: str) -> str:
        try:
            re.compile(v)
        except re.error as exc:
            raise ValueError(f"Invalid regex syntax: {exc}") from exc
        return v

    @field_validator("min_confidence")
    @classmethod
    def validate_threshold(cls, v: float) -> float:
        if not 0.0 <= v <= 1.0:
            raise ValueError("Confidence threshold must be in [0.0, 1.0]")
        return v

class PatternLibrary(BaseModel):
    model_config = ConfigDict(strict=True, frozen=True)

    version: str
    schema_version: int = 2
    patterns: list[RegexPattern]
    metadata: dict = {}

Because frozen=True guarantees no downstream stage mutates a validated pattern, and extra="forbid" makes schema drift a build failure rather than a silent leak, the library version becomes a first-class audit artifact: every match can later be attributed to the exact pattern set that produced it. The concrete high-risk expressions this schema wraps — the email grammar, the checksum-aware SSN pattern, the invalid-range exclusions — are developed in Building a Regex Library for Email and SSN Detection, aligned with the identifier definitions in NIST SP 800-122.

Phase 2: Compilation & ReDoS-Safe Execution #

Naive iteration over unbounded text guarantees latency degradation and exposes the pipeline to Regular Expression Denial of Service (ReDoS), where a crafted document forces catastrophic backtracking and stalls a worker past the DSR deadline. Pre-compilation and bounded execution are therefore non-negotiable. Patterns compile once into a cached list, boundary assertions are applied uniformly, and scanning proceeds over overlapping chunks so an identifier straddling a chunk edge is never split. Python’s re module should be used per the official compiled-pattern guidance.

Pydantic models are frozen, so the effective boundary-wrapped pattern is built at compile time rather than by mutating a field:

import re

def compile_library(library: PatternLibrary) -> list[tuple[str, re.Pattern]]:
    """Pre-compile every pattern into a cached (type, Pattern) list."""
    compiled: list[tuple[str, re.Pattern]] = []
    for p in library.patterns:
        effective = p.pattern
        if p.boundary_enforced and not p.pattern.startswith((r"\b", r"(?<!\w)")):
            effective = rf"\b(?:{p.pattern})\b"
        compiled.append((p.pii_type, re.compile(effective, p.flags)))
    return compiled

def scan_chunk(
    text: str,
    compiled: list[tuple[str, re.Pattern]],
    chunk_size: int = 8192,
    overlap: int = 128,
) -> list[dict]:
    """Boundary-aware scanner; overlap prevents split-token misses at chunk edges."""
    if len(text) <= chunk_size:
        segments = [(0, text)]
    else:
        segments, start = [], 0
        while start < len(text):
            end = min(start + chunk_size, len(text))
            segments.append((start, text[start:min(end + overlap, len(text))]))
            start = end - overlap

    results: list[dict] = []
    for base_offset, segment in segments:
        for pii_type, pattern in compiled:
            for m in pattern.finditer(segment):
                results.append({
                    "type": pii_type,
                    "start": base_offset + m.start(),  # offset into the original text
                    "end": base_offset + m.end(),
                    "value": m.group(),
                    "confidence": 1.0,  # deterministic baseline for a structural match
                })
    return results

To guard against catastrophic backtracking, avoid nested quantifiers ((.*)*, .*+) and test every pattern against pathological inputs behind a timeout. Where lookbehind or possessive semantics are genuinely required, migrate to the third-party regex module and use atomic grouping (?>...) to guarantee linear-time execution. The overlap-and-rebase step above is what keeps a fragmented tax ID or an address split across a chunk boundary from producing either a miss or a bad offset — a precondition for the offset alignment the redaction stage depends on.

Phase 3: Connector Dispatch & Predicate Pushdown #

Once validated and compiled, the library must interface with the ingestion connectors enumerated during Cross-System Data Discovery & Sync. Each store demands a tailored execution strategy: relational databases benefit from pushing a coarse predicate to the engine so Python-level regex only runs over candidate rows; columnar formats favour vectorized pre-filtering; and free-text and object stores stream through the chunker from Phase 2. Running a full-table Python scan over a multi-gigabyte table is the anti-pattern that blows the SLA.

Connector implementations enforce query timeouts, apply exponential backoff on transient failures, and emit per-source extraction metrics over hashed identifiers so dashboards never expose subject data. Because a regex match returns a byte offset into a specific record, keeping alignment with Structured vs Unstructured Data Sync is what guarantees an extracted offset maps back to the original field without drift when the same subject appears in both a CRM row and an attached PDF.

Phase 4: Confidence Routing & Compliance Hooks #

A structural regex match is high-precision but not automatically dispositive — a nine-digit sequence that passes the SSN grammar might still be an internal record ID. Every emitted match is therefore scored against the pattern’s min_confidence and routed accordingly: matches at or above threshold flow to the deterministic redaction queue, borderline hits enter a role-gated manual-review workflow, and structurally valid but contextually ambiguous matches are handed to Confidence Scoring Thresholds and, where context is decisive, to NLP-Based Entity Recognition. This routing prevents both under-redaction of real PII and over-redaction of legitimate business data.

from enum import Enum

class Route(str, Enum):
    REDACT = "redact"
    REVIEW = "manual_review"
    NLP_DISAMBIGUATE = "nlp"

def route_match(match: dict, min_confidence: float,
                review_floor: float = 0.60) -> Route:
    """Route a regex match to redaction, human review, or NLP disambiguation."""
    conf = match["confidence"]
    if conf >= min_confidence:
        return Route.REDACT
    if conf >= review_floor:
        return Route.REVIEW
    return Route.NLP_DISAMBIGUATE

Every routed match is written to an append-only audit record so the trail satisfies regulatory examination under GDPR Art. 30 without itself becoming a PII store. The hooks the library must capture:

• Audit trail — immutable log of pattern-library version, a salted hash of the matched value (never the raw value), the source record, and the extraction timestamp in UTC.
• Threshold overrides — role-based approval gates for confidence scores in the 0.60–0.85 band, so a human decision is recorded rather than assumed.
• False-positive feedback — when reviewers consistently override a given pattern, that signal flags the pattern for revision in the next library version.

Edge Cases & Conflict Resolution #

Regex determinism breaks down at the margins, and the library must fail closed rather than silently. Masked and partial identifiers (***-**-1234) will not match a full-format pattern; the library ships an explicit masked-form pattern at a lower confidence that routes to review rather than auto-redaction, because a partially masked value is often already compliant. Overlapping matches — an email whose local part also matches a username pattern — are resolved by longest-span-wins, then by the higher min_confidence pattern, so a single character offset is never claimed by two conflicting types. Internationalized inputs are normalized (IDN domains to Punycode, Unicode NFC) before matching so a homoglyph cannot slip an identifier past an ASCII-only grammar.

When two patterns disagree on the same span, the library never guesses: it records both candidates and routes the span to manual review, preserving both hypotheses in the audit trail. A structurally valid match in a field the discovery manifest marked non-PII (a free-text notes column, say) is not discarded — it is escalated, because a suppressed true positive is a worse compliance outcome than an extra review.

Performance & Scale Considerations #

At DSR volume the library must scan millions of records inside the statutory clock. The compiled (type, Pattern) list is built once per worker and cached — never rebuilt per document — and the pattern set is small and ordered most-selective-first so common negatives fail fast. Compiled libraries keyed by version live in a shared Redis cache so a worker pool warms instantly on deploy rather than recompiling identical patterns per process. CPU-bound scanning parallelizes cleanly across a process pool partitioned by source system, keeping one slow store from starving the others, and each pattern carries a per-document timeout so a single hostile input is quarantined to the dead-letter queue instead of stalling the batch. Predicate pushdown from Phase 3 is the largest single lever: shrinking the candidate set at the engine typically cuts Python-side regex work by one to two orders of magnitude on wide tables, which is what makes a full-corpus scan finishable inside a 30-day GDPR window rather than a 45-day one.

Testing & Compliance Verification #

Pattern coverage is verified against a payload matrix, not spot-checked. Each pattern ships with positive fixtures (canonical, delimiter-variant, and boundary-adjacent forms), negative fixtures (near-miss lookalikes that must not match — invoice numbers, log IDs, base64 fragments), and adversarial fixtures (ReDoS inputs run behind a timeout that must complete under budget). A regression trigger fails the build whenever a new library version changes the match set on the held-out corpus without an accompanying, reviewed changelog entry.

def test_ssn_pattern_rejects_invalid_ranges():
    lib = PatternLibrary(version="2026.07", patterns=[
        RegexPattern(pii_type="us_ssn",
                     pattern=r"(?!000|666|9\d{2})\d{3}-(?!00)\d{2}-(?!0000)\d{4}"),
    ])
    compiled = compile_library(lib)
    assert scan_chunk("SSN 078-05-1120", compiled)          # valid → matched
    assert not scan_chunk("ID 000-12-3456", compiled)       # invalid area → no match
    assert not scan_chunk("ref 900-45-6789", compiled)      # never-issued → no match

Held-out regulatory regions — a national identifier format from a jurisdiction not yet in production — are run against the matrix before their patterns are enabled, so an unsupported format fails closed rather than emitting bad matches on day one. Each verification run writes its pattern-library version and the fixture pass/fail summary to the same audit log, giving an examiner a reproducible link between a redaction decision and the tested pattern set that produced it.

Frequently Asked Questions #

Related #

• PII Extraction & Redaction Pipelines — the parent pipeline architecture this deterministic first pass feeds
• NLP-Based Entity Recognition — the probabilistic layer that resolves regex matches too ambiguous to auto-redact
• Structured vs Unstructured Data Sync — keeping extracted offsets aligned across CRM rows and free-text attachments
• Confidence Scoring Thresholds — how borderline match scores are routed between redaction and review
• Building a Regex Library for Email and SSN Detection — the concrete, ReDoS-safe patterns and checksum exclusions this schema wraps