Evaluating NER Precision and Recall for PII

Within the broader NLP-Based Entity Recognition stage of the PII Extraction & Redaction Pipelines architecture, a recognizer is only trustworthy once you can measure how often it misses personal data and how often it over-reaches. This page is the evaluation harness that turns a probabilistic model’s output into two defensible numbers per entity type — precision and recall — plus the F1 score that summarizes them. The engineers who need it are the privacy and data teams promoting a model into a Data Subject Request (DSR) pipeline: before a model is allowed to auto-redact anything, it must clear a documented recall floor, because a missed identifier is a live disclosure under the confidentiality duty of GDPR Art. 5(1)(f), while an over-eager span destroys data the subject was entitled to receive under GDPR Art. 15. The scorer below is pure Python — it consumes spans, not a live model — so the same code grades the base recognizer and a tuned one from fine-tuning spaCy for legal document PII extraction without change.

Evaluation is itself a control: a held-out labeled set, an exact-span matcher, per-label tallies, and the arithmetic that fails a release when recall on a high-severity identifier slips.

Prerequisites #

• Python 3.11+ — for tuple[...] / list[...] generic syntax and dataclasses/dict ordering used by the tallies.
• Pydantic v2 — pip install "pydantic>=2.6". The scorer validates every gold and predicted span at the boundary so a malformed offset cannot silently corrupt a metric.
• No model runtime. The scorer deliberately depends on nothing from spaCy or a transformer stack. It takes spans as plain data, which keeps evaluation reproducible in CI and lets you grade any recognizer that can emit (start, end, label) triples.
• A held-out labeled evaluation set — documents the model never saw during training, annotated by hand to the same guidelines the model was trained on. Stratify it by source_format and by entity type so a strong PERSON score cannot mask a failing SSN recall. Keep a slice of jurisdiction-specific documents in the set, mirroring how GDPR vs CCPA request taxonomies differ, so the recognizer is not silently overfit to one regime’s identifier formats.

Step-by-step implementation #

Step 1 — Represent gold and predicted spans #

A span is an immutable triple: character offsets plus a label. Modeling it in Pydantic v2 with frozen=True means a span can be used as a dictionary key and set member, which is exactly what exact-span matching needs. Labels are upper-cased on the way in so Ssn and SSN never fork a metric, and a model_validator rejects a zero-width or inverted span before it can distort a count.

from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator

VALID_LABELS = {"PERSON", "EMAIL", "SSN", "PHONE", "ADDRESS", "ACCOUNT_NO"}


class Span(BaseModel):
    """An offset-anchored, typed PII span used for both gold and predicted sets."""

    model_config = ConfigDict(frozen=True, extra="forbid")

    start: int = Field(ge=0)
    end: int = Field(gt=0)
    label: str = Field(min_length=1)

    @field_validator("label")
    @classmethod
    def _canonical(cls, v: str) -> str:
        v = v.upper()
        if v not in VALID_LABELS:
            raise ValueError(f"unknown label {v!r}; extend VALID_LABELS deliberately")
        return v

    @model_validator(mode="after")
    def _ordered(self) -> "Span":
        if self.end <= self.start:
            raise ValueError(f"end {self.end} must exceed start {self.start}")
        return self

    def key(self) -> tuple[int, int, str]:
        """The identity used for exact-span matching."""
        return (self.start, self.end, self.label)

Gold and predicted spans use the same model. That symmetry is what lets the matcher treat the two sets as comparable — the only difference is provenance (a human annotator versus the recognizer), not shape.

Step 2 — Match by exact span and tally TP / FP / FN per label #

Entity-level scoring is unforgiving on purpose: a predicted span counts as a true positive only when its start, its end, and its label all equal a gold span’s. Anything else is either a false positive (predicted but not gold) or a false negative (gold but not predicted). Working over sets of (start, end, label) keys keeps the match O(n) and completely deterministic, so the same evaluation set produces byte-identical counts on every run.

from collections import defaultdict
from dataclasses import dataclass, field


@dataclass
class LabelCounts:
    """Per-label confusion tallies for entity-level scoring."""

    tp: int = 0
    fp: int = 0
    fn: int = 0


def tally_exact(gold: list[Span], pred: list[Span]) -> dict[str, LabelCounts]:
    """Match predicted spans to gold spans by exact (start, end, label)."""
    counts: dict[str, LabelCounts] = defaultdict(LabelCounts)
    gold_keys = {s.key() for s in gold}
    pred_keys = {s.key() for s in pred}

    for key in pred_keys:
        label = key[2]
        if key in gold_keys:
            counts[label].tp += 1        # exact corroboration
        else:
            counts[label].fp += 1        # over-redaction candidate

    for key in gold_keys - pred_keys:
        counts[key[2]].fn += 1           # missed PII — the reportable class
    return counts

Deduplicating into sets before tallying also neutralizes the overlapping detections that window-overlap chunking produces upstream in the NLP-Based Entity Recognition stage: two identical predicted spans collapse to one key and cannot double-count as two true positives.

Step 3 — Compute precision, recall, and F1 into a typed result #

Precision answers “when the model flagged PII, how often was it right?”; recall answers “of the PII that was actually there, how much did the model catch?” For a DSR pipeline recall is the load-bearing number, because a false negative is undetected personal data. F1 is their harmonic mean, useful as a single sortable figure but never a substitute for reading recall directly. The EvalResult model carries the counts alongside the derived rates so a reviewer can reconstruct the arithmetic.

from pydantic import BaseModel, ConfigDict, Field


def _safe_ratio(numerator: int, denominator: int) -> float:
    """Return numerator/denominator, or 1.0 when the denominator is zero.

    A zero denominator means the class had no positives to find (recall) or
    none predicted (precision); scoring it as a perfect 1.0 avoids penalizing
    a label that simply did not occur in this document.
    """
    return 1.0 if denominator == 0 else numerator / denominator


class EvalResult(BaseModel):
    """Immutable per-label evaluation outcome."""

    model_config = ConfigDict(frozen=True, extra="forbid")

    label: str
    tp: int = Field(ge=0)
    fp: int = Field(ge=0)
    fn: int = Field(ge=0)
    precision: float = Field(ge=0.0, le=1.0)
    recall: float = Field(ge=0.0, le=1.0)
    f1: float = Field(ge=0.0, le=1.0)


def score(counts: dict[str, "LabelCounts"]) -> dict[str, EvalResult]:
    """Turn confusion tallies into per-label precision, recall, and F1."""
    results: dict[str, EvalResult] = {}
    for label, c in counts.items():
        precision = _safe_ratio(c.tp, c.tp + c.fp)
        recall = _safe_ratio(c.tp, c.tp + c.fn)
        denom = precision + recall
        f1 = 0.0 if denom == 0 else 2 * precision * recall / denom
        results[label] = EvalResult(
            label=label, tp=c.tp, fp=c.fp, fn=c.fn,
            precision=round(precision, 4), recall=round(recall, 4), f1=round(f1, 4),
        )
    return results

Step 4 — Add a micro-average and per-label recall gate #

A macro view (average of per-label F1s) treats a rare SSN as equal to a common PERSON; a micro view (pool all TP/FP/FN, then divide) reflects the document-level error rate. Report both, but gate on per-label recall for the high-severity identifiers, because those are the leaks a regulator will care about.

def micro_average(counts: dict[str, LabelCounts]) -> EvalResult:
    """Pool every label's tallies into one document-level score."""
    tp = sum(c.tp for c in counts.values())
    fp = sum(c.fp for c in counts.values())
    fn = sum(c.fn for c in counts.values())
    precision = _safe_ratio(tp, tp + fp)
    recall = _safe_ratio(tp, tp + fn)
    denom = precision + recall
    f1 = 0.0 if denom == 0 else 2 * precision * recall / denom
    return EvalResult(label="__micro__", tp=tp, fp=fp, fn=fn,
                      precision=round(precision, 4), recall=round(recall, 4),
                      f1=round(f1, 4))


HIGH_SEVERITY = {"SSN", "ACCOUNT_NO", "ADDRESS"}


def recall_gate(results: dict[str, EvalResult], floor: float = 0.95) -> list[str]:
    """Return the high-severity labels whose recall is below the floor."""
    return [r.label for r in results.values()
            if r.label in HIGH_SEVERITY and r.recall < floor]

The recall floor is a release control, not a metric: a non-empty list from recall_gate fails the build, exactly as the fine-tuning workflow’s per-label recall assertion does before it promotes a model.

Step 5 — Token-level scoring and false-negative error analysis #

Entity-level scoring reports a boundary-mismatched span as both a false positive and a false negative — brutal, but honest, because a partially-redacted SSN still leaks digits. Token-level scoring is softer: it credits the overlap, which is useful for diagnosing why a span missed (a trailing check digit, a split address line) rather than judging the release. Compute both; gate on entity-level, debug with token-level.

import re

_TOKEN = re.compile(r"\S+")


def token_labels(text: str, spans: list[Span]) -> dict[tuple[int, int], str]:
    """Assign each whitespace token the label of the span that fully covers it."""
    labeled: dict[tuple[int, int], str] = {}
    for m in _TOKEN.finditer(text):
        for sp in spans:
            if m.start() >= sp.start and m.end() <= sp.end:
                labeled[(m.start(), m.end())] = sp.label
                break
    return labeled


def false_negatives(gold: list[Span], pred: list[Span]) -> list[Span]:
    """Return the gold spans the recognizer missed — the review worklist."""
    pred_keys = {s.key() for s in pred}
    return sorted((s for s in gold if s.key() not in pred_keys),
                  key=lambda s: s.start)

The false_negatives list is the artifact an analyst actually works from. Each missed gold span is pulled with its surrounding context window and triaged: is it a systematic gap (the model never learned a jurisdiction’s national-ID format) or a one-off? Systematic misses feed the active-learning corpus that raises recall in the next model version.

Configuration reference #

Verification #

The scorer is deterministic, so its output can be asserted exactly. This fixture encodes a boundary mismatch on the SSN (predicted 10–18 versus gold 10–19) and a spurious EMAIL, then checks the resulting confusion and rates.

def test_scorer_end_to_end():
    gold = [
        Span(start=0, end=5, label="PERSON"),
        Span(start=10, end=19, label="SSN"),
        Span(start=30, end=45, label="ADDRESS"),
    ]
    pred = [
        Span(start=0, end=5, label="PERSON"),    # exact TP
        Span(start=10, end=18, label="SSN"),     # boundary miss: FP + FN
        Span(start=30, end=45, label="ADDRESS"), # exact TP
        Span(start=50, end=60, label="EMAIL"),   # spurious FP
    ]
    counts = tally_exact(gold, pred)
    results = score(counts)

    assert results["PERSON"].recall == 1.0
    assert results["SSN"].tp == 0 and results["SSN"].fn == 1
    assert results["SSN"].recall == 0.0        # a missed SSN is a hard fail

    micro = micro_average(counts)
    assert micro.tp == 2 and micro.fp == 2 and micro.fn == 1
    assert micro.precision == 0.5
    assert round(micro.recall, 3) == 0.667

    assert recall_gate(results) == ["SSN"]     # high-severity floor breached
    assert [s.label for s in false_negatives(gold, pred)] == ["SSN"]

Expect the boundary-mismatched SSN to appear in both the false-positive and false-negative tallies — the entity-level scorer refuses to credit a span that leaks a digit. recall_gate returns ["SSN"], which a CI job treats as a failed release.

Troubleshooting #

Boundary mismatches inflate both FP and FN. Root cause: the model consistently clips or extends a span by one character (a trailing SSN check digit, a leading title). Fix: confirm the pattern with false_negatives plus token_labels; if the overlap is high but the exact match fails, the model is close — retrain on the boundary cases rather than loosening the matcher, because exact-span is the honest control.

Label mapping makes a correct span look wrong. Root cause: the gold set uses US_SSN while the model emits SSN, so identical offsets never match. Fix: map both sides through one canonical taxonomy (the _canonical validator) before scoring, and fail loudly on any unmapped label so a rename cannot masquerade as a recall drop.

Tokenization drift shifts every offset. Root cause: gold spans were annotated against raw text but the model ran on NFC-normalized, control-stripped text, so every offset is off by the bytes normalization removed. Fix: normalize once at ingress and annotate against the same normalized text — the freeze-then-anchor discipline from the parent recognizer — so an offset always indexes the exact bytes both the annotator and the model saw.

A perfect micro-F1 hides a failing rare class. Root cause: PERSON and EMAIL dominate the pool, so a SSN recall of 0.4 barely moves the micro average. Fix: never gate on micro alone; enforce the per-label recall floor on HIGH_SEVERITY labels and read the macro average alongside it.

Empty evaluation set silently passes. Root cause: a mis-wired fixture yields no gold spans, so every ratio hits the zero-denominator branch and returns 1.0. Fix: assert a minimum gold-span count per label before trusting any score, so an empty or truncated held-out set fails the run instead of reporting a flawless model.

Frequently Asked Questions #

Related #

• NLP-Based Entity Recognition — the parent stage that emits the confidence-bearing spans this harness grades.
• Fine-Tuning spaCy for Legal Document PII Extraction — the model whose per-label recall floor this scorer enforces before promotion.
• Confidence Scoring & Thresholds for PII Detection — calibrating the scores whose operating point precision and recall measure.
• Regex Pattern Libraries for PII — the deterministic overlay that corroborates the spans a recognizer might otherwise miss.
• PII Extraction & Redaction Pipelines — the end-to-end architecture this evaluation control protects.