Choosing Masking vs Tokenization vs Deletion

A DSR redaction stage fails a regulator not by doing nothing, but by doing the plausible wrong thing — tokenizing a value in response to an erasure request because it kept referential integrity, or exporting an access response that still names a third party. The engineers who hit this are building the transform step of a DSR pipeline: detection has already located and classified the personal data, and now a single function must map each value to exactly one technique that is legally correct for the request in hand. Within the broader Redaction Strategies for DSR Fulfillment approach — itself part of the PII Extraction & Redaction Pipelines framework — this page is the concrete, runnable decision function: given a request type and a data role, return a technique, apply it, and emit a receipt. The confidence gating that decides which values reach this function automatically versus via human review is covered in the sibling Confidence Scoring & Thresholds stage.

The core rule is small and unforgiving: an erasure under GDPR Art. 17 must be irreversible, so it can never resolve to tokenization, which by the Art. 4(5) definition of pseudonymisation is reversible with additional information the controller holds; while an access export under Art. 15(4) must keep the subject’s own data and remove third parties’. The decision function encodes exactly that.

Prerequisites #

The implementation targets Python 3.10+ (for match statements and X | Y union syntax) and depends on:

• pydantic (v2) — runtime validation of the decision inputs and outputs. The code uses v2 idioms (model_config = ConfigDict(...), field_validator, model_validate); v1 patterns (class Config, @validator, .dict()) will not work.
• hashlib / hmac (stdlib) — one-way commitments and irreversible masking with no retained mapping.
• A tokenization vault client — any service that stores an original value under a surrogate and returns the surrogate. The examples treat it as an injected object with put(namespace, value) -> str; adapt for your provider. Keys are managed per NIST SP 800-57.

This page assumes each value arrives already classified by request type and data role — the upstream reconciliation that assigns those tags across databases and documents is the job of Structured vs Unstructured Data Sync.

Step-by-step implementation #

Step 1 — Model the decision inputs and the technique enum #

Both axes are enums so the decision function can match on them exhaustively, and the technique is a closed set so no caller can invent an unlisted transform.

from enum import Enum


class RequestType(str, Enum):
    ACCESS = "access"
    PORTABILITY = "portability"
    ERASURE = "erasure"


class DataRole(str, Enum):
    SUBJECT_OWN = "subject_own"    # the requester's own personal data
    THIRD_PARTY = "third_party"    # another identifiable person's data
    LINKAGE_KEY = "linkage_key"    # cross-system join key, not disclosed
    RETAINED_UNDER_HOLD = "hold"   # erasure owed but retention required


class Technique(str, Enum):
    HARD_DELETE = "hard_delete"
    NULLIFY = "nullify"
    TOKENIZE = "tokenize"
    MASK = "mask"                  # irreversible fixed-token / hash
    FORMAT_PRESERVING = "fpm"      # irreversible, structure-preserving

Step 2 — Implement the four transforms #

Each transform returns a redacted value; the reversible one also returns a vault surrogate. Crucially, hard_delete and mask retain no mapping, so they are irreversible by construction.

import hashlib


REDACTION_TOKEN = "[REDACTED]"


def irreversible_mask(value: str) -> str:
    """Replace a value with a fixed token. No mapping is retained."""
    return REDACTION_TOKEN


def hashed_mask(value: str, *, salt: bytes) -> str:
    """One-way SHA-256 mask; the digest cannot recover the input."""
    return hashlib.sha256(salt + value.encode()).hexdigest()


def format_preserving_mask(value: str) -> str:
    """Keep length and the last four characters; irreversible otherwise."""
    if len(value) <= 4:
        return "*" * len(value)
    return "*" * (len(value) - 4) + value[-4:]


def tokenize(value: str, *, namespace: str, vault) -> str:
    """Store the original in a vault and return a reversible surrogate.

    Reversal is possible only for the vault holder — this is Art. 4(5)
    pseudonymisation, NOT erasure.
    """
    return vault.put(namespace, value)


def hard_delete(value: str) -> None:
    """Destroy the value with no recovery path. Returns nothing."""
    return None

Step 3 — Model the decision output with Pydantic v2 #

RedactionDecision binds a technique to the reason it was chosen and enforces the one invariant that matters: an erasure request may never carry a reversible technique. The field_validator makes that a hard failure at construction time rather than a silent leak.

from pydantic import BaseModel, ConfigDict, model_validator

REVERSIBLE = {Technique.TOKENIZE}


class RedactionDecision(BaseModel):
    """The chosen technique for one value, with its legal justification."""

    model_config = ConfigDict(frozen=True, extra="forbid")

    locator: str
    request_type: RequestType
    data_role: DataRole
    technique: Technique
    legal_basis: str  # e.g. "GDPR Art. 17"

    @model_validator(mode="after")
    def _erasure_must_be_irreversible(self) -> "RedactionDecision":
        if self.request_type is RequestType.ERASURE and self.technique in REVERSIBLE:
            raise ValueError(
                f"erasure request {self.locator} resolved to reversible "
                f"technique {self.technique.value}; Art. 17 requires irreversibility"
            )
        return self

Step 4 — The decision function #

select_technique maps (request_type, data_role) to exactly one technique. It reads top-down: erasure first (it dominates every role), then the access/export role branch.

def select_technique(request_type: RequestType, data_role: DataRole) -> Technique:
    """Return the single legally-correct technique for a classified value."""
    # Erasure dominates role: Art. 17 requires irreversible removal.
    if request_type is RequestType.ERASURE:
        if data_role is DataRole.RETAINED_UNDER_HOLD:
            return Technique.NULLIFY      # cannot delete; make unusable, log basis
        return Technique.HARD_DELETE

    # Access / portability export: protect the rights of others, Art. 15(4).
    match data_role:
        case DataRole.THIRD_PARTY:
            return Technique.MASK          # suppress another person's data
        case DataRole.LINKAGE_KEY:
            return Technique.TOKENIZE      # reversible pseudonymisation, Art. 4(5)
        case DataRole.SUBJECT_OWN:
            return Technique.FORMAT_PRESERVING  # readable, subject is entitled
        case _:
            return Technique.MASK          # fail safe: over-redact, never leak


_BASIS = {
    Technique.HARD_DELETE: "GDPR Art. 17",
    Technique.NULLIFY: "GDPR Art. 17 (retention-limited)",
    Technique.TOKENIZE: "GDPR Art. 4(5)",
    Technique.MASK: "GDPR Art. 15(4)",
    Technique.FORMAT_PRESERVING: "GDPR Art. 15(4)",
}


def decide(locator: str, request_type: RequestType, data_role: DataRole) -> RedactionDecision:
    """Build a validated decision; raises if the invariant is violated."""
    technique = select_technique(request_type, data_role)
    return RedactionDecision(
        locator=locator,
        request_type=request_type,
        data_role=data_role,
        technique=technique,
        legal_basis=_BASIS[technique],
    )

The fail-safe case _ defaults to irreversible masking: on an ambiguous role the pipeline over-redacts rather than risk disclosing a third party, because under-redaction on an access export is a reportable breach while over-redaction is a recoverable service issue.

Configuration reference #

Verification #

Two assertions are non-negotiable: an erasure request can never resolve to a reversible technique, and an access export masks third-party data while preserving the subject’s own. The RedactionDecision validator also guarantees the invariant is enforced even if a future edit to select_technique regresses.

import pytest
from pydantic import ValidationError


def test_erasure_is_never_reversible():
    for role in DataRole:
        decision = decide(f"db.users#1:{role.value}", RequestType.ERASURE, role)
        assert decision.technique not in REVERSIBLE
    # A held value nullifies rather than deletes, but is still irreversible.
    held = decide("db.users#1", RequestType.ERASURE, DataRole.RETAINED_UNDER_HOLD)
    assert held.technique is Technique.NULLIFY


def test_access_masks_third_party_keeps_subject():
    third = decide("ticket#9.body", RequestType.ACCESS, DataRole.THIRD_PARTY)
    own = decide("ticket#9.body", RequestType.ACCESS, DataRole.SUBJECT_OWN)
    assert third.technique is Technique.MASK
    assert own.technique is Technique.FORMAT_PRESERVING


def test_model_rejects_illegal_combination():
    # Force the invariant: erasure + tokenize must fail construction.
    with pytest.raises(ValidationError):
        RedactionDecision(
            locator="db.users#1", request_type=RequestType.ERASURE,
            data_role=DataRole.LINKAGE_KEY, technique=Technique.TOKENIZE,
            legal_basis="GDPR Art. 4(5)",
        )

Round-tripping through RedactionDecision.model_validate(...) on a persisted decision confirms that a stored erasure record can never deserialize into a reversible technique — the validator runs on load as well as on construction, so a tampered ledger row is caught on read.

Troubleshooting #

Erasure silently tokenized to preserve foreign keys — Root cause: an engineer wired the delete path to the vault to avoid breaking referential integrity. Fix: the RedactionDecision validator rejects (ERASURE, TOKENIZE) at construction, so surface the failure and re-model the foreign key to tolerate a tombstone rather than reversing the erasure. Reversibility defeats Art. 17.

Third-party data leaks into an access export — Root cause: values classified by field type instead of data role, so a third party’s name in a shared column was treated as the subject’s. Fix: ensure each value carries a DataRole, and confirm select_technique routes THIRD_PARTY to MASK; the fail-safe case _ should also mask unclassified values.

Format-preserving mask still identifies the subject — Root cause: fpm_visible_tail too high, or the field is short enough that four visible characters are quasi-identifying. Fix: lower fpm_visible_tail, and for short high-cardinality fields prefer full masking over format preservation.

Tokens differ for the same value across systems — Root cause: divergent token_namespace values between the structured and unstructured detection paths. Fix: scope the namespace to the subject, not the source system, so one value maps to one surrogate and linkage reconciles.

Held value deleted when the hold was still active — Root cause: hold_action overridden to HARD_DELETE, or the hold flag not propagated to the classifier. Fix: keep hold_action at NULLIFY, verify the RETAINED_UNDER_HOLD role is set upstream, and only schedule the deferred hard_delete when the hold is released.

Frequently Asked Questions #

Related #

• Redaction Strategies for DSR Fulfillment — the parent approach mapping request type and data role to a redaction technique.
• PII Extraction & Redaction Pipelines — the framework whose detection stages classify the values this function transforms.
• Confidence Scoring & Thresholds — the gate deciding which matches are redacted automatically versus sent to human review.
• Structured vs Unstructured Data Sync — reconciling a single subject across systems so each value carries the right data-role tag.
• Audit Logging & Compliance Evidence — where each decision and its verification result are recorded for accountability.