The text beside the image,
measured end to end

01Detection

What an ensemble catches, what it wrongly catches, and what it costs.

The cost–quality front, in full

Cost ∥ is the slowest member, which is what a parallel run pays; Cost Σ is the sum, which is what a serial one pays. The paper reports only the first. Information-weighted precision scales each masked token by how poorly its context predicts it. This search covers singles, pairs and triples only, so the recommended 13-detector ensemble lies outside it and exceeds the “maximum” row on every corpus — CARDIO:DE 0.998 against 0.992; TAB 0.961 against 0.925; OntoNotes 0.801 against 0.747; Enron 0.998 against 0.997. Enron's four rows are re-scored against the corrected gold described under the next table. The selection was not re-run: these triples were chosen by the 1,743-source sweep as it scored on the superseded gold, so a fresh sweep could pick different members. The rows say what these ensembles achieve now, not that they would still win. Every column, the weighted one included, is re-scored under the sweep's own weight model (unigram:enron).

The recommended ensemble under every combining rule

The 13 detectors under each combining rule, both error rates together. Union is the maximum-sensitivity configuration of this ensemble: no other rule over the same members can find more, and every leakage figure on this page describes that release. The rows beneath it price the alternative — agreement buys specificity and pays for it in sensitivity. Precision mixes the two denominators and is reported beside them, never optimised for.

The 13 detectors

• gliner:urchade/gliner_multi-v2.1
• gliner:urchade/gliner_multi_pii-v1
• hf:Davlan/xlm-roberta-large-ner-hrl
• hf:StanfordAIMI/stanford-deidentifier-base
• hf:obi/deid_roberta_i2b2
• llm:GaleneAI/Magistral-Small-2509-FP8-Dynamic
• llm:Microsoft/Phi-4-mini-instruct
• llm:RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8
• llm:RedHatAI/gemma-4-31B-it-FP8-block
• llm:google/gemma-4-E4B-it
• llm:ibm-granite/granite-4.1-3b
• presidio
• privacy_tagger

Two further large language models were run as single detectors and excluded from the recommended ensemble: preliminary runs made them one to two orders of magnitude slower per document without improving detection.

02By language

A pooled score describes no language in particular. Scripts differ in how many subword tokens they need, annotation differs in what it marks, and the detectors were not trained evenly across the four.

The 13-detector union throughout, so every row describes the release the attacks were run against. Only OntoNotes is multilingual; German, and the English of TAB and Enron, are whole corpora. Sensitivity and specificity are read against their own denominators — identifier tokens and everything else — so neither can be judged without the other.

OntoNotes, split three ways

What the combining rule costs, language by language. The gap between sensitivity and person sensitivity is the point: the three languages differ far more on the full identifier set than on people, and people are what the attacks recover.

03Utility

Frozen models read all three conditions. The question is not whether the text changed but whether the task survives, so every comparison is paired per document and carries the original-text score beside it. Entity agreement was measured on all four corpora; the clinical information-extraction and section tasks exist only for the clinical letters.

Entity agreement, all four corpora

Mean ± one standard deviation over documents, every document of each corpus. Column A is 1.000 everywhere by construction and is not a result: ner_agreement scores the frozen recogniser against its own output on the unmodified text, so A is the definition of perfect agreement, not a measurement of it. It is printed because the paired design requires the original-text score beside every condition. Read B and C down the two rules of one corpus, never across corpora: the genres differ in how much of a document is a name.

CARDIO:DE, every task

Mean ± one standard deviation over documents, on CARDIO:DE, which is the only corpus carrying the clinical tasks. ner_agreement is 1.000 on A by construction: it scores the frozen recogniser against its own output on the original text. medication_ie:in_narrative is retained here but excluded from the paper's conclusions, because the frozen model is near chance on the original text and so cannot measure a loss.

The paired tests in full

Wilcoxon signed-rank on paired documents of CARDIO:DE, Benjamini–Hochberg corrected within task families. Effect is the rank-biserial correlation. A rank-biserial of −1.000 means every document lost, not merely the average.

04Leakage

Three attacks against the released text, all scored over the full denominator with collisions counted rather than dropped, because an attacker has no ground truth to drop them with.

Candidate ranking, every cell

Ten candidates per query, so chance is Rank-1 0.100 and Rank-5 0.500. The recommended 13-detector release is listed first for each corpus; the 15-detector rows are the earlier run over the full pool, kept because they show the finding does not depend on the ensemble. The two attacker rows are the same surrogate release read by an adversary who knows it is pseudonymised: one simply told so, one replacing name-like spans it finds with a public name list. Neither changes what the defender published.

Frequency matching

Aligned counts identities recovered by matching frequencies, which is the attack. Read in clear counts names the detector missed, which the release simply printed: the attacker recovers those by reading, not by inferring, and the two must never be added into one rate. The oracle prior is the corpus's own distribution, which no real adversary holds. ρ is Spearman's correlation between the surrogate and real frequency orders.

Prior quality: the adversary-knowledge sweep

The prior-quality sweep Bindschaedler et al. prescribe: vary what the adversary knows, along size and along age, and report accuracy per cell. Across all 32 cells, spanning three orders of magnitude of prior size, frequency alignment recovers 1 identity in total. Better knowledge buys the attack nothing here, which is the point of running the sweep rather than asserting a single prior is representative.

Abstention

Abstention follows Narayanan and Shmatikov's eccentricity rule: the attacker declines when (max − max₂)/σ falls below φ, computed from its own scores and never from the truth. c@1 credits a warranted abstention at the rate the attacker achieves where it does answer. Above φ = 0 the attack abstains on almost everything, which is the attack telling us it cannot tell which of its answers to trust.

Context and learned linkage

Five-fold cross-validation, mean ± one standard deviation over folds. Folds withhold labels, never candidates: a held-out person stays in the complete candidate list with every distractor, so the attacker is never told that a query has a match. Ceiling on A is the same attack against unmodified text, which is what gives the protected numbers a scale.

05Exposure

A corpus-level token rate is the least informative way to state what a pseudonymisation run left behind. Data-protection officers release documents and cases, and one unchanged mention puts a person back in the clear however many others were replaced.

Route one: the released patch sets, which is what the reader of the release sees, rather than the union of detected spans. The two differ because the engine writes one span of any overlapping group and skips the rest — 16.5% of resolved detections on CARDIO:DE and 46.0% on Enron, so where a short span wins over a longer one the rest of the name is published. Those are the clipped mentions, and no measure computed on detected spans can see them. The two person denominators are also kept apart: someone appearing in forty Enron messages is one person and forty entity–document pairs. The paper's leakage table uses the pair denominator, but on the other route — the next table.

The same exposure, measured on the detected spans

Route two: the union of detected spans. A mention counts as exposed only when no detector covered it, so a name that was found and then clipped by the overlap rule is counted as protected here and as exposed above. Both questions are legitimate and the answers are not interchangeable: this route measures the detectors, the route above measures the release. The entity–document-pair column is the paper's Exp. column — the risk that a person is still named in a given document after processing, not the share of people, which is the column beside it.

Where the residue is

The same accounting over every identifier class the conditions replace, not people alone, and on route one. Dates, quantities and miscellaneous spans are passed through by design and are in neither denominator.

06Stability

A keyed pseudonym is only useful if it is stable, and only safe if it is not too stable. These are the ways the mapping fails.

A collision gives two distinct people one surrogate, fragmentation gives one person several inside a document, and drift changes a person's surrogate between documents. All three are properties of the keyed mapping in condition B; typed placeholders merge every entity of a type by design and have none of them. The paper reports three of these numbers; this is the grid they come from.

07About these numbers

Every experiment is driven by a config plus a seed, and each result records the library versions, model ids and commit hash that produced it. The page is generated from one JSON export of those result files by docs/build_site.py, so it cannot drift from the measurements the way a hand-maintained table does.

Text, and only text. The motivation is multimodal — a radiograph travels with its report, a cardiology study with its discharge letter, and releasing either half requires both to be protected — but the study itself is not. Nothing here is an image experiment: no scan is defaced, no burned-in pixel text is read, and no claim is made about what image de-identification costs. Every measurement on this page is made on written documents, which is the half of such a release that identifier replacement has to carry.

Corpora are not redistributed here. Several are bound by a data-use agreement, licensed, or contain real personal data. This export carries aggregates only: no corpus text, no surface form, no identity. CARDIO:DE in particular is single-user under its agreement with Heidelberg, and its gold names were inserted into an already de-identified release, which makes its name distribution synthetic and its clinical realism partial.

Enron is used deliberately rather than silently: it is the only public e-mail corpus with real names in a natural frequency distribution, and excluding it would have protected nobody who is not already in every copy of the corpus.

What the attacks do not show. Frequency alignment failing on these data does not establish safety against other adversaries. Linkage is measurable only where identity crosses documents, so TAB and OntoNotes figures understate rather than bound their risk. Candidate ranking presents ten names, which bounds an adversary holding a shortlist, not one searching a population. Every number describes one release under one key.