PII Masking
TS-06-06Help providers transform confidential data into usable, secure datasets by removing identifiers and possibly replacing them with synthetic but realistic fictional information.
About This Task
Legal help organizations, courts, and justice system actors handle large volumes of confidential data: case notes, intake records, court filings, scanned documents, images, and communications. This data holds potential value for research, service improvement, AI training, and policy analysis. But before it can be used for any of those purposes, personally identifying information must be removed to protect the privacy and safety of clients and staff.
The PII Masking task is a system that scans confidential records and either redacts or replaces the relevant identifiers. It takes in documents in any common format: PDF (digital or scanned), images, Word, HTML, or plain text. It identifies PII fields like names, dates of birth, addresses, phone numbers, Social Security numbers, and case numbers. Then it produces a clean output where sensitive strings are removed, replaced with placeholders (like [NAME]), or substituted with consistent synthetic identifiers (e.g., "Martha Pizzoli" becomes "Client A07" throughout a record set).
The system doesn't just black things out. A good PII masker produces three outputs: the masked document itself, a machine-readable manifest of what was found and how it was handled (entity type, position, confidence, method), and risk flags for anything the system is unsure about ("possible missed PII," "low OCR quality," "handwriting detected").
Who this serves: Legal aid data managers, court IT teams, researchers building datasets for justice AI, program managers sharing data with funders or partners, and any organization that needs to safely share or analyze confidential records.
What Success Looks Like
The data has been de-identified with high confidence. No identifying details remain in the output. The resulting synthetic data preserves accuracy, consistency, and utility for its intended purpose. A legal worker can still read the masked document and understand the legal situation, deadlines, and procedural requirements. The system catches the things that must be caught (SSN, full name + address combinations, minor names) and leaves alone the things that must remain readable (hearing dates, court names, filing deadlines, rent amounts).
What Counts as PII?
Our working group asked subject-matter experts at legal help teams to classify common legal document fields as Clear PII, Possible PII, or Not PII. The results establish a three-tier classification that any PII masking system should use as a starting configuration.
High risk: Definite PII (mask by default)
SSN (full), bank and financial account numbers, driver's license and state ID (full), passport number. Phone number, email. Date of birth. Signature. Home and mailing address, property address. Insurance policy number. Criminal history details (often highly sensitive and identifying in context).
Medium risk: Possible or indirect PII (mask depends on profile and context)
Last 3 digits of SSN/ID. Case and court number (can be identifying via public record linkage). Initials, age. Income amount, financial assets. Medical condition, pregnancy status. Employer, school, and daycare names. VIN, license plate, and professional license numbers. Place of birth, immigration identifiers (e.g., A-number). Address components alone (ZIP code, census tract) depending on purpose.
Low risk: Usually not PII (generally do not mask)
Hearing date and time, document date. Rent amount, generic financial totals. Judge name, insurance company name. Laws, citations, and legal labels.
Note: many teams distinguish PII (identifiers) from sensitive information (medical, pregnancy, income, assets). Your profile may choose to treat "sensitive" as maskable even when it is not a direct identifier.
Technical Starting Points
Microsoft Presidio is the strongest open-source starting point. It provides detection and anonymization modules with extensible recognizers, and includes an evaluation guide grounded in precision and recall. It handles multiple languages and can be extended with custom recognizers for legal-domain patterns (case numbers, court-specific formats).
Google Cloud Sensitive Data Protection (Cloud DLP) provides classification, redaction, and de-identification transformations for text and images. Designed for pipeline integration. Useful if your organization is already on Google Cloud.
Hybrid detection pipeline is the recommended architecture: rules and regex for structured identifiers (phones, emails, SSNs, case numbers), NER models for names and locations, and domain-specific recognizers for court case formats and local address conventions. OCR-first branch for scans, text-first branch for native PDFs.
NIST SP 800-122 provides the foundational guidance on identifying PII and choosing appropriate protections, with emphasis on tailoring safeguards by impact level.
OCR quality is a first-order driver of PII performance in scanned legal documents. Evaluate your OCR and PII masking end-to-end, not in isolation. Low OCR confidence should trigger human review, not automatic sharing.
Sharing Contexts Matter
PII masking is not one thing. The level of masking depends on who will see the output. Define your sharing contexts and configure masking profiles accordingly.
Internal collaboration: staff reviewing cases together. Can keep more identifiers. Masking may be limited to removing SSNs and banking details.
External expert review: sharing with outside attorneys or subject-matter experts. Mask names, addresses, and contact information. Keep legal details, dates, and procedural content.
Public dataset release: sharing data for research or AI training. Strictest masking. All direct identifiers removed. Quasi-identifiers (age, ZIP, employer) evaluated for re-identification risk.
Building a Training and Evaluation Dataset
To evaluate PII masking tools or build your own, you need labeled data. Start with 200-500 documents across 6-10 document types: notices (addresses + names), pleadings (names + case numbers), intake notes (phones, emails, DOBs), letters (free text, harder cases), and scans with stamps and signatures.
Labeling workflow: Extract text from documents with layout preserved. Label PII spans (start position, end position, entity type). Tag severity tier per entity. Mark "must-preserve" content that cannot be destroyed (deadlines, court location, legal claims). Tools: Label Studio (open source), Prodigy (paid, faster UX), or simple Airtable for tiny pilots.
Quality control: Double-label 10-20% of records. Resolve disagreements. Update the labeling guide when ambiguity appears.
Split deliberately: 60% training, 20% validation, 20% benchmark (locked). The benchmark set must be frozen. All future improvements are measured against it without touching it.
The Legal Help Commons has an inventory of 500+ legal document types across multiple states and issue areas that can serve as a starting corpus for PII masking evaluation.
Quality Standards
What does good performance look like for this task? Use these criteria to evaluate systems.
This rubric was developed by the Stanford Legal Design Lab working with a multi-state cohort to evaluate PII masking tools on real legal documents. Each dimension is scored 1-3 (1 = Not OK, 2 = OK with cleanup, 3 = Good). A Hard Fail on any dimension overrides the score.
Hard Fail Conditions
Any of the following means the output is Not OK regardless of dimension scores:
A HIGH/DEFINITE PII item is still visible (SSN, DOB, address, ID, phone, email, signature). You can still find PII by searching or copy-pasting the output (text layer not cleaned). Masking is reversible: annotations or layers can be removed to reveal the original content.
Dimension 1: Catches PII
Did the system find the PII that's there?
Score 3 (Good): Catches all HIGH/DEFINITE PII throughout the document, including repeated instances. Catches key MEDIUM PII as expected by the masking profile.
Score 2 (OK with cleanup): Catches most HIGH PII but misses some repeated instances (e.g., name appears in header and body, caught only in body). MEDIUM PII caught inconsistently.
Score 1 (Not OK): Misses HIGH PII in a primary location. Multiple MEDIUM PII items missed. Pattern: the system finds the easy ones and misses the hard ones.
Dimension 2: Masks the Right Things
Did it mask what it should and leave what it shouldn't?
Score 3 (Good): Masks all HIGH PII. Does NOT mask LOW/NOT PII items (dates, rent totals, judge name, citations). The output retains legal meaning and operational value.
Score 2 (OK with cleanup): Masks HIGH PII correctly but also masks some items that should have been preserved (over-redaction). A legal worker could still use the document with some effort.
Score 1 (Not OK): Either under-masks (misses HIGH PII) or over-masks to the point of destroying the document's usefulness. Entire paragraphs blacked out because of one identifier.
Dimension 3: Masking is Real
Is the PII truly gone, not just hidden?
Score 3 (Good): PII cannot be found by searching or copy-pasting the output text. Works across export formats (PDF, PNG, TXT) without leakage. Text layer is clean.
Score 2 (OK with cleanup): Visual masking is correct but the text layer still contains PII (e.g., black rectangle covers the text visually but copy-paste reveals it). Fixable with additional processing.
Score 1 (Not OK): PII is trivially recoverable. Annotation-based masking that can be removed. Hidden text layers preserved.
Dimension 4: Avoids Over-masking
Did it avoid hiding important non-PII?
Score 3 (Good): All legal meaning preserved. Dates, amounts, court names, deadlines, legal claims, and key events remain readable. A legal worker can use this document for its intended purpose.
Score 2 (OK with cleanup): Some non-PII content obscured but the core legal meaning is recoverable. A few dates or amounts incorrectly masked.
Score 1 (Not OK): Wholesale redaction of paragraphs or sections. Legal meaning destroyed. A worker cannot determine what the case is about, when the hearing is, or what the deadlines are.
Dimension 5: Consistent and Reviewable
Is it easy to verify and fix?
Score 3 (Good): Masks all instances of the same identifier throughout the document (headers, footers, tables, exhibits). Produces a clear log or manifest of what was masked. Manual correction is straightforward.
Score 2 (OK with cleanup): Mostly consistent but misses some instances in unusual locations (footers, table cells, exhibit labels). Log exists but is incomplete.
Score 1 (Not OK): Inconsistent masking across the document. Same name masked in one place, visible in another. No manifest or log. Manual review requires reading the entire document.
Overall Result
NOT OK: Any Hard Fail, or 2 or more dimensions scored "1."
OK WITH CLEANUP: No Hard Fail. Mostly "2" scores. A trained reviewer can fix the remaining issues in a reasonable time.
GOOD: No Hard Fail. Mostly "3" scores. Output is ready for its intended sharing context.
Evaluation Protocol
Step 1: Build a Benchmark Set
Minimum viable benchmark: 300 samples, split evenly between text-first (clean digital PDFs, copied text) and scan-first (OCR required, varying quality). Stratify by document type (notices, pleadings, intake narratives, letters), language mix, and presence of tables, stamps, and signatures.
Step 2: Create Gold Labels
Human-labeled PII spans with start position, end position, and entity type. Each entity tagged with severity tier (high/medium/low). "Must-preserve" annotations marking content that cannot be destroyed: deadlines, court names, legal claims, hearing dates, filing requirements.
Step 3: Run Evaluation
Compare system output spans to gold labels. Measure recall, precision, and F-beta by entity type. Report the doc-level "any high-risk false negative" rate: what percentage of documents have at least one missed high-risk PII item. Have a human reviewer score a sample of outputs on a "utility" measure: can a legal worker still use this document?
Include stress tests: low OCR quality pages, handwritten fragments, non-English names and addresses, PII appearing in margins, headers, footers, and table cells.
Step 4: Document Failures
Any missed high-risk PII is an automatic fail for that sample. Any over-redaction that deletes required operational content (deadlines, court location, filing requirements) is also a fail. This reflects how real systems should treat the tradeoff: optimize for high recall on dangerous items, then manage precision through policy configuration and post-processing review.
Key Benchmarks
PII Recall by entity type: The primary safety metric. Measure separately for high-risk entities (SSN, DOB+Name, minor name, full address, phone/email) and medium-risk entities. Target: 99%+ recall on high-risk, 95%+ on medium-risk.
False Negative severity rate: Percentage of documents with any missed high-risk PII. This is the number that matters most for trust. Target: under 1%.
Over-redaction rate: How often non-PII content is removed. Measured as precision plus human "utility" ratings. Target: over 90% precision, with no more than 5% of documents rated "unusable" by a legal worker.
End-to-end throughput and cost: Documents per minute at required quality. Important when PII masking is a pipeline component feeding into a larger data-sharing or AI training workflow.
Robustness: Performance across low OCR quality, messy formatting, multilingual content, and handwriting. Report performance separately for each condition so organizations know where the system breaks.
Using This Rubric
For researchers: The five dimensions map directly to annotation categories. The Hard Fail conditions define your binary classification for safety-critical evaluation. Report recall and false-negative rates by entity type. The benchmark set methodology (300 samples, stratified, frozen) gives you a reproducible evaluation framework.
For builders: Use the rubric during development as a checklist of failure modes. Test against each Hard Fail condition before deployment. Implement the hybrid detection pipeline (rules + NER + domain-specific recognizers) to cover different PII patterns. Treat OCR quality as a first-class variable in your testing.
For funders: Require that any funded PII masking project report its false-negative rate on high-risk PII as a primary metric. Ask for the "any high-risk FN" rate per document, not just aggregate precision/recall. Require a human utility score alongside automated metrics.
Other Administration & Strategy Tasks
Explore related tasks in the same workflow category.
