OCR/Data Extractor

This rubric was developed by the Stanford Legal Design Lab's Data Extraction working group to evaluate OCR and data extraction tools on real legal documents. Each dimension is scored 1-3 (1 = Not OK, 2 = OK with cleanup, 3 = Good). A Hard Fail on any dimension overrides the overall score.

Hard Fail Conditions

Any of the following means the output is Not OK regardless of dimension scores:

The output is not usable text or data: garbled, missing most text, or wrong file processed. Key fields are consistently wrong or swapped (e.g., case number becomes hearing date). The system gives no way to spot uncertainty: no confidence flags, no review view. Processing creates a privacy or security problem: uploads stored unnecessarily, no access control, unclear data handling.

Dimension 1: Input Coverage

Can the system ingest common legal document formats?

Score 3 (Good): Ingests PNG, JPG, TIF, and PDF (including scanned PDFs) reliably. Handles multi-page documents without loss.

Score 2 (OK with cleanup): Handles most common formats but fails on some (e.g., scanned PDFs with mixed orientation, TIF files). Multi-page documents mostly work but occasionally drop pages.

Score 1 (Not OK): Cannot reliably process scanned PDFs or images. Fails on multi-page documents. Limited to a single input format.

Dimension 2: Accuracy of Extraction

Did the system correctly read the text and numbers, and extract the right values?

Score 3 (Good): Key fields are read correctly: case number, party names, dates, amounts, addresses. Values are assigned to the correct field labels (no "field swaps"). Errors are rare and minor — they don't change the meaning.

Score 2 (OK with cleanup): Most key fields are correct but some have character-level errors that affect meaning (e.g., "1,250"readas"1,250" read as " 1,250"readas"I,25O"). Some field swaps occur. A reviewer can fix errors in reasonable time.

Score 1 (Not OK): Frequent errors in key fields. Field swaps are common. Character-level mistakes change legal meaning (wrong dates, wrong amounts, wrong case numbers).

Dimension 3: Coverage Across Document Types

Does the system work on the real-world variety of legal documents?

Score 3 (Good): Works on scanned forms, letters, notes, screenshots, and contracts. Handles both clean prints and messy scans (skew, stamps, faint text). Captures most required fields across formats, not just one "happy path" form.

Score 2 (OK with cleanup): Works well on clean documents but degrades on messy scans. Handles the primary document type well but struggles with variations.

Score 1 (Not OK): Only works on one specific document format. Fails on scanned documents, messy formatting, or any deviation from the expected layout.

Dimension 4: Structure and Layout

Does the system preserve layout and correctly pair labels with values?

Score 3 (Good): Form fields are paired correctly (label stays with its value). Tables are captured with correct rows and columns. Multi-column pages are read in the right order.

Score 2 (OK with cleanup): Most label-value pairs are correct but some are mismatched. Tables mostly work but some cells are in the wrong position. Minor column-reading order issues.

Score 1 (Not OK): Labels and values are frequently mismatched. Tables are scrambled. Multi-column pages produce garbled output.

Dimension 5: Handwriting

Can the system read handwriting when it appears?

Score 3 (Good): Handwritten names, notes, and dates are captured accurately enough to be useful. When unsure, the system flags handwriting fields for human review rather than guessing confidently.

Score 2 (OK with cleanup): Handwriting is partially captured. Some fields are correct, others are wrong or missing. Flagging is inconsistent.

Score 1 (Not OK): Handwriting is not captured or is consistently wrong. No flagging of uncertain handwriting fields. System treats handwriting with the same confidence as printed text.

Dimension 6: Data Cleanliness

Is the extracted data clean and consistently formatted for downstream use?

Score 3 (Good): Dates are normalized (e.g., MM/DD/YYYY) and not ambiguous. Names and addresses are cleanly parsed without random symbols or noise. Currency amounts are consistent (e.g., $1,250.00, not "I,25O.OO").

Score 2 (OK with cleanup): Mostly clean but some normalization issues. A developer can write cleanup rules to handle the remaining inconsistencies.

Score 1 (Not OK): Output contains frequent noise, mixed formats, and unparseable values. Requires extensive manual cleanup before downstream use.

Dimension 7: Error Transparency and Review

Can a human quickly verify and fix mistakes?

Score 3 (Good): Flags low-confidence fields (blurry, conflicting, partial). Provides a review interface showing highlighted fields on the source document page. Allows quick correction and export of revised structured data.

Score 2 (OK with cleanup): Some confidence flagging but incomplete. Review requires switching between source document and extracted data manually. Corrections are possible but cumbersome.

Score 1 (Not OK): No confidence flagging. No way to compare extraction against the source document. Corrections require re-processing.

Dimension 8: Output and Integration

Is the output easy to import into a case management system or database?

Score 3 (Good): Produces structured output (CSV or JSON) with stable, predictable field names. Field mapping is clear and consistent across runs. Minimal manual rework before import.

Score 2 (OK with cleanup): Structured output exists but field names or formats vary between runs. Some mapping work required before import.

Score 1 (Not OK): Output is unstructured or inconsistently structured. Significant manual work required to map fields for import.

Dimension 9: Speed and Reliability

Is the system fast and predictable enough for real legal workflows?

Score 3 (Good): Finishes in under 1 minute for typical documents. Does not crash or time out on common files. Produces consistent results when rerun on the same document.

Score 2 (OK with cleanup): Usually finishes in reasonable time but occasionally slow or inconsistent. Rare crashes or timeouts.

Score 1 (Not OK): Frequently slow, crashes, or produces different results on the same document.

Overall Result

NOT OK: Any Hard Fail, or 2 or more dimensions scored "1."

OK WITH CLEANUP: No Hard Fail. Mostly "2" scores. A trained reviewer can fix the remaining issues in reasonable time.

GOOD: No Hard Fail. Mostly "3" scores. Output is ready for import into downstream systems with minimal human review.

Evaluation Protocol

Step 1: Build a Test Corpus

Select 50-100 documents across the document types your system will process. Include clean digital PDFs, messy scans, documents with tables, documents with handwriting, and documents with stamps or signatures. For each document type, pick 5-10 key fields to test (see the field list below).

Step 2: Create Gold-Standard Labels

Have a human expert manually extract the correct values for each key field in every test document. Record the exact text, position, and expected format. This is your ground truth. Double-label 10-20% of documents to check consistency.

Step 3: Run Extraction and Score

Run your system on the full test corpus. Compare extracted values to gold-standard labels. Measure field-level accuracy: precision, recall, and F1 score for each field type. Measure the Hard Fail rate: what percentage of documents trigger any Hard Fail condition.

Step 4: Stress Tests

Run the same evaluation on deliberately difficult inputs: low-resolution scans (150 DPI or lower), rotated or skewed pages, documents with heavy stamps or watermarks, handwritten sections, multi-column layouts, and non-English content. Report accuracy separately for each stress condition.

Step 5: Utility Review

Have a legal worker review 20+ extraction outputs and rate each on a simple scale: "Can I use this data for its intended purpose without re-reading the source document?" This catches problems that automated metrics miss: field swaps that are technically "accurate" for each field but meaningless in combination, or over-extraction of irrelevant data that buries the important fields.

Key Fields to Test

Pick 5-10 per document type: case number, court name and location, party names (plaintiff and defendant), address and property address, hearing date and time, document date, rent amount or amount claimed, deadlines (e.g., "within 5 days"), checkbox status (selected or unselected) if forms are present, table cell contents (fees, ledger rows) if tables are present.

Using This Rubric

For researchers: The 9 dimensions provide a standard evaluation framework. Use the Hard Fail conditions as binary classifiers for safety-critical assessment. Report field-level precision and recall by entity type. The stress test protocol gives you a reproducible methodology for comparing systems.

For builders: Use the rubric during development as a checklist. Test against each Hard Fail condition first. Invest in OCR quality (Stage 2 of the pipeline) before optimizing LLM prompts (Stage 3). The two-pass extract-then-validate pattern consistently improves accuracy. Build the human review interface early: even the best systems need a human in the loop.

For funders: Require field-level accuracy metrics on a documented test corpus, not just demo-quality outputs on cherry-picked documents. Ask for the Hard Fail rate. Ask for stress test results on messy scans and handwriting. Ask for the cost model: what does this cost per document at the volume the organization actually processes?