You have a folder of 200,000 PDFs. Invoices, contracts, packing slips, scanned forms someone faxed in 2019. Your team types the numbers out of them by hand. Someone pitched you an AI extraction tool, ran a demo on five clean documents, hit 95 percent accuracy, and called it solved.
Then you point it at the real folder. The 95 percent that worked was the easy 95 percent. The remaining 5 percent is where the money is — the malformed line items, the handwritten amendments, the two-page tables that split across a page break. And 95 percent accuracy on a financial field is not a feature. It's a 1-in-20 chance of a wrong dollar amount flowing into your books, undetected, forever.
Why "95 percent accuracy" is a lie you tell investors
Accuracy as a single number is meaningless until you say accuracy of what, measured how, on which documents.
A vendor reports 95 percent. Ask: is that character-level OCR accuracy, field-level extraction accuracy, or document-level (every field right)? They are wildly different numbers. A document with 20 fields at 97 percent per-field accuracy has a 0.97^20 = 54 percent chance of being fully correct. Just over a coin flip. That is the number that matters for "can I trust this document without a human looking at it," and the vendor never quotes it.
Then ask: accuracy on what distribution? Demos run on clean, native-text PDFs from one template. Production is scans of scans, phone photos taken at an angle, a vendor who redesigned their invoice last quarter, documents in three languages, and the occasional page that is upside down. The accuracy curve falls off a cliff the moment the input stops looking like the demo set.
The honest framing is per-field accuracy at a measured confidence threshold, on a held-out test set that matches your real document mix. Everything else is marketing.
The pipeline, stage by stage
A production document pipeline is not one model. It's a sequence of stages, each with its own failure mode and its own measurement.
Stage 1 — ingestion and OCR with layout
Raw documents arrive in every format. The first job is turning pixels into text with position. Plain OCR gives you a wall of words. Layout-aware OCR gives you words plus bounding boxes plus reading order plus table structure. That spatial information is what lets a later stage know that "1,240.00" sits in the column headed "Total" on the row labeled "Invoice Amount" — not in the "Tax" column three pixels over.
Skip layout and you spend the rest of the pipeline guessing at relationships the document already encoded visually. Tables are where this matters most: a line-item table reconstructed from flat text is a reconstruction, and reconstructions drift.
Stage 2 — extraction
Now you pull structured fields out of the laid-out text. This is where a vision-language model or a layout-aware extraction model earns its cost — handling the variation that rule-based parsers choke on. A regex for invoice numbers works until you onboard the vendor who writes them as INV / 2024 / Q3 / 0042.
The non-negotiable design choice: every extracted field carries a confidence score and a provenance pointer back to the source region. Not just total: 1240.00. Instead total: 1240.00, confidence: 0.91, source: page 1, bbox [x,y,w,h]. Without provenance you cannot audit, cannot debug, and cannot show a reviewer where the number came from when they need to check it.
Stage 3 — validation
Extraction gives you candidate values. Validation decides whether to believe them. This stage is deterministic code, not a model, and it catches the failures the model cannot see in itself.
Validation rules encode what you know about the domain:
- Line items must sum to the subtotal. Subtotal plus tax must equal total. If the math doesn't close, something was extracted wrong — flag the document even if every field's confidence was high.
- Dates must be plausible (an invoice dated 1899 is an OCR error reading 1999).
- Cross-field constraints: a US ZIP is five digits; a tax ID matches a known format; a currency code is real.
- Reference checks: does this vendor exist in your system? Does this PO number match an open order?
Validation is where most of your real-world accuracy comes from. The model proposes; deterministic rules dispose. A field that passes both a high confidence score and every cross-check is one you can post without a human. A field that fails either is one you can't.
Stage 4 — human-in-the-loop for the low-confidence tail
Some documents will never clear the bar automatically, and pretending otherwise is how wrong numbers reach production. The pipeline routes any document with a low-confidence field, a failed validation rule, or an unrecognized layout into a review queue.
The review UI is the part teams under-build. It should show the document image, the extracted value overlaid on the source region, and a single keystroke to confirm or correct. A reviewer should clear a flagged document in seconds, not minutes — because they're checking three fields the system was unsure about, not re-keying twenty it got right.
Every correction is labeled training data. Feed corrections back, and the low-confidence tail shrinks over months. The review queue should get quieter as a function of volume, not louder.
Measuring accuracy so the number means something
You cannot improve what you don't measure, and you cannot measure extraction without a labeled ground-truth set. Build one: a few hundred documents, hand-labeled, spanning the real distribution — clean and messy, every template, every language, the upside-down page.
Track per-field accuracy and document-level accuracy against that set on every pipeline change. Track the auto-clear rate: what fraction of documents pass confidence plus validation with no human touch. That number, times your volume, is the labor you actually eliminated — the only ROI metric that survives contact with finance.
Watch the confidence calibration. A well-calibrated pipeline is one where "0.9 confidence" empirically means "right 90 percent of the time." If your high-confidence fields are wrong more often than their score implies, the threshold is lying to you, and you're auto-posting errors.
The failure modes nobody demos
Silent miscalibration. The model is confidently wrong — high confidence, incorrect value — and validation has no rule that catches it. This is the dangerous one, because it skips review and lands in your books. Defense: more cross-field validation, and a continuous sample of auto-cleared documents pulled for spot-audit.
Template drift. A high-volume vendor changes their invoice layout. Accuracy on that vendor quietly collapses while your aggregate number barely moves. Defense: per-source accuracy monitoring, not just a global average.
The new-document-type ambush. Someone starts feeding contracts into a pipeline tuned for invoices. It produces output, because models always produce output. The output is garbage. Defense: a document-type classifier up front that rejects what the pipeline wasn't built for.
Partial-page and split-table errors. A table that spans a page break gets read as two unrelated tables, and a line item vanishes. Defense: layout-aware stitching and a line-item-count sanity check against any stated total count.
What fixed looks like
A pipeline where documents flow in, get OCR'd with layout, extracted with per-field confidence and provenance, validated against deterministic domain rules, and split cleanly into two streams: auto-cleared documents you can trust without looking, and a small flagged queue a human resolves in seconds.
You have a labeled test set and a dashboard showing per-field accuracy, document-level accuracy, auto-clear rate, and per-source trends. When accuracy moves, you see which field, which source, which day. Corrections feed back as training data. The auto-clear rate climbs and the review queue shrinks as volume grows.
The honest number isn't "95 percent." It's "84 percent of documents auto-clear with audited 99.4 percent field accuracy, and the other 16 percent get a 12-second human check." That sentence is something you can put in front of an auditor.
This is for you if
You process documents at a volume where manual entry is a real line item — tens of thousands of documents a month or more — and where a wrong field has a downstream cost in dollars, compliance, or trust. Financial operations, insurance, logistics, healthcare intake, procurement.
A production pipeline — layout-aware OCR, extraction with confidence and provenance, a deterministic validation layer, a review UI, and an evaluation harness — runs $50k+, scaling toward $150k+ with document-type variety, multi-language input, and integration into your existing systems of record.
This is not for you if you have a few hundred documents a month on one clean template. A scripted parser and a spreadsheet will beat any AI pipeline on cost and reliability, and you should build that instead. It's also not for you if you want a tool that's right every time with no human in the loop — that tool does not exist for messy real-world documents, and anyone who sells it to you is selling the demo, not the folder.