have been speculated to be learn by a machine. Outdated lodge invoices, financial institution statements, payslips, mortgage purposes, medical payments, customs kinds, courtroom filings, work orders.
Most firms use free instruments alongside paid APIs to attempt to convert these paperwork, and if you need structured output, APIs like Textract Structured run you as much as round $65 per 1k pages.
In the previous couple of years, although, a whole lot of new choices have appeared: smaller open-source imaginative and prescient fashions specialised for OCR, normal vision-language fashions, and doc parsing instruments like LlamaParse — altering what’s attainable and the associated fee thereof.
So it felt like a good time to do my very own experiment to check a few of these towards paperwork of various problem.
I scouted 93 docs that would act as a proxy for what firms use OCR for — handwritten notes, tables, monetary legacy docs, scanned invoices, receipts, charts, outdated newspapers, tax kinds — then ran all of them by way of 14 completely different engines.
The thought was to see how they dealt with two issues: textual content restoration and the power to protect helpful desk construction.
The primary query I needed answered: do you really want to pay $65 for 1k structured pages, or are you able to slice that right down to a fraction? And does the specialised fashions win over the final ones?
When doing experiments like this you at all times discover fairly just a few unusual issues, which I’ll cowl too. However to reply that essential query I’ll take you thru what OCR is (skip if not new), the economics, the check, among the outcomes, and what else this confirmed me.
Observe: I didn’t check full discipline extraction, since that’s more durable to match cleanly throughout fourteen engines.
TL;DR
There is no such thing as a single finest OCR engine. OCR is a routing downside.
For clear high-volume paperwork, Tesseract continues to be exhausting to beat as a result of it’s free and quick. For combined manufacturing paperwork, Gemini Flash was the most effective all-rounder on this check. For tables, Mistral OCR seemed just like the cheaper structured choice.
The smaller specialist fashions seemed good inside their consolation zone, however failed more durable at paperwork they hadn’t seen. So, for high-stakes or messy paperwork, it is smart to escalate to a bigger mannequin.
The primary takeaway is financial: don’t pay for costly structured OCR when the doc doesn’t want it. Classify your docs, check engines by yourself information, and route based mostly on price, accuracy, construction, and failure tolerance.
Benchmarks are helpful for discovery, however they won’t inform you what works in your paperwork.
Clarify the OCR house to me
OCR (Optical Character Recognition) is how a machine turns an image into machine readable textual content. Easy in precept, and for simpler docs largely solved, however more durable when issues grow to be extra human.
Simply to present you a fast overview, older OCR discovered textual content on a web page, sliced it into characters, and matched every one towards a library of identified shapes. Tesseract has performed this because the Eighties.
Fashionable OCR nonetheless (together with newer variations of Tesseract) often makes use of a neural community that appears on the complete web page directly and outputs the doc as textual content. So, in case your doc is a clear PDF or a high-quality scan in a regular font, OCR is usually a solved downside.
It stops being solved the second issues get messier: photographed receipts, handwritten notes, bizarre graphs and charts, dense monetary tables, or scanned tax kinds and mortgage purposes.
Corporations want this performed effectively for apparent causes, because it’s one thing each downstream system acts on. The higher OCR will get, the extra paperwork turns into one thing a system can cause over as an alternative of one thing a human has to learn by hand.
There’s additionally the truth that if we feed AI methods badly parsed docs, every thing after it is going to be exhausting to belief.
I’m all about economics, so this house caught my eye as soon as I noticed how a lot cash is being poured into it. The Clever Doc Processing (IDP) market is projected to develop to someplace between $20 billion and $90 billion by the early 2030s, relying on which analyst you ask.
In all probability pushed by firms paying $15–25 per bill in guide dealing with prices.
And since I keep near the tech world, I’ve watched a wave of specialised small OCR fashions ship over the previous yr (largely Chinese language), now being utilized by builders in all places.
Which raises the query I needed to check: can the small open-source fashions truly do the work the costly APIs cost for or ought to we truly look in the direction of the final imaginative and prescient fashions to deal with OCR too?
Skip the following part if you wish to perceive what this experiment confirmed. I’ve to undergo the check setup first.
The docs, the engines, and the metrics
This experiment comes down to a few questions: what engines we used, what docs we examined with, and the way we determined who received.
For the engines, I needed a lineup that coated all the alternatives I talked about, this meant: outdated and new, open and closed, native and cloud, specialised and normal.
Tesseract grew to become the classical selection. It runs domestically and could be very quick. Then I added two document-parsing pipelines: Docling and Marker. Docling is slower however runs on CPU, Marker is open-weight however needing a GPU to run quick, which exhibits up later within the value.
Then for the brand new wave of specialised open OCR fashions: GLM-OCR, PaddleOCR-VL, DeepSeek-OCR, and MinerU 2.5 (a borderline case, actually a pipeline with a VLM inside). I picked them off OpenDataLab’s OmniDocBench leaderboard, the place they ranked first, second, fourth, and fifth.
I hosted them on Modal and served the relevant ones with vLLM, batching to hurry issues up. I counted the scale-up time when measuring latency later.
I additionally added one closed purpose-built mannequin, Mistral OCR, which I’d heard good issues about.
On the open aspect, I used Qwen3-VL (8B, from Alibaba), additionally hosted on Modal with the remainder of the smaller fashions. I ought to flag that I gave it a plain transcription immediate slightly than the optimized serving setup it was designed for, so I could not have given it a good shot.
On the closed aspect, for the final fashions, I picked Gemini Flash 3.1 Lite (presently first on the IDP Leaderboard, the western counterpart constructed on OmniDocBench v1.5) and Claude Sonnet 4.6, at sixth.
For the cloud doc providers: LlamaParse and AWS Textract, in each its textual content and structured kinds. Structured Textract can do way over I requested of it. I solely examined its textual content accuracy throughout the board and its desk extraction towards eight of the opposite engines.
Let’s flip to the paperwork. I picked seventeen doc varieties that have been both straightforward, medium, or exhausting. Ninety-three information in all.
Simple was the stuff OCR largely solved years in the past: clear invoices and receipts. Medium got here largely from the OmniAI OCR Benchmark dataset: financial institution statements, medical notes, photographed receipts, delivery paperwork, tax kinds.
Laborious was chosen when issues turned harder: charts, kinds, handwritten notes, weirdly scanned monetary tables, authorized papers, newspapers, and outdated legacy studies.
Some docs have been actually fairly troublesome, such because the legacy scanned docs you see beneath, and this was simply because I used to be curious if some may truly do it effectively.
A few of these photos got here with gold floor reality and a few didn’t, and the bottom reality I did have wasn’t at all times constant, some information labeled accurately, some not, which is why we must always briefly cowl the metrics too.
Since each engine emits completely different markup, the standard scoring didn’t fairly match. One may choose Precision and Recall for a case like this.
Precision appears to be like at how lots of the OCR output’s phrases truly match within the GT whereas Recall measures what number of instances every GT phrase was captured.
Precision would punish engines that emit markdown construction the GT doesn’t include, moreover the GT typically skipped labels totally which might punish the engine unfairly. Recall would measure the phrases however punish the frequency.
So, I added on a 3rd metric referred to as Protection. I simply needed to measure how a lot of the bottom reality exhibits up someplace within the engine’s output. It isn’t good, however it tells me whether or not an engine caught most of what mattered, with out penalizing it for gaps that have been the bottom reality’s fault slightly than the engine’s.
For the paperwork with no gold floor reality in any respect, I fell again on an LLM choose, with Gemini 3 Professional as the bottom mannequin and anybody who’s used one is aware of that is fickle enterprise.
What this experiment confirmed
We mapped each doc towards the Protection metric to construct a scatter chart, and tracked latency on a separate chart. The factor a generalized chart can’t inform you although is that the engines failed in several methods.
The bubble graph confirmed that the majority engines fall someplace within the center high, with two outliers on either side of it.
Gemini Flash and Textract Textual content did very effectively throughout the board with some edge circumstances. The specialised fashions all fell beneath the final fashions and specialised APIs. Sonnet carried out the very best but additionally with a steeper price ticket.
This may occasionally not have been a shock because the check set was extremely uncommon. A number of the specialised fashions could not have seen lots of them. Moreover, this check was on English paperwork and most of those smaller fashions have Chinese language origin.
After we additionally mapped latency, among the fashions turned out to be very gradual, however once more most wound up someplace within the center.
The outliers right here have been: Tesseract, Claude Sonnet 4.6, and Docling. Tesseract was extremely quick in comparison with all different engines. It must be your go-to for simpler paperwork.
These graphs generalise throughout all of the paperwork, however I did separate the outcomes based mostly on the sort and problem stage.
To start out with the straightforward docs. On invoices, each engine did effectively, Tesseract particularly. Receipts knocked everybody down slightly.
The one outlier was Docling, which struggled throughout a whole lot of the classes, even the straightforward ones.
Once I seemed into the Docling failures I discovered issues like Ifjointreturn as an alternative of “joint return,” and worse, strings like Metropolis,wrostffielfouaveaoreignadresalcomletacesb. DeepSeek additionally missed key particulars right here like bill quantity and date, which is why its quantity sits low.
The identical sample holds within the medium class, although that’s the place PaddleOCR began degrading on particular varieties: financial institution statements, delivery, tax kinds. Tax kinds have been exhausting for everybody, however PaddleOCR and Docling wound up on the backside.
Textract was the most effective engine on a whole lot of the medium varieties, together with Claude Sonnet 4.6 and Mistral OCR.
On the more durable varieties, Gemini Flash began rising, beating Textract on kinds and handwritten notes, matching it elsewhere. It did remarkably effectively in all places. Tesseract and Docling failed exhausting on handwritten, and kinds have been powerful for them too.
Nearly all of the specialised fashions didn’t pull by way of on these more durable docs besides on monetary tables, the place they held about even.
For the docs with no floor reality (newspapers, authorized, studies, some scanned legacy paperwork) we used an LLM choose. These are genuinely exhausting, so it’s no shock nearly everybody failed on the studies and newspapers.
Besides Gemini Flash that did fairly effectively in all places. Mistral OCR additionally did effectively for newspapers. Gemini Flash received in all places with the choose, although we used Gemini Professional because the choose so take that with a grain of salt (however I did double test myself).
Earlier than rounding off: I additionally ran 8 engines towards Textract Structured to see how they did on monetary tables, extracting an HTML desk. I used Textract Structured’s output as the bottom reality for TEDS (Tree Edit Distance Similarity) and scored Claude Sonnet 4.6, LlamaParse, Mistral OCR, Gemini Flash, Marker, MinerU, DeepSeek-OCR, and Docling towards it.
Mistral OCR, and LlamaParse, and Sonnet did very effectively whereas being less expensive. I additionally ran it by way of an LLM choose, and the winners have been the identical three (even earlier than Textract Structured) although I’d need to construct that check higher earlier than I absolutely belief it.
Now, let’s discuss what it prices to scale this up, and what would make sense the place.
When does what make sense
Let’s run by way of what it prices to scale up with these engines, after which based mostly on these docs what you’ll selected the place.
First, the price of utilizing these engines fluctuate wildly, as you noticed earlier than. Generally it helps to see the associated fee not only for one doc, however hundreds as much as 1,000,000.
We’re self-hosting on Modal, so these prices come from precise utilization there. You may run domestically, however my pc wouldn’t enable it and I didn’t need to attempt it.
When you have been to only use one engine that handles each straightforward and exhausting paperwork, I might suppose you wound up with a much bigger invoice than essential. Utilizing Textract Structured for any paperwork that aren’t wanted would hand you a invoice of $6.5k per 100k docs.
I do marvel what number of firms go the straightforward manner right here and choose the costly choices for straightforward in addition to exhausting docs and depart some huge cash on the desk.
The important thing concept to take with you right here is that there’s no single finest engine for each use case, it is dependent upon doc kind, privateness, desk construction, failure tolerance, price, and so forth.
For the docs we’ve right here, Gemini Flash 3.1-Lite is a transparent winner. This one was right from wanting on the leaderboards. Mistral OCR did effectively on structured tables whereas staying low-cost. Claude Sonnet 4.6 did very effectively too, however it’s very gradual and costly comparatively.
Docling is so very gradual on my laptop computer. I’m positive there are methods to hurry it up, however it additionally failed in ways in which make it inherently unstable (nonetheless a small check although).
The specialised OCR fashions have been a little bit of a headache, particularly on English docs; I noticed output errors in Chinese language that I’ll cowl in a bit, so I ponder if that’s a part of it.
Textract is a secure selection, however structured buys you nearly no extra textual content accuracy so if you happen to’re paying that steep markup for structured output, ensure you truly use it. I’m guessing it’s a reasonably good enterprise mannequin for them.
So, generally for this very small check: for clear, high-volume print, simply use Tesseract. For normal heterogeneous manufacturing, go Gemini Flash. For a cost-floor with desk construction, check Mistral OCR. For top-stakes docs, path to Sonnet or a bigger mannequin.
Since everybody did effectively in several methods you’ll need to contact me for specifics but when you have to go non-public it might be price it to have a look at fine-tuning a mannequin in your docs. Or, use a small specialised mannequin and escalate on failures.
Let me simply shortly discuss some issues that stood out after doing this experiment.
Different stuff I ought to point out
A handful of issues surfaced from this which might be price pulling out on their very own.
First, if you wish to perceive how a mannequin or engine will do in your docs, the one manner is to check on these docs, you’ll be able to’t depend on benchmarks to inform you. This was the primary perception this confirmed. OCR usefulness relies upon by yourself doc combine, layouts, languages, scans, tables, handwriting, and failure tolerance.
Don’t pay for construction if you happen to don’t want it. I ponder what number of are utilizing sure APIs or fashions for a cause they’ll’t justify. Map the associated fee to grasp what you’re dropping by not utilizing the proper engine for the paperwork.
The specialist fashions, as talked about earlier than, have sharp boundaries. That is apparent, they are often glorious inside their coaching distribution however fail outdoors of it. That is the place the final fashions will win.
If you wish to fine-tune it might assist, however provided that the stream is secure as it should additionally fail whether it is always launched new doc lessons.
Lastly, the failure modes advised us greater than the averages.
PaddleOCR had repetition loops, column-merging, fallback into Chinese language textbook template textual content like 书名:___ repeated tons of of instances. Whereas Docling has character errors, word-merging, and column misalignment all stacking collectively.
DeepSeek OCR has chart blindness and empty outputs on some docs. Tesseract did superb on clear docs (as talked about) however failed on photograph/handwriting altogether outputting rubbish.
Caveats to take into account
Earlier than we spherical up, let me cowl how this check is in the end imperfect by naming the problems within the GT, the metrics used, and the pattern dimension.
I coated this in one of many part above, however the floor reality differs between paperwork relying on the dataset the place they have been discovered. Typically tokenization artifacts could make right OCR look worse than it’s.
Most engines have completely different codecs, some return plain textual content, some markdown, some HTML/wealthy markdown and it’s exhausting to generalize throughout all.
We’re utilizing Protection, after which additionally another metrics, however these aren’t good. Protection received’t cost the engine if it outputs an excessive amount of textual content or the construction of it’s off. Although I did discover that for the engines that failed, they did so initially or mid-way by way of slightly than on the finish.
This implies it’s helpful for rating however not an ideal technique to rating.
LLM judges are usually not impartial reality: I’ve coated this previously, however they’re biased and really immediate delicate.
Then I simply have to say that this check is fascinating however not that large, the pattern dimension is manner too small to make use of this as a factual examine. However, I don’t absolutely belief these metrics nor the choose so it was the one manner for me to have the ability to double test the outcomes alone with out this turning right into a yr lengthy undertaking.
So, this check is beneficial for path and getting a way of what works, however for getting a way of your use case, you have to run it by way of together with your particular docs.
Lastly, latency and reproducibility is unstable. Serverless chilly begins make timing noisy, and API fashions can silently change over time, so precise replica is difficult.
Like at all times with these articles, it takes fairly a bit to do an experiment like this however I don’t simply do it for content material, I do it as a result of I’m genuinely curious.
What it appears to be like like although is that OCR appears to be a routing downside, and maybe an analysis downside. Classify your docs and run them by way of a number of engines, then attempt to construct an honest router and validator in your pipeline to escalate failures after which log the prices.
If you have to get the total outcomes from this experiment otherwise you need me to run it by way of your docs, get in contact.
You may observe my writing on Medium, my web site or join with me by way of LinkedIn.
❤
All datasets used on this benchmark are publicly accessible and sourced from HuggingFace. Licenses embrace MIT, CC-BY-4.0, and fair-use frameworks (UCSF Business Paperwork Library) masking analysis, scholarship, and training. No supply paperwork are reproduced — datasets have been used solely as analysis inputs to measure OCR engine efficiency.
