OCR Extraction Benchmark: DeepSeek API vs NuExtract-tiny vs Tesseract

Every freelancer in Quebec has the same tax-time ritual: a stack of receipts, hours of manual data entry, and the nagging feeling that somewhere, a deduction slipped through the cracks.

I set out to automate this. The goal: feed a PDF receipt through Tesseract, extract structured data (vendor, date, subtotal, TPS, TVQ, total, payment method), and dump it straight into an expense report. The question was which extraction method works best for real Quebec receipts — the kind with French text, QC-specific tax lines, and the occasional handwritten total.

The Contenders

Three approaches, spanning the spectrum from fully local to cloud API:

DeepSeek is the heavy lifter — a frontier LLM with deep reasoning. NuExtract-tiny is a 0.5B parameter model purpose-built for structured extraction, running locally via llama-cpp-python. Tesseract gives us the raw OCR text as a baseline.

The Method

I built a benchmark framework (benchmark/) with three layers:

1. Fixtures — 18 Synthetic Receipts

Each fixture is a realistic Quebec receipt as Tesseract would see it: item lines, subtotals, TPS/TVQ lines (at 5% and 9.975%), payment method, and footer text. Every receipt is paired with ground-truth JSON of what the extraction should return.

The receipts span 15 professions and 18 stores — from a contractor buying lumber at RONA to a lawyer expensing a client dinner at Le Pois Penché:

Construction   → RONA, Canac
TI/Consulting  → Starbucks, Best Buy
Juridique      → Le Pois Penché, Stationnement Indigo
Santé          → Bureau en Gros
Coiffure       → Sally Beauty
Créatif        → L.L. Lozeau
Transport      → Shell
Restauration   → Marché Atwater
Immobilier     → L'Express
Comptable      → Bureau en Gros
Mécanique      → Napa
Fitness        → Sportium
Tuteur         → Librairie Renaud-Bray
Entretien      → Canadian Tire
Coaching       → Tim Hortons

2. Model Runners — Abstract Interface

Both runners implement the same ModelRunner protocol:

class ModelRunner(ABC):
    @abstractmethod
    def extract(self, ocr_text: str, fixture_id: str) -> RunnerResult: ...
    @abstractmethod
    def is_available(self) -> bool: ...
    def normalize(self, data: Dict[str, Any]) -> Dict[str, Any]: ...

DeepSeekRunner wraps the existing DeepSeekClient — sends OCR text, parses the structured response, normalizes field names (e.g. company → fournisseur).

NuExtractRunner loads a 350MB GGUF model locally and prompts it with a JSON schema template:

<|input|>
### Template:
{"fournisseur": {"nom": ""}, "reçu": {"date": "", "total": 0.0, ...}}
### Text:
[OCR text here]
<|output|>

Temperature is pinned to 0.0 for deterministic output. The model is 0.5B parameters — it runs comfortably on a laptop CPU in a few seconds.

3. Evaluator — ±2¢ Tolerance

The evaluator compares each extracted field against ground truth. Numeric fields (subtotal, taxes, total) get a ±$0.02 tolerance. String fields (vendor, date, payment method) require exact case-insensitive match. Each of the 7 fields is scored per receipt, then aggregated:

FixtureScore:
  fixture_id, store, profession
  total_fields: 7
  matched_fields: N
  accuracy: N/7
  success: True if all 7 match

The summary rolls up per-model stats plus drill-down by field and by profession.

Results (To Be Run)

The benchmark is wired and tested (17 unit tests ✅) but hasn't produced live numbers yet — NuExtract-tiny needs the GGUF model downloaded and llama-cpp-python installed. Once that's done, the runner auto-detects available models and produces a formatted report:

$ python3 -m benchmark.run --verbose

🔍 Searching for available models...
  ✅ DeepSeek API — available
  ✅ NuExtract-tiny — available
  ⏭️  Tesseract only — available

📋 Fixtures: 18 receipts across 15 professions

==================================================
🚀 DeepSeek API
==================================================
  [1/18] RONA (Construction/Rénovation)    ✅ accuracy=100% (7/7)  time=1.2s
  ...

  → 120/126 fields correct (95.2%) · 15/18 full receipts · 24.3s total

==================================================
🚀 NuExtract-tiny
==================================================
  ...

  → 108/126 fields correct (85.7%) · 11/18 full receipts · 8.7s total

==================================================
📊 BENCHMARK REPORT
==================================================
  Model                     Accuracy  Success   Receipts  Avg time
  ─────────────────────────────────────────────────────────────
  DeepSeek API                95.2%      83%    18/15    1.35s
  NuExtract-tiny              85.7%      61%    18/11    0.48s

These are projected figures — the actual benchmark fill be filled in once models run.

Architecture

The benchmark was designed for extensibility:

benchmark/
├── __init__.py
├── fixtures.py              # 18 synthetic Quebec receipts + ground truth
├── model_runners/
│   ├── base.py              # Abstract ModelRunner + RunnerResult
│   ├── deepseek_runner.py   # DeepSeek API via existing client
│   └── nuextract_runner.py  # NuExtract-tiny via llama-cpp-python
├── evaluator.py             # Field-by-field comparison ±2¢
└── run.py                   # CLI entry point + report

Adding a new model runner means subclassing ModelRunner and implementing extract() and is_available(). The evaluator and report pipeline stay the same.

What I Expect to Learn

Even without live results, the design surfaces some clear trade-offs:

1. DeepSeek will win on accuracy — frontier LLMs are absurdly good at structured extraction. But every receipt costs fractions of a cent and requires internet.

2. NuExtract-tiny will win on speed + privacy — sub-second CPU inference, fully offline, 350MB on disk. The question is whether 85% accuracy is enough, and which fields it struggles on (likely taxes, which require arithmetic context).

3. Tesseract alone is the floor — useful as a baseline to measure how much value the extraction model adds. Raw OCR text is gibberish for expense reports.

Cost Analysis (Projected)

For a solo freelancer processing 50-100 receipts per month, the cost of DeepSeek is negligible. For a firm processing thousands, NuExtract's zero marginal cost and privacy advantage becomes decisive — especially with client-confidential receipts.

What's Next

1. Download NuExtract-1.5-tiny GGUF and run the full benchmark
2. Add a Tesseract-only runner as baseline
3. Test on real scanned receipts (not synthetic)
4. Build the hybrid pipeline: Tesseract → extraction model → expense report CSV

Built with Python, PyQt6, and too many Quebec receipts.