From Extraction to Accuracy: Evaluating Extracted Invoice Data with LLM-as-a-Judge

Last Updated on March 11, 2026 by Editorial Team Author(s): Krishnan Srinivasan Originally published on Towards AI. (A practical, end-to-end guide to building a ground-truth-based evaluation pipeline, complete with synthetic data and runnable SQL on Snowflake) In the earlier parts of this Agentic AI series, we explored how AI systems can reason, use tools, retrieve knowledge, and orchestrate complex workflows. But as AI systems become more capable and autonomous, an equally important question starts to take center stage. How do we evaluate whether the AI actually performed correctly? Whether the task is handled by a single model, an AI pipeline, or a multi-agent workflow, the outcome still needs to be measured against something objective. In other words, capability without evaluation is incomplete. Imagine you have built an AI pipeline that reads supplier invoices and pulls out three key fields: 📄 Invoice ID, 💰 Total Amount, 🏢 Supplier Name. The extraction runs. The data lands in your database. But now comes the hard question: How do you know if what was extracted is actually correct? Manually checking thousands of documents does not scale. Rule-based validation is brittle. Simple string comparisons fail when formatting differences appear. This is where LLM-as-a-Judge comes in ⚖️ Instead of writing fragile validation logic or manually auditing records, we can use a language model to act as an evaluator. The model compares what the AI pipeline extracted against ground truth (human-verified values) and produces a structured evaluation with: an accuracy score a match classification a short explanation for the decision What is LLM-as-a-Judge? LLM-as-a-Judge is an evaluation pattern where you use a large language model not to do the primary task, but to grade the output of another model (or pipeline) doing the primary task. It has become popular in production AI systems because: It scales: you can evaluate thousands of records without a human reviewer for every one.It is flexible: it can handle fuzzy matches, formatting differences, and partial answers that a simple string comparison would flag as wrong.It is auditable: you get a score AND a human-readable explanation for every decision. Without ground truth, LLM-as-a-Judge can only check plausibility. i.e whether the extracted value looks reasonable against the source document. With ground truth (known-correct values), it becomes a true accuracy measurement. In this post, we walk through a complete end-to-end implementation: creating the evaluation tables, generating synthetic invoice data with varied extraction quality, building the LLM-as-a-Judge function in Snowflake Cortex, running the evaluation pipeline, and analysing the results. The outcome is a closed-loop evaluation framework where AI outputs are continuously measured, monitored, and improved. This is an essential capability as Agentic AI systems become more deeply embedded in enterprise workflows. The full pipeline has three layers: End-to-End LLM-as-a-Judge Evaluation Process This diagram illustrates how AI-extracted invoice data is evaluated end-to-end using LLM-as-a-Judge inside Snowflake Cortex. AI-generated extractions and human-verified ground truth are stored as structured tables and fed into Cortex, where a deterministic LLM acts as an impartial evaluator. Each field is scored independently, producing explainable, auditable results that flow into analytics and dashboards. The outcome is a closed-loop, enterprise-grade evaluation pipeline that makes document AI accuracy measurable, actionable, and continuously improvable, entirely within Snowflake. Implementation Steps Initial Setup: We will begin by creating a dedicated database and schema for this walkthrough. We will now proceed with the implementation steps. Step 1 — Create the Tables We need three tables. The extractions table holds what your AI pipeline pulled from each document. The ground truth table holds the correct answers, verified by a human. The results table is where the judge writes its scores. 1a. Extractions Table: This is the output of your existing invoice extraction pipeline. For this tutorial, we will populate it with synthetic data that has a deliberate mix of correct, partially-correct, and wrong extractions. 1b. Ground Truth Table: This is your reference dataset containing the known-correct values. In a real project, a small team of reviewers annotates a representative sample. Even 50 to 100 verified invoices gives you a meaningful benchmark. 1c. Evaluation Results Table: The judge will write one row per field per invoice. Each row captures the extracted value, the ground truth, the score (0.0 to 1.0), a match type category, and a plain-English explanation. Step 2: Insert Synthetic data for Ground Truth Rather than waiting for a real batch of invoices, we will create 10 synthetic invoice documents with a deliberate variety of extraction outcomes. This lets you see the full range of judge scores in one run. For simplicity, let us assume we have already processed the invoices and have stored the data in a strctured format. The table below summarizes the error patterns we have intentionally introduced: Ground Truth Insert: Insert the verified correct values. In production these come from your annotation tool or a manually curated spreadsheet. Step 3: Extractions Insert Let us now insert the AI-extracted values. Note: In a real pipeline, these values would typically be extracted into structured tables using Document AI capabilities such as AI_EXTRACT. For this walkthrough, we assume the extraction step has already been completed and the results are available in a structured table. The focus of this article is on LLM-as-a-Judge evaluation, not the extraction mechanics. If you would like to understand how document fields can be extracted from invoices using Snowflake Cortex, please refer to my earlier blog on AI_EXTRACT, where the extraction workflow is explained in detail. For the purpose of this walkthrough, a few extraction errors have been intentionally introduced in order to verify how the LLM-as-a-Judge evaluation behaves across different scenarios. Notice how some documents have subtle bugs: DOC006 grabs the pre-tax subtotal instead of the total, DOC007 confuses the buyer and seller, DOC008 returns NULL for the invoice ID, and DOC009 completely misidentifies all three fields. The ones listed above are those with issues (in some cases minor and in some cases completely wrong etc). Rest of the invoice entries are correct and not included in the above screen capture. The complete […]