End-to-End Document Extraction

Core-Field Auto Extraction for Customs Declarations

Extracts KCS import-declaration fields from unstructured documents such as invoices and B/Ls, then outputs usable JSON/EDI.

FastAPIOCRRegex IRSHACL QCCLI
API Docs

One-line summary: Automated the path from input documents to text acquisition, candidate IR, scoring, SHACL validation, normalization, and EOC output across UI/CLI/API.

Role: end-to-end design/build · schema and quality assurance · documentation/dashboard

How It Works

Pre-inspection → preprocessing/OCR → routing → candidate extraction → scoring/SHACL → post-processing/normalization → EOC merge

  1. Pre-InspectCheck input state and required items
  2. Preprocess/OCRAcquire text and add OCR fallback
  3. RoutingDecide document type and processing path
  4. Candidate ExtractionFind field candidates from IR
  5. ScoringRun score and SHACL quality checks
  6. NormalizePost-process and merge into EOC
  7. OutputReview results through UI, CLI, or API

Evidence: Quality & Validation

Evaluation
2 vendors · 3 invoices
Validation
OCR · IR · SHACL · Normalization
Quality
field scores · violation logs
Output Contract
JSON/EDI-ready structure
Ops Rule
Add F1/WER after sample expansion
Latency
document-level measurement design
Cost
local / no external API mode

Text Quality

  • OCR preference based on printable_ratio, pages, and file size
  • Quality inference through text statistics

SHACL Validation

  • Conformance bonuses and violation penalties
  • QC metrics and scoring

Normalization Guards

  • Two-decimal amounts and YYYY-MM-DD dates
  • Header-like string and category correction

Note: Metrics explicitly compare sample and operating-environment differences. “100% normalization” means accuracy under schema casting.

Data Contract

Supports a built-in DEFAULT_SCHEMA plus external report_format.json overrides, with loose casting through cast_contract_to_schema.

{
  "vendor": "string",
  "invoice_no": "string",
  "date": "YYYY-MM-DD",
  "amount_total": "number(2-digit decimals)",
  "currency": "string",
  "items": [
    { "desc": "string", "qty": "number", "unit_price": "number", "line_total": "number" }
  ],
  "meta": {
    "used_ocr": "bool",
    "printable_ratio": "float",
    "shacl": { "score": "float", "violations": "int" }
  }
}

Conceptual sample structure; actual fields follow the project schema.

API Sketch

POST /analyze

curl -X POST http://localhost:8080/analyze \
  -H "Content-Type: multipart/form-data" \
  -F "files=@sample.pdf"

Input: files[]. Output: per-document candidates, ir, and pipeline artifacts.

POST /render

curl -X POST http://localhost:8080/render \
  -H "Content-Type: application/json" \
  -d '{"doc_id":"...", "select":"best"}'

Applies post-processing and normalization to the selected result.

Module Summary

cli / poc_runner

  • Pipeline orchestration and final EOC merge
  • run_pipeline, run, _merge_docs_to_eoc

pre_inspect

  • MIME/page/readability/size checks and prefer_ocr decision
  • Outputs inputs[] metadata

pre_flight

  • Text acquisition with PDF extraction and OCR fallback
  • Outputs docs[] text/statistics

routing

  • Classifies PDF and image paths
  • Passes through docs

multi_engine

  • Builds IR and regex-based candidates
  • Outputs candidates and ir

scoring

  • Combines weights, sanity checks, SHACL score, and best selection
  • Outputs score and qc{shacl}

post_process

  • Schema casting, amount/date normalization, and cross-field correction
  • Produces final final payload

Utilities

  • pdf_text: text statistics and post-processing
  • ocr: page OCR
  • compare: normalized diff

Dashboard / API

  • FastAPI: GET /, POST /analyze, /render
  • Includes Uvicorn execution scripts

Folder/File Structure Check