Tensorlake

Document parsing benchmarks have been measuring the wrong thing. We tested every major parser on real enterprise documents. The results will change how you think about OCR accuracy 🧵

The results were clear: Tensorlake: 86.8% TEDS, 91.7% F1 AWS Textract: 80.7% TEDS, 88.4% F1 Azure: 78.1% TEDS, 88.1% F1 Docling: 63.8% TEDS, 68.9% F1 The gap? 670 fewer manual reviews per 10k documents.

Document parsing benchmarks have been measuring the wrong thing. We tested every major parser on real enterprise documents. The results will change how you think about OCR accuracy 🧵

Traditional benchmarks test on clean PDFs and measure text accuracy. But your production failures come from: - Collapsed tables - Jumbled reading order - Missing visual content - Hallucinated extractions None of this shows up in your scores.

New: Vision Language Models now power key document processing features We're using VLMs for: - Page classification in large documents - Table/figure summarization - Fast structured extraction (skip_ocr mode) Here's what this means for document processing 🧵

Where we leverage VLM support: 📄 Page Classification: Large docs, specific sections needed 📊 Table/Figure Summarization: Visual data in reports ⚡ skip_ocr=True: When reading order is complex and for diagrams and scanned docs Text extraction still uses OCR for best quality

New: Vision Language Models now power key document processing features We're using VLMs for: - Page classification in large documents - Table/figure summarization - Fast structured extraction (skip_ocr mode) Here's what this means for document processing 🧵

Our solution: VLMs understand document structure visually Example: Extracting crypto holdings from SEC filings 1. VLM classifies which pages contain financial data (~50 out of 200 pages) 2. Extract only from relevant pages 3. Skip 70% of processing Result: 80-90% faster ⚡

New: Vision Language Models now power key document processing features We're using VLMs for: - Page classification in large documents - Table/figure summarization - Fast structured extraction (skip_ocr mode) Here's what this means for document processing 🧵

Most parsers strip all tracked changes when you extract the text. That means: ❌ Lost audit trails ❌ Manual review of revision history ❌ No programmatic access to reviewer comments ❌ Workflows that can't route based on specific edits

Now you can parse .docx files with tracked changes preserved as clean, structured HTML: - <del> tags for deletions - <ins> tags for insertions - <span class="comment"> for reviewer notes

OCR engines constantly mess up document hierarchy. Section 2.2 becomes a top-level header (##) instead of nested (###). We just shipped automatic header correction. 🧵 How it works:

Every section header now returns: - level: 0 for #, 1 for ##, 2 for ###, etc - content: clean text - proper nesting for up to 6 levels Enable with: cross_page_header_detection=True That's it.

OCR engines constantly mess up document hierarchy. Section 2.2 becomes a top-level header (##) instead of nested (###). We just shipped automatic header correction. 🧵 How it works:

Citations. When users ask "where did this come from?" your system should point to the exact page fragment...not just "file_name.pdf". Built citation-aware RAG with spatial metadata has: → Parse docs with bounding boxes → Embed citation anchors in chunks → Return page numbers + coordinates A 🧵

Step 3: Generating AI responses with verifiable citations Once your chunks carry anchors, retrieval doesn’t change. You can use the dense, hybrid, or reranker setup you already have. Consider hiding the anchors in prose, while keeping them in output and making IDs clickable.

Citations. When users ask "where did this come from?" your system should point to the exact page fragment...not just "file_name.pdf". Built citation-aware RAG with spatial metadata has: → Parse docs with bounding boxes → Embed citation anchors in chunks → Return page numbers + coordinates A 🧵

Step 1: Parse docs with bounding boxes Using our Document AI API you get a full document layout. For each page fragment you have access to the page number, fragment type, content, and bounding box. Making it easy to add metadata and anchor points to chunks before embedding.

In finance, clinical trials, or performance benchmarks, dense tables contain mission-critical data. But flatten that data like most parsers do and trust is lost. Tensorlake restores trust by preserving structure, generating summaries for effective embeddings, and attaching evidence via b-boxes.

“RAG is dead” is lazy. What’s dead is cosine‑N without a retrieval plan. We ship advanced RAG...out of the box: • Classify pages → target sections • Extract structured fields → filter by form_type, fiscal_period • Verify data; cite page/bbox Want to know how? 🧵👇

Step 4: Test your context-aware agents This is the fun part, use Tensorlake to extract key claims from news articles, then use your @langchain.bsky.social agent to query ChromaDB and determine whether the claims are rooted in fact.

“RAG is dead” is lazy. What’s dead is cosine‑N without a retrieval plan. We ship advanced RAG...out of the box: • Classify pages → target sections • Extract structured fields → filter by form_type, fiscal_period • Verify data; cite page/bbox Want to know how? 🧵👇

TIP: Keep it fresh! Don’t just rebuild indexes monthly or annually, watch for new/changed filings and re-ingest continuously. The only rule: idempotency keyed on the SEC accession number.

“RAG is dead” is lazy. What’s dead is cosine‑N without a retrieval plan. We ship advanced RAG...out of the box: • Classify pages → target sections • Extract structured fields → filter by form_type, fiscal_period • Verify data; cite page/bbox Want to know how? 🧵👇

Step 1: Ingest and pre-process your documents With a single API call, you can turn messy PDFs into page-aware, table-preserving, structured context. Simply: 1. Define page classes for different documents 2. Define schemas to extract relevant data 3. Parse

Build a smart real estate agent (no license required). 🧠 LangGraph (by @langchain.bsky.social) + 📝 Tensorlake Contextual Signature Detection = ✅ Knows who signed ✅ When they signed ✅ If it’s ready to close Full tutorial + code linked below 👇

Want to see the tool in action? Check out this quick demo or try it out in the Colab Notebook (linked in the comments)

Build a smart real estate agent (no license required). 🧠 LangGraph (by @langchain.bsky.social) + 📝 Tensorlake Contextual Signature Detection = ✅ Knows who signed ✅ When they signed ✅ If it’s ready to close Full tutorial + code linked below 👇