Building with Databricks Document Intelligence and Lakeflow

DATA + AI SUMMIT ![Image 5: Data+ai summit promo JUNE 15–18|SAN FRANCISCO Last chance to save 50% — ends April 30. Register](http://www.databricks.com/dataaisummit?itm_source=www&itm_category=home&itm_page=home&itm_location=navigation&itm_component=navigation&itm_offer=dataaisummit)

Blog
/ Product
/ Article

* *

Contents in this story

Step 1: Secure Ingestion with Lakeflow Connect

Building with Databricks Document Intelligence and Lakeflow

Turn locked enterprise knowledge into queryable, trusted intelligence

Image 6: Databricks Lakeflow + Databricks Document Intelligence

Published: April 16, 2026

Product6 min read

by Giselle Goicochea and Joanna Zouhour

Share this post

[](https://www.linkedin.com/shareArticle?mini=true&url=https://www.databricks.com/blog/building-databricks-document-intelligence-and-lakeflow&summary=&source=)
[](https://twitter.com/intent/tweet?text=https://www.databricks.com/blog/building-databricks-document-intelligence-and-lakeflow)
[](https://www.facebook.com/sharer/sharer.php?u=https://www.databricks.com/blog/building-databricks-document-intelligence-and-lakeflow)

Keep up with us

Subscribe

#### Summary

Most enterprise knowledge is inaccessible in unstructured documents, while current intelligent document processing (IDP) is often brittle and unreliable
Databricks Document Intelligence and Lakeflow enable data engineers to easily build and automate an end-to-end IDP workflow: ingesting unstructured data, parsing it with AI intelligence grounded in enterprise context, and then orchestrating at scale, all within a governed platform
Data teams can surface previously hidden documents into trusted, queryable datasets that help unlock new insights, agentic workflows, and value for their business

Despite decades of perfecting structured data pipelines, 80% of enterprise knowledge remains functionally invisible, trapped in PDFs, images, and office documents.

Traditionally, Intelligent Document Processing (IDP) has been a fragmented nightmare. Before the era of Generative AI, organizations were forced to rely on disconnected NLP and computer vision APIs that were outside of their primary data platforms. These siloed OCR (optical character recognition) vendors offered limited accuracy and lacked formal governance protocols, creating significant friction. To deliver on the promise of Enterprise AI, we need a unified approach that integrates data intelligence directly into the data lifecycle.

Today, we’re showing how data engineers can leverage Lakeflow, Databricks’ unified data engineering solution, and Databricks Document Intelligence to unlock that data and turn it into business-impacting intelligence by building production-grade autonomous IDP in their Databricks Platform.

Image 8: Lakeflow Connect - Document Intelligence - Lakeflow Jobs

Step 1: Secure Ingestion with Lakeflow Connect

Enterprise documents live in siloed graveyards, accessible only through fragile, custom-coded API integrations that break the moment a folder is renamed. Lakeflow Connect, Databricks' solution for ingesting data into the lakehouse, changes the game with built-in connectors for many popular enterprise applications, databases, and file sources including SharePoint and Google Drive.

This solution offers zero-maintenance ingestion by removing the need to manage complex OAuth flows or custom Python scripts. Documents land directly in Unity Catalog Volumes and tables, so access control, lineage, and auditing apply as soon as the file is in the lakehouse, and you can reuse the same fine‑grained, attribute‑based policies you already rely on for structured data.

You also get fast and efficient ingestion at scale thanks to Lakeflow Connect’s robust capabilities, including incremental reads and writes which avoids full re‑pulls of large libraries for both batch backfills and near‑real‑time document flows when combined with streaming downstream.

Step 2: Getting started with Databricks Document Intelligence

These enterprise documents carry some of your organization’s most valuable insights but are inherently messy, variable and inconsistent. Scanned pages, handwritten notes and nested tables trap your most valuable insights. To fix this, you don’t just need another document extraction tool; as Forrester notes, you need a “reasoning-first architectural evolution.” With this approach, Gartner predicts GenAI will reduce the need for custom-trained document models by 70%.

Today, with Databricks Document Intelligence, you can bring state-of-the-art document understanding directly to your data. Your data engineering teams can leverage purpose-built AI functions that can reliably parse, structure, and enrich complex documents right alongside your existing data pipelines, all seamlessly governed by Unity Catalog.

**ai_parse_document**(new - GA): This function converts unstructured files into structured representations using the Variant data type. It natively handles input complexity that typically trips up traditional parsers, such as scanned images, handwriting, and variable layouts, while preserving critical document structure (e.g., nested tables, sections, and headers) that flat text extraction would lose. This allows you to evolve schemas over time without breaking your pipelines. Downstream, you treat the VARIANT output as a flexible bronze/silver representation, projecting it into Delta columns in your silver/gold layers using SQL or PySpark in Lakeflow Spark Declarative Pipelines.

On top of the parsed structure, you can chain additional research-tuned AI Functions:

**ai_extract**(PuPr) to pull structured insights such as contract effective and expiration dates, counterparties, invoice totals, taxes, currency, and PO numbers.
**ai_classify**(PuPr)to route documents by type (invoice, PO, SOW, NDA), urgency/risk, or owning business unit.
**ai_prep_search**(new - Beta) to intelligently divide documents into chunks for high-quality downstream embedding, preparing them for retrieval or search use cases

Below is a simple example of chaining ai_parse_document and ai_extract together.

_Note: this example shows PySpark, but you can also use SQL (see documentation)._

python

from pyspark.sql.functions import expr

# Load raw documents as binary data from a Unity Catalog Volume
df = 
spark.read.format("binaryFile").load("/Volumes/samples/sec/contracts")

# Step 1: Parse the unstructured documents
parsed_docs = df.withColumn(
    "parsed_contract",
    expr("ai_parse_document(content, map('version', '2.0'))")
)

# Step 2: Extract specific entities based on a defined schema
json_schema = '''{
    "parties": {
        "type": "array",
        "items": {"type": "string"}
    },
    "agreement_date": {
        "type": "string",
        "description": "Date the contract was signed, formatted as MM-DD-YYYY."
    },
    "governing_law": {
        "type": "string",
        "description": "State or country whose laws govern the contract"
    }
}'''

# Step 3: Extract specific entities based on the defined schema
contract_details_df = parsed_docs.select(
    "path",
    expr(f"ai_extract(parsed_contract, '{json_schema}')").alias("contract_details")
)

display(contract_details_df)

Because these are managed AI Functions integrated into the Databricks Platform, Document Intelligence can combine them with your enterprise context (catalog metadata, business semantics, existing tables) to power agentic workflows that reason over your data with high accuracy, grounded in your enterprise domain context.

Step 3: Productionizing IDP Workloads at Scale

Once you have ingestion and parsing working in notebooks, you need to productionize your IDP: orchestrate ingestion, parsing, enrichment, and serving. But you also want to monitor SLAs, failures, and retries in CI/CD to ensure pipelines remain healthy.

With Lakeflow Jobs, Databricks’ native orchestrator, you can turn IDP workloads into robust, automated pipelines with the same orchestration system you use for ETL, analytics, and ML. It provides unified orchestration for every task in the IDP DAG, so you can chain notebooks, Python scripts, SQL queries, pipelines, LLMs, or agent calls in a single job and model the full flow from document ingestion.

Lakeflow Jobs also comes with built-in advanced control flow(including if/else conditions, for each, retries, etc.) and triggers (table update, file arrival, continuous, etc.). This makes it easy to 1) re‑process only failed partitions or specific document batches and 2) manage jobs to fit specific schedules, event‑based triggers, or continuous mode for real‑time document streams.

With Lakeflow Jobs’ serverless compute with native observability, you also get automatic scaling with spikes in document volume while surfacing real‑time monitoring, metrics, and alerts so you can pinpoint bottlenecks and repair failures without needing to re-run successful tasks.

Image 9: DAG displaying productionizing IDP workloads

GUIDE

Your compact guide to modern analytics

Read now

Grounding AI in Enterprise Context

IDP is most valuable when it is backed by enterprise context: your unique schemas, business definitions, and custom semantics.

Unity Catalog

Unity Catalog provides unified governance and discoveryacross structured data, unstructured files, ML models, and business metrics on any cloud. For IDP, that means:

A single place to define access policies, lineage, and auditing for both raw documents and derived structured tables
Support for open formats(Delta, Apache Iceberg, Hudi, Parquet) so you aren’t locked into a proprietary document representation
Business semantics and catalog‑level metadata that agents can use to consistently name and interpret entities such as “Vendor,” “Customer,” or “Contract Value.”

Document Intelligence

**Document Intelligence** uses this context to build production AI agents that know which tables, tools, and models to use for a given IDP task, are governed end‑to‑end so they never access more than they should, and continuously improve via LLM‑based quality scoring, task‑specific benchmarks, and learning loops. For developers, Databricks provides APIs and SDKs so you can define these agents as code and integrate them into your existing CI/CD pipelines, just like any other data or ML asset.

Best Practices for the Modern IDP Stack

To move from pilot to platform, keep these best practices in mind:

Data Enrichment: Don’t just extract a "Vendor Name." Join it with your internal Master Data or third-party sources (like Dun & Bradstreet) to provide full business context.
Operational Excellence: Use Service Principals for Lakeflow Jobs to ensure pipeline stability.
Monitoring: Use Lakehouse Monitoring to track model drift and extraction accuracy over time.

The Path to Modern Data Intelligence

With Databricks, you can own the full lifecycle of Intelligent Document Processing on a modern data platform. Combining Lakeflow and AI functions lets you turn unstructured, hidden data into trusted, queryable datasets and seamlessly run observable document pipelines alongside your core ETL and ML.

Now that we’ve covered the strategic value of autonomous document intelligence, it’s time to build it. Check out our companion post, From PDF to Insights, for a step-by-step technical walkthrough on deploying this exact architecture using Databricks.

You can also explore the Document Intelligence and Lakeflow documentation to start building your first IDP pipeline today!

Keep up with us

Subscribe

Contents in this story

Step 1: Secure Ingestion with Lakeflow Connect

Recommended for you

Image 12: Agent Bricks: The Governed Enterprise Agent Platform

Product

April 14, 2026/6 min read

#### Agent Bricks: The Governed Enterprise Agent Platform

Product

February 24, 2026/7 min read

#### An AI-First Approach to Data Engineering with Lakeflow and Agent Bricks

Share this post

[](https://www.linkedin.com/shareArticle?mini=true&url=https://www.databricks.com/blog/building-databricks-document-intelligence-and-lakeflow&summary=&source=)
[](https://twitter.com/intent/tweet?text=https://www.databricks.com/blog/building-databricks-document-intelligence-and-lakeflow)
[](https://www.facebook.com/sharer/sharer.php?u=https://www.databricks.com/blog/building-databricks-document-intelligence-and-lakeflow)

Never miss a Databricks post

Subscribe to our blog and get the latest posts delivered to your inbox

Sign up

*

Work Email

*

Country:

By clicking “Subscribe” I understand that I will receive Databricks communications, and I agree to Databricks processing my personal data in accordance with its Privacy Policy.

Subscribe

What's next?

More from the Authors

Image 16: Generating Coding Tests for LLMs: A Focus on Spark SQL

Data Engineering

October 2, 2024/10 min read

#### Generating Coding Tests for LLMs: A Focus on Spark SQL

Image 18: How to present and share your Notebook insights in AI/BI Dashboards

Product

November 21, 2024/3 min read

#### How to present and share your Notebook insights in AI/BI Dashboards

Image 20: databricks logo

Why Databricks

Discover

Customers

Customer Stories

Partners

Why Databricks

Discover

Customers

Customer Stories

Partners

Product

Databricks Platform

Pricing

Open Source

Integrations and Data

Product

Databricks Platform

Pricing

Open Source

Integrations and Data

Solutions

Databricks For Industries

Cross Industry Solutions

Data Migration

Professional Services

Solution Accelerators

Solutions

Databricks For Industries

Cross Industry Solutions

Data Migration

Professional Services

Solution Accelerators

Resources

Documentation

Customer Support

Community

Learning

Events

Blog and Podcasts

Resources

Documentation

Customer Support

Community

Learning

Events

Blog and Podcasts

About

Company

Careers

Press

Security and Trust

About

Company

Careers

Press

Security and Trust

Image 22: databricks logo

Databricks Inc.

160 Spear Street, 15th Floor

San Francisco, CA 94105

1-866-330-0121

[](https://www.linkedin.com/company/databricks)
[](https://www.facebook.com/pages/Databricks/560203607379694)
[](https://twitter.com/databricks)
[](https://www.databricks.com/feed)
[](https://www.glassdoor.com/Overview/Working-at-Databricks-EI_IE954734.11,21.htm)
[](https://www.youtube.com/@Databricks)

Image 24

See Careers

at Databricks

[](https://www.linkedin.com/company/databricks)
[](https://www.facebook.com/pages/Databricks/560203607379694)
[](https://twitter.com/databricks)
[](https://www.databricks.com/feed)
[](https://www.glassdoor.com/Overview/Working-at-Databricks-EI_IE954734.11,21.htm)
[](https://www.youtube.com/@Databricks)

We Care About Your Privacy

Databricks uses cookies and similar technologies to enhance site navigation, analyze site usage, personalize content and ads, and as further described in our Cookie Notice. To disable non-essential cookies, click “Reject All”. You can also manage your cookie settings by clicking “Manage Preferences.”

Manage Preferences

Reject All Accept All

Image 27: Databricks Company Logo

Privacy Preference Center

Opt-Out Preference Signal Honored

Privacy Preference Center

### Your Privacy
### Strictly Necessary Cookies
### Performance Cookies
### Functional Cookies
### Targeting Cookies
### TOTHR

#### Your Privacy

When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. This information might be about you, your preferences or your device and is mostly used to make the site work as you expect it to. The information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies. Click on the different category headings to find out more and change our default settings. However, blocking some types of cookies may impact your experience of the site and the services we are able to offer.

#### Opting out of sales, sharing, and targeted advertising

Depending on your location, you may have the right to opt out of the “sale” or “sharing” of your personal information or the processing of your personal information for purposes of online “targeted advertising.” You can opt out based on cookies and similar identifiers by disabling optional cookies here. To opt out based on other identifiers (such as your email address), submit a request in our Privacy Request Center.

More information

#### Strictly Necessary Cookies

Always Active

These cookies are necessary for the website to function and cannot be switched off in our systems. They assist with essential site functionality such as setting your privacy preferences, logging in or filling in forms. You can set your browser to block or alert you about these cookies, but some parts of the site will no longer work.

#### Performance Cookies

[x] Performance Cookies

These cookies allow us to count visits and traffic sources so we can measure and improve the performance of our site. They help us to know which pages are the most and least popular and see how visitors move around the site.

#### Functional Cookies

[x] Functional Cookies

These cookies enable the website to provide enhanced functionality and personalization. They may be set by us or by third party providers whose services we have added to our pages. If you do not allow these cookies then some or all of these services may not function properly.

#### Targeting Cookies

[x] Targeting Cookies

These cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant advertisements on other sites. If you do not allow these cookies, you will experience less targeted advertising.

#### TOTHR

[x] TOTHR

Cookie List

Consent Leg.Interest

[x] checkbox label label

[x] checkbox label label

[x] checkbox label label

Clear

- [x] checkbox label label

Apply Cancel

Confirm My Choices

Allow All