DuckLake 1.0: Data Lake Format with SQL Catalog Metadata

DuckLake 1.0: Data Lake Format with SQL Catalog Metadata - InfoQ

[BT](http://www.infoq.com/int/bt/ "bt")

InfoQ Software Architects' Newsletter

A monthly overview of things you need to know as an architect or aspiring architect.

View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

Close

Live Webinar and Q&A: Portable by Design: Data Mobility & Recovery Patterns for Multi-Cloud Systems (May 21, 2026)Save Your Seat

Close

Toggle Navigation

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

English edition

• English edition
• Chinese edition
• Japanese edition
• French edition

[Write for InfoQ](http://www.infoq.com/write-for-infoq/ "Write for InfoQ")

Search

RegisterSign in

Unlock the full InfoQ experience

Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with content, and download exclusive resources.

Log In

or

Don't have an InfoQ account?

Register

• Stay updated on topics and peers that matter to youReceive instant alerts on the latest insights and trends.
• Quickly access free resources for continuous learningMinibooks, videos with transcripts, and training materials.
• Save articles and read at anytimeBookmark articles to read whenever youre ready.

Logo - Back to homepage

NewsArticlesPresentationsPodcastsGuides

Topics

[Development](http://www.infoq.com/development/ "Development")

• [Java](http://www.infoq.com/java/ "Java")
• [Kotlin](http://www.infoq.com/kotlin/ "Kotlin")
• [.Net](http://www.infoq.com/dotnet/ ".Net")
• [C#](http://www.infoq.com/c_sharp/ "C#")
• [Swift](http://www.infoq.com/swift/ "Swift")
• [Go](http://www.infoq.com/golang/ "Go")
• [Rust](http://www.infoq.com/rust/ "Rust")
• [JavaScript](http://www.infoq.com/javascript/ "JavaScript")

Featured in Development

• #### From VR to Flat Screens: Bridging the Input and Immersion Gap

Dany Lepage discusses the architectural journey of porting a hit VR title to seven non-VR platforms. He explains how his team solved the challenges of cross-progression, diverse input paradigms, and maintaining release velocity across Steam, iOS, and PlayStation. Beyond the tech, he shares candid lessons on the "product fit" gap when translating immersive social presence to 2D screens.


All in developmentFollow Topic

[Architecture & Design](http://www.infoq.com/architecture-design/ "Architecture & Design")

• [Architecture](http://www.infoq.com/architecture/ "Architecture")
• [Enterprise Architecture](http://www.infoq.com/enterprise-architecture/ "Enterprise Architecture")
• [Scalability/Performance](http://www.infoq.com/performance-scalability/ "Scalability/Performance")
• [Design](http://www.infoq.com/design/ "Design")
• [Case Studies](http://www.infoq.com/Case_Study/ "Case Studies")
• [Microservices](http://www.infoq.com/microservices/ "Microservices")
• [Service Mesh](http://www.infoq.com/servicemesh/ "Service Mesh")
• [Patterns](http://www.infoq.com/DesignPattern/ "Patterns")
• [Security](http://www.infoq.com/Security/ "Security")

Featured in Architecture & Design

• #### Stripe’s Docdb: How Zero-Downtime Data Movement Powers Trillion-Dollar Payment Processing

Jimmy Morzaria discusses the evolution of Stripe’s database tier to support 5 million QPS with 5.5 nines of reliability. He explains the architecture of DocDB and shares how Stripe leverages a custom zero-downtime data movement platform to perform horizontal sharding, version upgrades, and multi-tenant migrations - all while maintaining the strict consistency required for global commerce.


All in architecture-designFollow Topic

[AI Infrastructure](http://www.infoq.com/ai-ml-data-eng/ "AI Infrastructure")

• [Big Data](http://www.infoq.com/bigdata/ "Big Data")
• [Machine Learning](http://www.infoq.com/machinelearning/ "Machine Learning")
• [NoSQL](http://www.infoq.com/nosql/ "NoSQL")
• [Database](http://www.infoq.com/database/ "Database")
• [Data Analytics](http://www.infoq.com/data-analytics/ "Data Analytics")
• [Streaming](http://www.infoq.com/streaming/ "Streaming")

Featured in AI, ML & Data Engineering

• #### The Next Generation of AI Products

Hilary Mason shares her journey from academia to building AI products at scale. She discusses the shift from discrete engineering to probabilistic mindsets, explaining why managing "human considerations" is the hardest part of the stack. She explains the "existential crisis" for engineers, arguing that great architecture today is about context management, systems thinking, and good taste.


All in ai-ml-data-engFollow Topic

[Culture & Methods](http://www.infoq.com/culture-methods/ "Culture & Methods")

• [Agile](http://www.infoq.com/agile/ "Agile")
• [Diversity](http://www.infoq.com/diversity/ "Diversity")
• [Leadership](http://www.infoq.com/leadership/ "Leadership")
• [Lean/Kanban](http://www.infoq.com/lean/ "Lean/Kanban")
• [Personal Growth](http://www.infoq.com/personal-growth/ "Personal Growth")
• [Scrum](http://www.infoq.com/scrum/ "Scrum")
• [Sociocracy](http://www.infoq.com/sociocracy/ "Sociocracy")
• [Software Craftmanship](http://www.infoq.com/software_craftsmanship/ "Software Craftmanship")
• [Team Collaboration](http://www.infoq.com/team-collaboration/ "Team Collaboration")
• [Testing](http://www.infoq.com/testing/ "Testing")
• [UX](http://www.infoq.com/ux/ "UX")

Featured in Culture & Methods

• #### Panel: Building a Culture that Works

The panelists share insights on evolving company culture. They discuss leveraging feedback loops, lending social capital, and the friction between legacy bureaucracy and agile engineering. The panel explains how to maintain cohesion in remote teams and use interviews to uncover the true "unmanicured" culture of a firm.


All in culture-methodsFollow Topic

DevOps

• [Infrastructure](http://www.infoq.com/infrastructure/ "Infrastructure")
• [Continuous Delivery](http://www.infoq.com/continuous_delivery/ "Continuous Delivery")
• [Automation](http://www.infoq.com/automation/ "Automation")
• [Containers](http://www.infoq.com/containers/ "Containers")
• [Cloud](http://www.infoq.com/cloud-computing/ "Cloud")
• [Observability](http://www.infoq.com/observability/ "Observability")

Featured in DevOps

• #### AI-Powered SRE for Autonomous Incident Response

The presenters discuss incident response, how AI-enhanced SRE platforms connect signals from logs, metrics, traces, and historical incidents to enable autonomous decisions.


All in devopsFollow Topic

[Events](https://events.infoq.com/ "Events")

Helpful links

• [About InfoQ](http://www.infoq.com/about-infoq "About InfoQ")
• [InfoQ Editors](http://www.infoq.com/infoq-editors "InfoQ Editors")
• [Write for InfoQ](http://www.infoq.com/write-for-infoq "Write for InfoQ")
• [About C4Media](https://c4media.com/ "About C4Media")
• [Diversity](https://c4media.com/diversity "Diversity")

Choose your language

• [En](http://www.infoq.com/news/2026/05/ducklake-sql-catalog/# "InfoQ English")
• 中文
• 日本
• Fr


[InfoQ Homepage](http://www.infoq.com/ "InfoQ Homepage")[News](http://www.infoq.com/news "News")DuckLake 1.0: Data Lake Format with SQL Catalog Metadata

[AI, ML & Data Engineering](http://www.infoq.com/ai-ml-data-eng/ "AI, ML & Data Engineering")

Online InfoQ Architect Certification (May 7): Peer conversations that change how you think.

DuckLake 1.0: Data Lake Format with SQL Catalog Metadata

May 02, 2026 2 min read

by

• Renato Losio

Follow Cloud Expert | AWS Data Hero

#### Write for InfoQ

Feed your curiosity.Help 550k+ global

senior developers

each month stay ahead.Get in touch

Log in to listen to this article

Audio ready to play

Your browser does not support the audio element.

0:00 0:00

Normal 1.25x 1.5x

Like

• Reading list

DuckDB Labs recently released DuckLake 1.0, a data lake format that stores table metadata in a SQL database rather than across many files in object storage. The first implementation is available as a DuckDB extension and includes catalog-stored small updates, improved sorting and partitioning options, and compatibility with Iceberg-style data features.

According to the DuckDB team, file-based metadata in lake formats leads to complex coordination, slow metadata operations, and many small files in object storage. While Apache Iceberg, Delta Lake, and Apache Hudi store metadata mainly as files in object storage, sometimes adding catalog services on top, DuckLake stores it directly in a SQL database.

A year ago, the so-called "DuckLake manifesto" was published, arguing that lakehouse metadata should be stored in a database rather than spread across many files in object storage. The team writes:

We are happy to announce DuckLake v1.0, almost a year after we released our first sketch of the specification. This is a production-ready release with guaranteed backward-compatibility. DuckLake v1.0 ships a stable specification, a feature-rich and fast reference implementation (the DuckDB ducklake extension), as well as a roadmap for future development.

DuckLake 1.0 adds several features to improve lakehouse operations and performance. These include data inlining to handle small inserts, updates, and deletes without creating new files, sorted tables to speed up filtered queries, bucket partitioning for high-cardinality columns, improved support for geometry data types, and deletion vectors compatible with Iceberg. Discussing data inlining, the team notes:

Data inlining is one of the flagship features of DuckLake. It basically enables performing small insert, delete and update operations in the catalog database, avoiding the proliferation of "the small file problem". DuckLake v1.0 brings full inlining of updates and deletes. This feature is now on by default with a default threshold of 10 rows.

On a popular Reddit thread, user _SutMinSnabel4_ asks:

Could you add first-class support for the SMB protocol? I do not mean locally mounted filesystems, since those are OS-dependent. It should work well in traditional enterprise Windows environments too, with DFS, Kerberos, and the whole shebang, but it should also work on macOS and Linux machines (...) A lot of enterprises still rely on SMB on-premises.

On Hacker News, Alexander Dahl, data platform engineer, writes:

Very exciting! The numbers seem to crush Iceberg. Has anyone tried it out for "real" workloads?

DuckLake clients are available for Apache DataFusion, Apache Spark, Trino, and Pandas. MotherDuck also offers a hosted DuckLake service that manages the catalog database and storage.

DuckLake v1.1 is expected to add improvements such as variant inlining across catalogs and multi-deletion vector Puffin files. According to the roadmap, DuckLake v2.0 will introduce Git-like branching for datasets and built-in role-based permissions.

The awesome-ducklake repository provides DuckLake use cases and libraries. DuckLake 1.0 is available on GitHub under an MIT license.

About the Author

#### Renato Losio

Renato has extensive experience as a cloud architect, tech lead, and cloud services specialist. Currently, he lives in Berlin and works remotely as a principal cloud architect. His primary areas of interest include cloud services and relational databases. He is an editor at InfoQ and a recognized AWS Data Hero. You can connect with him on LinkedIn.

Show more Show less

#### This content is in the AI, ML & Data Engineering topic

Follow Topic

##### Related Topics:

• Architecture & Design### Architecture & Design

Followers: 10215

Follow Topic

• AI, ML & Data Engineering### AI, ML & Data Engineering

Followers: 5886

Follow Topic

• Apache Iceberg### Apache Iceberg

Followers: 5

Follow Topic

• Data Partitioning### Data Partitioning

Followers: 5

Follow Topic

• Data Lake### Data Lake

Followers: 18

Follow Topic

• duckdb### duckdb

Followers: 0

Follow Topic

• SQL### SQL

Followers: 31

Follow Topic

• Data Storage### Data Storage

Followers: 16

Follow Topic

• Data Catalog### Data Catalog

Followers: 8

Follow Topic

* #### Popular in AI, ML & Data Engineering

• ##### Anthropic Introduces Managed Agents to Simplify AI Agent Deployment

• ##### Cloudflare Announces Agent Memory, a Managed Persistent Memory Service for AI Agents

• ##### Microsoft's Russinovich and Hanselman Warn AI Is Hollowing out the Junior Developer Pipeline

• ##### Orchestrating Agentic and Multimodal AI Pipelines with Apache Camel

• ##### MCP in the Java World: Bringing Architectural Strategy to LLM Integrations

* #### Related Sponsors

• ##### Inside MCP: A Protocol for AI Integration

• ##### Portable by Design: Data Mobility & Recovery Patterns for Multi-Cloud Systems (Live Webinar May 21, 2026) - Save Your Seat

• ##### AI Agent Swarm Patterns (InfoQ Webinar) - Watch Now On-Demand

• ##### Gartner® Research: Hype Cycle™ for Cloud Platform Services, 2025

• ##### The Essential Guide to AI Tools for Jakarta EE Developers

* #### Related Sponsor


• May 21, 2026, 12 PM EDT

##### Portable by Design: Data Mobility & Recovery Patterns for Multi-Cloud Systems

Presented by: Liore Shai - Solutions Architect at Eon

SPONSORED BY EON Save your seat

Related Content

• ##### Google Cloud Highlights Ongoing Work on PostgreSQL Core Capabilities

Apr 10, 2026

• ##### Lakehouse Tower of Babel: Handling Identifier Resolution Rules across Database Engines

Apr 17, 2026 

• ##### Dropbox Redesigns Compaction to Reclaim Space from Underfilled Storage Volumes

Apr 30, 2026

• ##### AWS Introduces S3 Files, Bringing File System Access to S3 Buckets

Apr 16, 2026

• ##### JobRunr Introduces ClawRunr, an Open-Source Java AI Agent

May 01, 2026

• ##### Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance

May 01, 2026

• ##### Meta Deploys Unified AI Agents to Automate Performance Optimization at Hyperscale

May 01, 2026

• Icon##### The Next Generation of AI Products

May 01, 2026 

• ##### Vitest 4.1: Test Tags, Native Node.js Execution and AI Agent Reporter

May 01, 2026

Related Sponsors

• #### Design for Failure: How to Guarantee Data Access During Cloud Outages

Learn how to maintain data access during cloud outages by using independent, cross-cloud backup strategies. This ensures you can search and restore data even when major providers like AWS, Azure, or Google Cloud experience critical downtime.

• #### Multi-Cloud Backup Is Broken: Architectural Patterns That Actually Work

Learn how to navigate multi-cloud backup challenges across AWS, Azure, and Google Cloud. By addressing tool fragmentation and siloed data, teams can ensure consistent policies, reduce costs, and maintain seamless recovery across all providers.

• Sponsored by


Related Content

• ##### Securing Autonomous AI Agents on Kubernetes: Trust Boundaries, Secrets, and Observability for a New Category of Cloud Workload

May 01, 2026 

• ##### Broadcom Donates Velero to CNCF, Shifting Kubernetes Backup to Community Governance

May 01, 2026

• ##### NVIDIA Launches Ising Open Models for Quantum Computing

Apr 30, 2026

• ##### NestJS v12 Roadmap: Full ESM Migration, Standard Schema Validation and Modernised Toolchain

Apr 30, 2026

• ##### Driving and Measuring the Impact of Platform Engineering

Apr 30, 2026

• ##### Cloudflare Announces Agent Memory, a Managed Persistent Memory Service for AI Agents

Apr 30, 2026

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

• Development

• ##### [QCon San Francisco 2026: 12 Tracks Announced](http://www.infoq.com/news/2026/04/qconsf-2026-tracks-announced/ "QCon San Francisco 2026: 12 Tracks Announced")

• ##### [Microsoft's Russinovich and Hanselman Warn AI Is Hollowing out the Junior Developer Pipeline](http://www.infoq.com/news/2026/04/junior-developer-pipeline-crisis/ "Microsoft's Russinovich and Hanselman Warn AI Is Hollowing out the Junior Developer Pipeline")

• ##### [C++26: Reflection, Memory Safety, Contracts, and a New Async Model](http://www.infoq.com/news/2026/04/cpp-26-reflection-safety-async/ "C++26: Reflection, Memory Safety, Contracts, and a New Async Model")

• Architecture & Design

• ##### [Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance](http://www.infoq.com/news/2026/05/confluent-kafka-header-schema-id/ "Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance")

• ##### [Meta's Approach to Migrating their Systems to Post-Quantum Cryptography](http://www.infoq.com/news/2026/04/meta-quantum-crypto-migration/ "Meta's Approach to Migrating their Systems to Post-Quantum Cryptography")

• ##### [Stripe’s Docdb: How Zero-Downtime Data Movement Powers Trillion-Dollar Payment Processing](http://www.infoq.com/presentations/docdb-online-database/ "Stripe’s Docdb: How Zero-Downtime Data Movement Powers Trillion-Dollar Payment Processing")

• Culture & Methods

• ##### [Driving and Measuring the Impact of Platform Engineering](http://www.infoq.com/news/2026/04/measure-platform-engineering/ "Driving and Measuring the Impact of Platform Engineering")

• ##### [How Observability and Telemetry Can Enhance the Practice of Software Engineering](http://www.infoq.com/news/2026/04/observability-telemetry/ "How Observability and Telemetry Can Enhance the Practice of Software Engineering")

• ##### [Panel: Building a Culture that Works](http://www.infoq.com/presentations/panel-positive-culture/ "Panel: Building a Culture that Works")

• AI, ML & Data Engineering

• ##### [DuckLake 1.0: Data Lake Format with SQL Catalog Metadata](http://www.infoq.com/news/2026/05/ducklake-sql-catalog/ "DuckLake 1.0: Data Lake Format with SQL Catalog Metadata")

• ##### [The Next Generation of AI Products](http://www.infoq.com/presentations/ai-products/ "The Next Generation of AI Products")

• ##### [NVIDIA Launches Ising Open Models for Quantum Computing](http://www.infoq.com/news/2026/04/nvidia-ising-quantum/ "NVIDIA Launches Ising Open Models for Quantum Computing")

• DevOps

• ##### [Meta Deploys Unified AI Agents to Automate Performance Optimization at Hyperscale](http://www.infoq.com/news/2026/05/meta-ai-agents-hyperscale/ "Meta Deploys Unified AI Agents to Automate Performance Optimization at Hyperscale")

• ##### [Broadcom Donates Velero to CNCF, Shifting Kubernetes Backup to Community Governance](http://www.infoq.com/news/2026/05/broadcom-velero-cncf/ "Broadcom Donates Velero to CNCF, Shifting Kubernetes Backup to Community Governance")

• ##### [Dropbox Redesigns Compaction to Reclaim Space from Underfilled Storage Volumes](http://www.infoq.com/news/2026/04/dropbox-tiered-compaction/ "Dropbox Redesigns Compaction to Reclaim Space from Underfilled Storage Volumes")

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

• Get a quick overview of content published on a variety of innovator and early adopter technologies
• Learn what you don’t know that you don’t know
• Stay up to date with the latest information from the topics you are interested in

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

**May 7 | June 10, 2026 | Online** Architecture decisions are hard to validate while shipping. Join a **5-week online cohort** for **senior engineers, architects, and team leads** to pressure-test real decisions, apply practical frameworks, and work through challenges with a confidential peer group. Facilitated by Luca Mezzalira, Principal Architect at AWS, this cohort helps you: * Pressure-test real decisions. * Apply frameworks to real problems. * Publish on InfoQ.com and earn your certification. **RESERVE YOUR PLACE**

#### Events

• ##### Online InfoQ Architect Certification

May 7, 2026

• ##### QCon AI Boston

June 1-2, 2026

• ##### Online InfoQ Architect Certification

June 10, 2026

• ##### QCon San Francisco

November 16-20, 2026

#### Follow us on

Youtube 232K FollowersLinkedin 26K FollowersInstagram NewRSS 19K ReadersX 57.1k FollowersFacebook 21K LikesBluesky New

#### Stay in the know

The InfoQ PodcastEngineering Culture PodcastThe Software Architects' Newsletter

General Feedback [[email protected]](mailto:[email protected]) Advertising [[email protected]](mailto:[email protected]) Editorial [[email protected]](mailto:[email protected]) Marketing [[email protected]](mailto:[email protected])

InfoQ.com and all content copyright © 2006-2026 C4Media Inc.

Privacy Notice, Terms And Conditions, Cookie Policy

Close

[BT](http://www.infoq.com/int/bt/ "bt")