Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions

Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions - InfoQ

InfoQ Homepage News Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions

Architecture & Design

Online InfoQ AI Security & Privacy Engineering Certification (Oct 26): Shipping AI in a regulated environment?

Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions

Oct 19, 2026 2 min read

by

• Leela Kumili

#### Follow us on

Youtube

232K Followers

Linkedin

26K Followers

Instagram

New

RSS

19K Readers

X

57.1k Followers

Facebook

21K Likes

Bluesky

Listen to this article -

0:00

Audio ready to play

Your browser does not support the audio element.

Normal

1.25x

1.5x

Like

new dropdown rading list

• Reading list

Pinterest has introduced an event time-based partition finalization mechanism for its next-generation database ingestion framework, allowing downstream jobs to determine when streaming data is sufficiently complete to consume. The approach uses Apache Flink to measure event time progress and Apache Iceberg metadata to publish a non-regressing finalization watermark, addressing a challenge in continuous Change Data Capture (CDC) ingestion where data can arrive after a partition’s wall clock interval has ended.

Pinterest’s ingestion framework uses CDC , Kafka, Flink, Spark and Iceberg to move database changes from MySQL, TiDB and KVStore into analytical tables. Pinterest says the platform supports petabyte-scale data and thousands of pipelines, with online changes typically available within 15 minutes to an hour. Unlike the previous batch-based approach, where job completion indicated that a partition was ready, streaming ingestion continuously commits changes without an equivalent completion point.

Pinterest architecture showing Kafka CDC events flowing through Flink into a CDC Iceberg table (Source: Pinterest Blog Post )

Nitesh Singhal , Architect at Google, highlighted the design in a LinkedIn post, writing ,

The non-regressing safe-to-read watermark is the elegant bit here.

The new mechanism runs inside the Flink to Iceberg sink. For each CDC record, the system extracts a configured event time. During Flink checkpoints, it collects minimum, maximum, and percentile event time statistics. These are stored in the Iceberg snapshot summary, with a t-digest representing the distribution in a few hundred bytes.

After an Iceberg commit, the system calculates a partition finalization watermark from recent checkpoint statistics. The default policy uses the minimum event time across a configurable number of recent commits and rounds it down to the relevant hour. The watermark is monotonic, preventing late-arriving records from moving an already published value backward. Percentile-based policies can trade some completeness for fresher data.

Pinterest describes the change by noting the watermark is stored as an Iceberg table property, allowing downstream sensors to read it without a separate coordination service. Spark also propagates the metadata to the base table during its CDC upsert commit and emphasizes

Finalization is no longer a byproduct of a job finishing.

The design builds on existing Flink and Iceberg capabilities. Flink watermarks track event time progress in out-of-order streams, while Iceberg snapshots provide persistent table metadata. Pinterest uses these capabilities together to derive a table-specific completeness signal and store it alongside the data.

Timeline showing late-arriving CDC events, Flink checkpoints, event time progress and the monotonically increasing finalization watermark. (Source: Pinterest Blog Post )

Tharun Kumar Reddy Polu , Engineer at Salesforce, identified the streaming and batch tradeoff in a LinkedIn comment, writing ,

The streaming vs batch tradeoff framing is what makes this land. Getting freshness without losing the guarantees batch gave us is the core challenge.

Pinterest implemented the mechanism using extensibility points in its Flink-to-Iceberg sink. Accumulators collect record statistics, while commit processors execute finalization logic around Iceberg commits. Separately, an Apache Iceberg issue discusses extending IcebergSink for downstream integrations, including custom committable metadata such as watermark information.

main wrapper for authors section

About the Author

section title

main wrapper for each author

#### Leela Kumili

Show more

Show less

#### This content is in the Apache Kafka topic

##### Related Topics:

• Development

• Architecture & Design

• Change Data Capture

• Apache Kafka

• Data Partitioning

• Partitioning

• Data Lake

• Streaming

• Apache Spark

• Event Driven Architecture

• Apache Flink

• Apache Iceberg

• Distributed Systems

• Related Editorial

• Related Sponsors AI's Physical Constraints: How AI Rewired the Data Center

• Related Sponsor PostgreSQL Built for Time-Series Scale Handle massive time-series data with PostgreSQL, without sacrificing performance. Discover how Tiger Data delivers high-speed ingest, fast queries, and scalable storage. Learn More .

The InfoQ Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

We protect your privacy.