Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance

Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance - InfoQ

Your choice regarding cookies on this site

We use cookies to optimise site functionality and give you the best possible experience.

I Accept I Do Not Accept Settings

[BT](http://www.infoq.com/int/bt/ "bt")

InfoQ Software Architects' Newsletter

A monthly overview of things you need to know as an architect or aspiring architect.

View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

Close

Live Webinar and Q&A: Architecting for Autonomous Reliability: Embedding AI into Your Observability Stack (Jun 25, 2026)Save Your Seat

Close

Toggle Navigation

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

English edition

• English edition
• Chinese edition
• Japanese edition
• French edition

[Write for InfoQ](http://www.infoq.com/write-for-infoq/ "Write for InfoQ")

Search

RegisterSign in

Unlock the full InfoQ experience

Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with content, and download exclusive resources.

Log In

or

Don't have an InfoQ account?

Register

• **Stay updated on topics and peers that matter to you**Receive instant alerts on the latest insights and trends.
• **Quickly access free resources for continuous learning**Minibooks, videos with transcripts, and training materials.
• **Save articles and read at anytime**Bookmark articles to read whenever youre ready.

Logo - Back to homepage

NewsArticlesPresentationsPodcastsGuides

Topics

[Development](http://www.infoq.com/development/ "Development")

• [Java](http://www.infoq.com/java/ "Java")
• [Kotlin](http://www.infoq.com/kotlin/ "Kotlin")
• [.Net](http://www.infoq.com/dotnet/ ".Net")
• [C#](http://www.infoq.com/c_sharp/ "C#")
• [Swift](http://www.infoq.com/swift/ "Swift")
• [Go](http://www.infoq.com/golang/ "Go")
• [Rust](http://www.infoq.com/rust/ "Rust")
• [JavaScript](http://www.infoq.com/javascript/ "JavaScript")

Featured in Development

• #### From VR to Flat Screens: Bridging the Input and Immersion Gap

Dany Lepage discusses the architectural journey of porting a hit VR title to seven non-VR platforms. He explains how his team solved the challenges of cross-progression, diverse input paradigms, and maintaining release velocity across Steam, iOS, and PlayStation. Beyond the tech, he shares candid lessons on the "product fit" gap when translating immersive social presence to 2D screens.


All in developmentFollow Topic

[Architecture & Design](http://www.infoq.com/architecture-design/ "Architecture & Design")

• [Architecture](http://www.infoq.com/architecture/ "Architecture")
• [Enterprise Architecture](http://www.infoq.com/enterprise-architecture/ "Enterprise Architecture")
• [Scalability/Performance](http://www.infoq.com/performance-scalability/ "Scalability/Performance")
• [Design](http://www.infoq.com/design/ "Design")
• [Case Studies](http://www.infoq.com/Case_Study/ "Case Studies")
• [Microservices](http://www.infoq.com/microservices/ "Microservices")
• [Service Mesh](http://www.infoq.com/servicemesh/ "Service Mesh")
• [Patterns](http://www.infoq.com/DesignPattern/ "Patterns")
• [Security](http://www.infoq.com/Security/ "Security")

Featured in Architecture & Design

• #### Stripe’s Docdb: How Zero-Downtime Data Movement Powers Trillion-Dollar Payment Processing

Jimmy Morzaria discusses the evolution of Stripe’s database tier to support 5 million QPS with 5.5 nines of reliability. He explains the architecture of DocDB and shares how Stripe leverages a custom zero-downtime data movement platform to perform horizontal sharding, version upgrades, and multi-tenant migrations - all while maintaining the strict consistency required for global commerce.


All in architecture-designFollow Topic

[AI Infrastructure](http://www.infoq.com/ai-ml-data-eng/ "AI Infrastructure")

• [Big Data](http://www.infoq.com/bigdata/ "Big Data")
• [Machine Learning](http://www.infoq.com/machinelearning/ "Machine Learning")
• [NoSQL](http://www.infoq.com/nosql/ "NoSQL")
• [Database](http://www.infoq.com/database/ "Database")
• [Data Analytics](http://www.infoq.com/data-analytics/ "Data Analytics")
• [Streaming](http://www.infoq.com/streaming/ "Streaming")

Featured in AI, ML & Data Engineering

• #### The Next Generation of AI Products

Hilary Mason shares her journey from academia to building AI products at scale. She discusses the shift from discrete engineering to probabilistic mindsets, explaining why managing "human considerations" is the hardest part of the stack. She explains the "existential crisis" for engineers, arguing that great architecture today is about context management, systems thinking, and good taste.


All in ai-ml-data-engFollow Topic

[Culture & Methods](http://www.infoq.com/culture-methods/ "Culture & Methods")

• [Agile](http://www.infoq.com/agile/ "Agile")
• [Diversity](http://www.infoq.com/diversity/ "Diversity")
• [Leadership](http://www.infoq.com/leadership/ "Leadership")
• [Lean/Kanban](http://www.infoq.com/lean/ "Lean/Kanban")
• [Personal Growth](http://www.infoq.com/personal-growth/ "Personal Growth")
• [Scrum](http://www.infoq.com/scrum/ "Scrum")
• [Sociocracy](http://www.infoq.com/sociocracy/ "Sociocracy")
• [Software Craftmanship](http://www.infoq.com/software_craftsmanship/ "Software Craftmanship")
• [Team Collaboration](http://www.infoq.com/team-collaboration/ "Team Collaboration")
• [Testing](http://www.infoq.com/testing/ "Testing")
• [UX](http://www.infoq.com/ux/ "UX")

Featured in Culture & Methods

• #### Panel: Building a Culture that Works

The panelists share insights on evolving company culture. They discuss leveraging feedback loops, lending social capital, and the friction between legacy bureaucracy and agile engineering. The panel explains how to maintain cohesion in remote teams and use interviews to uncover the true "unmanicured" culture of a firm.


All in culture-methodsFollow Topic

DevOps

• [Infrastructure](http://www.infoq.com/infrastructure/ "Infrastructure")
• [Continuous Delivery](http://www.infoq.com/continuous_delivery/ "Continuous Delivery")
• [Automation](http://www.infoq.com/automation/ "Automation")
• [Containers](http://www.infoq.com/containers/ "Containers")
• [Cloud](http://www.infoq.com/cloud-computing/ "Cloud")
• [Observability](http://www.infoq.com/observability/ "Observability")

Featured in DevOps

• #### AI-Powered SRE for Autonomous Incident Response

The presenters discuss incident response, how AI-enhanced SRE platforms connect signals from logs, metrics, traces, and historical incidents to enable autonomous decisions.


All in devopsFollow Topic

[Events](https://events.infoq.com/ "Events")

Helpful links

• [About InfoQ](http://www.infoq.com/about-infoq "About InfoQ")
• [InfoQ Editors](http://www.infoq.com/infoq-editors "InfoQ Editors")
• [Write for InfoQ](http://www.infoq.com/write-for-infoq "Write for InfoQ")
• [About C4Media](https://c4media.com/ "About C4Media")
• [Diversity](https://c4media.com/diversity "Diversity")

Choose your language

• [En](http://www.infoq.com/news/2026/05/confluent-kafka-header-schema-id/# "InfoQ English")
• 中文
• 日本
• Fr


[InfoQ Homepage](http://www.infoq.com/ "InfoQ Homepage")[News](http://www.infoq.com/news "News")Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance

[Architecture & Design](http://www.infoq.com/architecture-design/ "Architecture & Design")

Portable by Design: Data Mobility & Recovery Patterns for Multi-Cloud Systems (Webinar May 21st)

Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance

May 01, 2026 2 min read

by

• Leela Kumili

Follow Lead Engineer

#### Write for InfoQ

**Feed your curiosity.**Help 550k+ global

senior developers

each month stay ahead.Get in touch

Log in to listen to this article

Audio ready to play

Your browser does not support the audio element.

0:00 0:00

Normal 1.25x 1.5x

Like

• Reading list

Confluent has introduced a new approach to managing schema metadata in Apache Kafka by enabling schema IDs to be stored in message headers rather than in the payload. The update is designed to simplify data governance and enable teams to adopt schema validation without changing existing event formats. The feature builds on Kafka’s native header support and integrates with Confluent Schema Registry, which is widely used by organizations managing event-driven architectures across microservices, analytics pipelines, and data platforms.

In traditional Kafka deployments using Confluent’s wire format, schema IDs are embedded directly in the message payload. This ensures consumers can correctly deserialize events, but it tightly couples schema metadata with the data itself. Over time, this coupling complicates schema evolution, especially in environments where multiple teams and systems consume the same event streams. It also increases coordination overhead when schema changes are introduced across producers and consumers.

With the new approach, schema identifiers are stored in Kafka record headers while the payload remains unchanged. Consumers retrieve the schema from the schema registry at runtime using the ID in the header. This maintains compatibility with formats such as Avro, Protobuf, and JSON Schema while reducing dependence on tightly coupled wire formats. Schema resolution is decoupled from the payload, making event streams more flexible and easier to integrate across downstream systems and tooling.

!Image 12/filters:no_upscale()/news/2026/05/confluent-kafka-header-schema-id/en/resources/1conflunentheader-1776737160055.jpeg)

_Schema handling before and after moving schema IDs to Kafka headers (Source: Confluent Blog Post)_

Patrick Neff, CSTA Team Lead CEMEA at Confluent, highlights the importance of schema governance in enabling reuse across streaming and analytics systems in a LinkedIn post.

Schemas are the key enabler for unlocking the full value of your data.

The header-based approach also supports incremental adoption. Organizations can introduce schema governance without large-scale rewrites or coordinated changes across all producers and consumers. Schema IDs can be attached to existing event streams, allowing teams to gradually adopt stricter schema management practices while maintaining backward compatibility.

Gunnar Morling, Technologist at Confluent, emphasized improved interoperability with storage systems and downstream processing frameworks in post.

Schema ids into Kafka message headers rather than the message payload is a massive quality of life improvement: payloads become valid, self-contained.

Separating schema metadata from payloads enables independent evolution of producers and consumers, with validation centralized in the schema registry. This reduces coordination overhead and simplifies schema evolution at scale. It also improves interoperability with tools like Apache Flink and analytics or ML systems by enabling consistent reuse of structured event data across pipelines.

David Araujo, Director of Product Management at Coflunent, describes how the feature enables zero downtime and client-independent adoption patterns.

By moving schema IDs to headers, you can attach schemas to existing data in Kafka without touching payload formats.

The transition may require updates to Kafka connectors and downstream tools that assume schema metadata is embedded in payloads, creating a period where both approaches may coexist, depending on ecosystem readiness. The feature is available in Confluent Cloud and is expected in Confluent Platform with Schema Registry support under existing licensing models.

About the Author


#### **Leela Kumili**

Leela is a Lead Software Engineer at Starbucks with deep expertise in building scalable, cloud-native systems and distributed platforms. She drives architecture, delivery, and operational excellence across the Rewards Platform, leading efforts to modernize systems, improve scalability, and enhance reliability. In addition to her technical leadership, Leela serves as an AI Champion for the organization, identifying opportunities to improve developer productivity and workflows using LLM-based tools and establishing best practices for AI adoption. She is passionate about building production-ready systems, enhancing developer experience, and mentoring engineers to grow in both technical and strategic impact. Her interests include platform engineering, distributed systems, developer productivity, and bridging technical solutions with business and product goals.

Show more Show less

#### This content is in the Machine Learning topic

Follow Topic

##### Related Topics:

• Development### Development

Followers: 4093

Follow Topic

• Architecture & Design### Architecture & Design

Followers: 10214

Follow Topic

• Protocol Buffers### Protocol Buffers

Followers: 24

Follow Topic

• Schema### Schema

Followers: 3

Follow Topic

• Machine Learning### Machine Learning

Followers: 14444

Follow Topic

• Data Analytics### Data Analytics

Followers: 255

Follow Topic

• Apache Flink### Apache Flink

Followers: 36

Follow Topic

• Data Pipelines### Data Pipelines

Followers: 63

Follow Topic

• JSON### JSON

Followers: 5

Follow Topic

• Apache Kafka### Apache Kafka

Followers: 305

Follow Topic

• Event Stream Processing### Event Stream Processing

Followers: 94

Follow Topic

• Avro### Avro

Followers: 1

Follow Topic

• Streaming### Streaming

Followers: 1127

Follow Topic

* #### Related Editorial

• ##### AWS Lambda Gains Native Avro and Protobuf Support for Kafka Events with Schema Registry Integration

• ##### Zod v4 Available with Major Performance Improvements and Introduction of Zod Mini

• ##### Helidon 4.4.0 Introduces Alignment with OpenJDK Cadence and Support via Java Verified Portfolio

• ##### Vercel Releases JSON-Render: a Generative UI Framework for AI-Driven Interface Composition

• ##### Agoda’s API Agent Converts Any API to MCP with Zero Code and Deployments

* #### Related Sponsors

• ##### AI Agent Swarm Patterns (InfoQ Webinar) - Watch Now On-Demand

• ##### Design for Failure: How to Guarantee Data Access During Cloud Outages

• ##### 8 Deployment Pattern Structures to Transform your CI/CD

• ##### The Essential Guide to AI Tools for Jakarta EE Developers

• ##### Why APIs Can’t Trust Clients—and How to Bridge the Gap

• #### Related Sponsor

**Test. Protect. Repeat. Guardsquare pairs mobile app testing and protection, delivering max security with zero performance trade-offs. Request a Quote.**

Related Content

• ##### Helidon 4.4.0 Introduces Alignment with OpenJDK Cadence and Support via Java Verified Portfolio

Apr 02, 2026

• Icon##### From Confusion to Clarity: Advanced Observability Strategies for Media Workflows at Netflix

Jan 02, 2026 

• ##### Building Streaming Infrastructure That Scales: Because Viewers Won't Wait until Tomorrow

Dec 23, 2025 

• ##### Uber’s Hive Federation Decentralizes 16K Datasets and 10+ PB for Zero-Downtime Analytics at Scale

Apr 09, 2026

• ##### Legare Kerrison and Cedric Clyburn on LLM Performance and Evaluations

Apr 28, 2026

• ##### Spring News Roundup: First Release Candidates of Boot, Security, Integration, Modulith, AMQP

Apr 27, 2026

• ##### Orchestrating Agentic and Multimodal AI Pipelines with Apache Camel

Apr 24, 2026 

• ##### Grafana Rearchitects Loki with Kafka and Ships a CLI to Bring Observability into Coding Agent

Apr 23, 2026

• ##### Lyft Scales Global Localization Using AI and Human-in-the-Loop Review

Apr 13, 2026

Related Sponsors

• #### The Case for Real-Time Threat Monitoring and Analysis in Modern Mobile App Security

Drive better mobile security with real‑time insights. This Guardsquare report shows why traditional client‑side defenses fall short against persistent threats and how continuous threat monitoring and analysis gives teams actionable visibility to protect apps, users, and revenue.

• #### Rethinking AppSec: Why Compiler‑Level Security Changes the Architecture Conversation (Live Webinar Jun 11th) - Save Your Seat

Security bolted on after the build process adds fragility and blind spots. Embedding protection at compile time improves performance and resilience. This session compares wrapper, runtime, and compiler approaches to help you choose where security belongs in your SDLC.

• Sponsored by


Related Content

• Icon##### Reimagining Platform Engagement with Graph Neural Networks

Apr 13, 2026 

• ##### Cloudflare and ETH Zurich Outline Approaches for AI-Driven Cache Optimization

Apr 08, 2026

• ##### Inside Spotify’s 2025 Wrapped Archive: AI Narratives at Scale and the Privacy Trade‑Off

Apr 08, 2026

• Icon##### Building Embedding Models for Large-Scale Real-World Applications

Feb 13, 2026 

• ##### Why Most Machine Learning Projects Fail to Reach Production

Feb 02, 2026 

• Icon##### How to Unlock Insights and Enable Discovery within Petabytes of Autonomous Driving Data

Jan 21, 2026 

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

• Development

• ##### [QCon San Francisco 2026: 12 Tracks Announced](http://www.infoq.com/news/2026/04/qconsf-2026-tracks-announced/ "QCon San Francisco 2026: 12 Tracks Announced")

• ##### [Microsoft's Russinovich and Hanselman Warn AI Is Hollowing out the Junior Developer Pipeline](http://www.infoq.com/news/2026/04/junior-developer-pipeline-crisis/ "Microsoft's Russinovich and Hanselman Warn AI Is Hollowing out the Junior Developer Pipeline")

• ##### [C++26: Reflection, Memory Safety, Contracts, and a New Async Model](http://www.infoq.com/news/2026/04/cpp-26-reflection-safety-async/ "C++26: Reflection, Memory Safety, Contracts, and a New Async Model")

• Architecture & Design

• ##### [Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance](http://www.infoq.com/news/2026/05/confluent-kafka-header-schema-id/ "Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance")

• ##### [Meta's Approach to Migrating their Systems to Post-Quantum Cryptography](http://www.infoq.com/news/2026/04/meta-quantum-crypto-migration/ "Meta's Approach to Migrating their Systems to Post-Quantum Cryptography")

• ##### [Stripe’s Docdb: How Zero-Downtime Data Movement Powers Trillion-Dollar Payment Processing](http://www.infoq.com/presentations/docdb-online-database/ "Stripe’s Docdb: How Zero-Downtime Data Movement Powers Trillion-Dollar Payment Processing")

• Culture & Methods

• ##### [Driving and Measuring the Impact of Platform Engineering](http://www.infoq.com/news/2026/04/measure-platform-engineering/ "Driving and Measuring the Impact of Platform Engineering")

• ##### [How Observability and Telemetry Can Enhance the Practice of Software Engineering](http://www.infoq.com/news/2026/04/observability-telemetry/ "How Observability and Telemetry Can Enhance the Practice of Software Engineering")

• ##### [Panel: Building a Culture that Works](http://www.infoq.com/presentations/panel-positive-culture/ "Panel: Building a Culture that Works")

• AI, ML & Data Engineering

• ##### [The Next Generation of AI Products](http://www.infoq.com/presentations/ai-products/ "The Next Generation of AI Products")

• ##### [NVIDIA Launches Ising Open Models for Quantum Computing](http://www.infoq.com/news/2026/04/nvidia-ising-quantum/ "NVIDIA Launches Ising Open Models for Quantum Computing")

• ##### [Vercel Releases Open Agents to Support Background AI Coding Workflows](http://www.infoq.com/news/2026/04/vercel-open-agents/ "Vercel Releases Open Agents to Support Background AI Coding Workflows")

• DevOps

• ##### [Meta Deploys Unified AI Agents to Automate Performance Optimization at Hyperscale](http://www.infoq.com/news/2026/05/meta-ai-agents-hyperscale/ "Meta Deploys Unified AI Agents to Automate Performance Optimization at Hyperscale")

• ##### [Broadcom Donates Velero to CNCF, Shifting Kubernetes Backup to Community Governance](http://www.infoq.com/news/2026/05/broadcom-velero-cncf/ "Broadcom Donates Velero to CNCF, Shifting Kubernetes Backup to Community Governance")

• ##### [Dropbox Redesigns Compaction to Reclaim Space from Underfilled Storage Volumes](http://www.infoq.com/news/2026/04/dropbox-tiered-compaction/ "Dropbox Redesigns Compaction to Reclaim Space from Underfilled Storage Volumes")

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

• Get a quick overview of content published on a variety of innovator and early adopter technologies
• Learn what you don’t know that you don’t know
• Stay up to date with the latest information from the topics you are interested in

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

**May 7 | June 10, 2026 | Online** Architecture decisions are hard to validate while shipping. Join a **5-week online cohort** for **senior engineers, architects, and team leads** to pressure-test real decisions, apply practical frameworks, and work through challenges with a confidential peer group. Facilitated by Luca Mezzalira, Principal Architect at AWS, this cohort helps you: * Pressure-test real decisions. * Apply frameworks to real problems. * Publish on InfoQ.com and earn your certification. **RESERVE YOUR PLACE**

[Home](http://www.infoq.com/ "Home")[Create account](http://www.infoq.com/reginit.action "Create account")Log In[QCon Conferences](http://qconferences.com/ "QCon Conferences")Events[Write for InfoQ](http://www.infoq.com/write-for-infoq/ "Write for InfoQ")[InfoQ Editors](http://www.infoq.com/infoq-editors/ "InfoQ Editors")[About InfoQ](http://www.infoq.com/about-infoq/ "About InfoQ")[About C4Media](https://c4media.com/ "About C4Media")[Media Kit](https://get.infoq.com/infoq-mediakit/ "Media Kit")[InfoQ Developer Marketing Blog](https://devmarketing.c4media.com/?utm_source=infoq "InfoQ Developer Marketing Blog")[Diversity](https://c4media.com/diversity "Diversity")

#### Events

• ##### Online InfoQ Architect Certification

May 7, 2026

• ##### QCon AI Boston

June 1-2, 2026

• ##### Online InfoQ Architect Certification

June 10, 2026

• ##### QCon San Francisco

November 16-20, 2026

#### Follow us on

Youtube 232K FollowersLinkedin 26K FollowersInstagram NewRSS 19K ReadersX 57.1k FollowersFacebook 21K LikesBluesky New

#### Stay in the know

The InfoQ PodcastEngineering Culture PodcastThe Software Architects' Newsletter

General Feedback [feedback@infoq.com](mailto:feedback@infoq.com) Advertising [sales@infoq.com](mailto:sales@infoq.com) Editorial [editors@infoq.com](mailto:editors@infoq.com) Marketing [marketing@infoq.com](mailto:marketing@infoq.com)

InfoQ.com and all content copyright © 2006-2026 C4Media Inc.

Privacy Notice, Terms And Conditions, Cookie Policy

Close

[BT](http://www.infoq.com/int/bt/ "bt")