How Meta Rebuilt Data Ingestion for Petabyte-Scale Reliability

How Meta Rebuilt Data Ingestion for Petabyte-Scale Reliability - InfoQ

Your choice regarding cookies on this site

We use cookies to optimise site functionality and give you the best possible experience.

I Accept I Do Not Accept Settings

[BT](https://www.infoq.com/int/bt/ "bt")

InfoQ Software Architects' Newsletter

A monthly overview of things you need to know as an architect or aspiring architect.

View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

Close

Live Webinar and Q&A: Rethinking AppSec: Why Compiler‑Level Security Changes the Architecture Conversation (Jun 11, 2026)Save Your Seat

Close

Toggle Navigation

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

English edition

• English edition
• Chinese edition
• Japanese edition
• French edition

[Write for InfoQ](https://www.infoq.com/write-for-infoq/ "Write for InfoQ")

Search

RegisterSign in

Unlock the full InfoQ experience

Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with content, and download exclusive resources.

Log In

or

Don't have an InfoQ account?

Register

• Stay updated on topics and peers that matter to youReceive instant alerts on the latest insights and trends.
• Quickly access free resources for continuous learningMinibooks, videos with transcripts, and training materials.
• Save articles and read at anytimeBookmark articles to read whenever youre ready.

Logo - Back to homepage

NewsArticlesPresentationsPodcastsGuides

Topics

[Development](https://www.infoq.com/development/ "Development")

• [Java](https://www.infoq.com/java/ "Java")
• [Kotlin](https://www.infoq.com/kotlin/ "Kotlin")
• [.Net](https://www.infoq.com/dotnet/ ".Net")
• [C#](https://www.infoq.com/c_sharp/ "C#")
• [Swift](https://www.infoq.com/swift/ "Swift")
• [Go](https://www.infoq.com/golang/ "Go")
• [Rust](https://www.infoq.com/rust/ "Rust")
• [JavaScript](https://www.infoq.com/javascript/ "JavaScript")

Featured in Development

• #### From VR to Flat Screens: Bridging the Input and Immersion Gap

Dany Lepage discusses the architectural journey of porting a hit VR title to seven non-VR platforms. He explains how his team solved the challenges of cross-progression, diverse input paradigms, and maintaining release velocity across Steam, iOS, and PlayStation. Beyond the tech, he shares candid lessons on the "product fit" gap when translating immersive social presence to 2D screens.


All in developmentFollow Topic

[Architecture & Design](https://www.infoq.com/architecture-design/ "Architecture & Design")

• [Architecture](https://www.infoq.com/architecture/ "Architecture")
• [Enterprise Architecture](https://www.infoq.com/enterprise-architecture/ "Enterprise Architecture")
• [Scalability/Performance](https://www.infoq.com/performance-scalability/ "Scalability/Performance")
• [Design](https://www.infoq.com/design/ "Design")
• [Case Studies](https://www.infoq.com/Case_Study/ "Case Studies")
• [Microservices](https://www.infoq.com/microservices/ "Microservices")
• [Service Mesh](https://www.infoq.com/servicemesh/ "Service Mesh")
• [Patterns](https://www.infoq.com/DesignPattern/ "Patterns")
• [Security](https://www.infoq.com/Security/ "Security")

Featured in Architecture & Design

• #### Context is the Key to the Agentic Architecture Revolution: a Conversation with Baruch Sadogursky

Michael Stiefel spoke to Baruch Sadogursky about software architecture in the age of agentic AI. LLM can function, albeit stochastically, as reasoning machines capable of interpreting human ambiguity. With the appropriate rigorous context artifacts to control the LLM’s reasoning, software specifications can become the source of truth, while the code becomes a disposable intermediate language.


All in architecture-designFollow Topic

[AI Infrastructure](https://www.infoq.com/ai-ml-data-eng/ "AI Infrastructure")

• [Big Data](https://www.infoq.com/bigdata/ "Big Data")
• [Machine Learning](https://www.infoq.com/machinelearning/ "Machine Learning")
• [NoSQL](https://www.infoq.com/nosql/ "NoSQL")
• [Database](https://www.infoq.com/database/ "Database")
• [Data Analytics](https://www.infoq.com/data-analytics/ "Data Analytics")
• [Streaming](https://www.infoq.com/streaming/ "Streaming")

Featured in AI, ML & Data Engineering

• #### Building Evals for AI Adoption: From Principles to Practice

Mallika Rao discusses the hidden risk of evaluation debt in production AI systems, drawing on her experience at Twitter, Walmart, and Netflix. She explains why traditional metrics fail modern architectures, breaks down a five-layer evaluation stack spanning infrastructure and UX, and shares a diagnostic maturity model to help engineering leaders eliminate silent semantic failures.


All in ai-ml-data-engFollow Topic

[Culture & Methods](https://www.infoq.com/culture-methods/ "Culture & Methods")

• [Agile](https://www.infoq.com/agile/ "Agile")
• [Diversity](https://www.infoq.com/diversity/ "Diversity")
• [Leadership](https://www.infoq.com/leadership/ "Leadership")
• [Lean/Kanban](https://www.infoq.com/lean/ "Lean/Kanban")
• [Personal Growth](https://www.infoq.com/personal-growth/ "Personal Growth")
• [Scrum](https://www.infoq.com/scrum/ "Scrum")
• [Sociocracy](https://www.infoq.com/sociocracy/ "Sociocracy")
• [Software Craftmanship](https://www.infoq.com/software_craftsmanship/ "Software Craftmanship")
• [Team Collaboration](https://www.infoq.com/team-collaboration/ "Team Collaboration")
• [Testing](https://www.infoq.com/testing/ "Testing")
• [UX](https://www.infoq.com/ux/ "UX")

Featured in Culture & Methods

• #### From Founding Engineer to CTO to CEO – At the Same Startup

Trisha Ballakur discusses her journey from a backend software engineer to CTO and CEO, using her startup Pointz as a case study. She explains how to implement bottom-up customer discovery to find product-market fit, effectively delegate to global contractors to reduce build times, customize open-source repos like Valhalla, and apply engineering test-case models to business development.


All in culture-methodsFollow Topic

DevOps

• [Infrastructure](https://www.infoq.com/infrastructure/ "Infrastructure")
• [Continuous Delivery](https://www.infoq.com/continuous_delivery/ "Continuous Delivery")
• [Automation](https://www.infoq.com/automation/ "Automation")
• [Containers](https://www.infoq.com/containers/ "Containers")
• [Cloud](https://www.infoq.com/cloud-computing/ "Cloud")
• [Observability](https://www.infoq.com/observability/ "Observability")

Featured in DevOps

• #### Realtime and Batch Processing of GPU Workloads

Joseph Stein discusses engineering an enterprise AI-as-a-Service platform within a private cloud data center. He explains how to maximize underutilized GPU pools via multi-namespace scheduling, leverage Valkey and Lua for atomic priority queuing and backpressure management, mitigate OWASP Top 10 LLM risks via central proxy gateways, and scale batch pipelines using a custom S3-to-Kafka proxy.


All in devopsFollow Topic

[Events](https://events.infoq.com/ "Events")

Helpful links

• [About InfoQ](https://www.infoq.com/about-infoq "About InfoQ")
• [InfoQ Editors](https://www.infoq.com/infoq-editors "InfoQ Editors")
• [Write for InfoQ](https://www.infoq.com/write-for-infoq "Write for InfoQ")
• [About C4Media](https://c4media.com/ "About C4Media")
• [Diversity](https://c4media.com/diversity "Diversity")

Choose your language

• [En](https://www.infoq.com/news/2026/05/meta-cdc-migration/# "InfoQ English")
• 中文
• 日本
• Fr


[InfoQ Homepage](https://www.infoq.com/ "InfoQ Homepage")[News](https://www.infoq.com/news "News")How Meta Rebuilt Data Ingestion for Petabyte-Scale Reliability

[AI, ML & Data Engineering](https://www.infoq.com/ai-ml-data-eng/ "AI, ML & Data Engineering")

Rethinking AppSec: Why Compiler‑Level Security Changes the Architecture Conversation (Webinar Jun 11th)

How Meta Rebuilt Data Ingestion for Petabyte-Scale Reliability

May 30, 2026 2 min read

by

• Renato Losio

Follow Cloud Expert | AWS Data Hero

#### Write for InfoQ

Feed your curiosity.Help 550k+ global

senior developers

each month stay ahead.Get in touch

Log in to listen to this article

Loading audio

Audio 2

0:00 0:00

Normal 1.25x 1.5x

Like

• Reading list

The engineering team at Meta recently outlined how the company migrated a data ingestion platform that transfers several petabytes of MySQL social graph data daily to improve reliability and operational efficiency. The team used techniques like reverse shadowing and continuous checksum monitoring to ensure zero downtime during the transition.

Meta operates one of the world’s largest MySQL deployments, with a data ingestion platform that supports analytics, reporting, machine learning, and internal product development workloads. The company recently redesigned its architecture, replacing customer-owned pipelines with a centralized, self-managed warehouse service.

With the migration, Meta replaced fragmented, pipeline-owned infrastructure with a centralized managed system, using staged migrations, automated validation, rollback controls, and compatibility layers to transition thousands of ingestion pipelines without disrupting downstream analytics and ML workloads.

Deploying distributed systems canarying at massive scale, Meta migrated ingestion jobs through three stages: a shadow phase that validated the new system against production data, a reverse shadow phase that swapped production ownership while preserving rollback capability, and a cleanup phase that retired the legacy pipeline after consistency and performance checks passed. Zihao Tao, software engineer at Meta, and colleagues from the engineering team explain:

We continuously monitored row count and checksum mismatches between the production jobs and the shadow jobs. When mismatches occurred, we quickly investigated the root cause and deployed fixes to the pre-production environment, then verified that the mismatch was resolved. During this step, we also measured the compute and storage quotas for the shadow jobs to ensure that the production environment had sufficient resources before proceeding.

/filters:no_upscale()/news/2026/05/meta-cdc-migration/en/resources/1Migrating-Data-Ingestion-Systems-at-Meta-Scale-image-1-e1778517437665-1779134836589.png)

_Source: Meta engineering blog_

Having now completed the migration of the entire data ingestion workload and retired the legacy system, the team acknowledges the challenge of the large-scale infrastructure transition:

Ensuring a seamless migration meant we had to effectively track the migration lifecycle for thousands of jobs and put robust rollout and rollback controls in place to handle issues that might arise during the migration process.

Each migration job had to be validated against strict correctness and performance checks before rollout, comparing row counts and checksums between old and new systems, monitoring latency and resource usage for regressions, and applying additional requirements for critical tables used by dependent teams. The team explains:

Both our legacy and new data ingestion systems used change data capture (CDC) to incrementally ingest data into the target table. Each data ingestion job has its own internal table for a full dump of source databases (full dump), an internal table for capturing changes of source databases (delta), and the target table consumed by the data customers. All the information about job entities, including table names and table schemas, is saved and managed by the central management service.

_/filters:no\_upscale()/news/2026/05/meta-cdc-migration/en/resources/1Migrating-Data-Ingestion-Systems-at-Meta-Scale-image-2-1779134836589.jpg)_

_Source: Meta engineering blog_

Syed Moeen Kazmi comments:

Migrating data ingestion at Meta scale isn't an upgrade. It's open-heart surgery on core business. The challenge isn't just moving data, it's maintaining consistency and zero downtime.

Because the CDC architecture relied on expensive full snapshots for initial loads and post-fix recovery, Meta minimized the creation of unnecessary shadow jobs until data quality issues were resolved. This avoided repeated large-scale full dumps and significantly improved migration efficiency. The team also reduced infrastructure load by reusing snapshot partitions from the legacy system during initial migration stages.

About the Author

#### Renato Losio

Renato has extensive experience as a cloud architect, advisor, and cloud services specialist. Currently, he lives in Berlin and works remotely as a principal cloud architect. His primary areas of interest include cloud services and relational databases. He is an editor at InfoQ and a recognized AWS Data Hero. You can connect with him on LinkedIn.

Show more Show less

#### This content is in the AI, ML & Data Engineering topic

Follow Topic

##### Related Topics:

• Architecture & Design### Architecture & Design

Followers: 10247

Follow Topic

• AI, ML & Data Engineering### AI, ML & Data Engineering

Followers: 5929

Follow Topic

• Big Data Infrastructure### Big Data Infrastructure

Followers: 36

Follow Topic

• migration### migration

Followers: 12

Follow Topic

• Facebook### Facebook

Followers: 17

Follow Topic

• MySQL### MySQL

Followers: 37

Follow Topic

• Scalability### Scalability

Followers: 103

Follow Topic

* #### Popular in AI, ML & Data Engineering

• ##### Designing a Multi-Agent System for Engineering Support at Scale: a Case Study from Grab

• ##### AWS MCP Server Reaches GA with Full API Coverage and IAM-Based Governance

• ##### Google Introduces Middleware Architecture for Genkit Applications

• ##### GitHub Slashes Agent Workflow Token Spend up to 62% with Daily Audits and MCP Pruning

• ##### Gemma 4 Multi-Token Prediction Delivers up to ~3x Faster Token Generation

* #### Related Sponsors

• ##### Akka Launches Agentic Platform for Autonomous, Real-Time & Edge AI Systems

• ##### Deliver Apps right from UI. Try WaveMaker Design-to-Code. Scale production with your own Design Systems.

• ##### Gartner® Research: Hype Cycle™ for Cloud Platform Services, 2025

• ##### Architecting for What’s Next: Key Software Trends Every Senior Practitioner Must Act On — Download the InfoQ eMag

• ##### Design for Failure: How to Guarantee Data Access During Cloud Outages

• #### Related Sponsor

Intelligent Cloud Infrastructure for your backup, data lakes, and AI. Teams from SoFi, Red Bull, and Structured Web use Eon to streamline backup, slash recovery time, and turn their data into live, searchable assets while reducing backup costs by up to 50%. [Learn more now >](https://www.infoq.com/url/f/9e2cbd91-4347-4ff1-bf4b-c882f3ee5045/)

Related Content

• ##### Bintrail: MySQL Time-Travel Queries Using Indexed Binlogs

May 21, 2026

• ##### MySQL 9.7: First Major LTS Since 8.4 Brings Enterprise Features to Community Edition

May 10, 2026

• ##### AWS Improves Aurora Serverless: 45% Faster Ramp-Up, 30% Higher Throughput

May 09, 2026

• ##### Cloudflare Builds High-Performance Infrastructure for Running LLMs

May 03, 2026

• Icon##### Architecting Autonomy: Decentralising Architecture Inside an Organization

May 15, 2026 

• ##### Cloudflare Introduces Workflows V2 with Deterministic Execution and 50K Concurrent Workflows

May 15, 2026

• Icon##### Realtime and Batch Processing of GPU Workloads

May 26, 2026 

• Icon##### The AI Gateway: Scaling Centralized Inference across Decentralized Teams

May 20, 2026 

• Icon##### Using AI as a Thinking Partner for Large-Scale Engineering Systems

May 15, 2026 

Related Sponsors

• #### Multi-Cloud Backup Is Broken: Architectural Patterns That Actually Work

Learn how to navigate multi-cloud backup challenges across AWS, Azure, and Google Cloud. By addressing tool fragmentation and siloed data, teams can ensure consistent policies, reduce costs, and maintain seamless recovery across all providers.

• #### How an AI Agent Deleted Production Data and Its Backups at a Company (and How to Protect Yours)

AI agents can trigger catastrophic data loss by deleting production and backups using valid credentials. This article explains why traditional backup models fail under autonomous systems and how isolated, immutable recovery layers prevent AI‑driven outages.

• Sponsored by


Related Content

• ##### Applying Best Simple System for Now for Software Design

May 07, 2026

• Icon##### The Human Scalability Problem: Why Your Teams Don’t Scale Like Your Code

May 04, 2026 

• Icon##### Building Embedding Models for Large-Scale Real-World Applications

Feb 13, 2026 

• Icon##### Scaling to 100+ as a Director: Lessons from Growing Engineering Organizations

Feb 11, 2026 

• Icon##### Beyond the Warehouse: Why BigQuery Alone Won’t Solve Your Data Problems

Feb 04, 2026 

• Icon##### Theme Systems at Scale: How to Build Highly Customizable Software

Jan 26, 2026 

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

• Development

• ##### [Pip 26.1 Ships Dependency Cooldowns and Experimental Lockfile Support to Combat Supply Chain Attacks](https://www.infoq.com/news/2026/05/pip-261-dependency-cooldowns/ "Pip 26.1 Ships Dependency Cooldowns and Experimental Lockfile Support to Combat Supply Chain Attacks")

• ##### [Cloudflare and Stripe Let AI Agents Create Accounts, Buy Domains, and Deploy to Production](https://www.infoq.com/news/2026/05/cloudflare-stripe-agent-commerce/ "Cloudflare and Stripe Let AI Agents Create Accounts, Buy Domains, and Deploy to Production")

• ##### [Google Introduces Cloud Fraud Defense as Successor to reCAPTCHA](https://www.infoq.com/news/2026/05/cloud-fraud-defense-recaptcha/ "Google Introduces Cloud Fraud Defense as Successor to reCAPTCHA")

• Architecture & Design

• ##### [How LinkedIn Identified a Kernel Lock Contention Issue Causing Recurring System Freezes](https://www.infoq.com/news/2026/05/linkedin-kernel-lock-freeze/ "How LinkedIn Identified a Kernel Lock Contention Issue Causing Recurring System Freezes")

• ##### [Uber Improves Restaurant Recommendations Using Real-Time Signals and Listwise Ranking](https://www.infoq.com/news/2026/05/uber-eats-ranking-system/ "Uber Improves Restaurant Recommendations Using Real-Time Signals and Listwise Ranking")

• ##### [Designing a Multi-Agent System for Engineering Support at Scale: a Case Study from Grab](https://www.infoq.com/news/2026/05/grab-multi-agent-support-system/ "Designing a Multi-Agent System for Engineering Support at Scale: a Case Study from Grab")

• Culture & Methods

• ##### [From Founding Engineer to CTO to CEO – At the Same Startup](https://www.infoq.com/presentations/framework-best-practices-startup/ "From Founding Engineer to CTO to CEO – At the Same Startup")

• ##### [Accountability is the Goal for AI, with EU Regulations Supporting Transparency](https://www.infoq.com/news/2026/05/accountability-AI-EU-regulations/ "Accountability is the Goal for AI, with EU Regulations Supporting Transparency")

• ##### [From Legacy to Sovereignty: Driving the Future of Insurance through Platform Engineering](https://www.infoq.com/presentations/insurance-platform-engineering/ "From Legacy to Sovereignty: Driving the Future of Insurance through Platform Engineering")

• AI, ML & Data Engineering

• ##### [How Meta Rebuilt Data Ingestion for Petabyte-Scale Reliability](https://www.infoq.com/news/2026/05/meta-cdc-migration/ "How Meta Rebuilt Data Ingestion for Petabyte-Scale Reliability")

• ##### [Building Evals for AI Adoption: From Principles to Practice](https://www.infoq.com/presentations/eval-ai-adoption/ "Building Evals for AI Adoption: From Principles to Practice")

• ##### [Designing AI Platforms for Reliability: Tools for Certainty, Agents for Discovery](https://www.infoq.com/presentations/ai-platforms-reliability/ "Designing AI Platforms for Reliability: Tools for Certainty, Agents for Discovery")

• DevOps

• ##### [Arm Open-Sources Metis, an AI Security Framework Outperforming Traditional SAST Tools](https://www.infoq.com/news/2026/05/arm-metis-agentic-security/ "Arm Open-Sources Metis, an AI Security Framework Outperforming Traditional SAST Tools")

• ##### [AI-Assisted Migration Tool Helps Teams Move from ingress-nginx to Higress in Minutes](https://www.infoq.com/news/2026/05/ai-nginx-higress/ "AI-Assisted Migration Tool Helps Teams Move from ingress-nginx to Higress in Minutes")

• ##### [GitHub Slashes Agent Workflow Token Spend up to 62% with Daily Audits and MCP Pruning](https://www.infoq.com/news/2026/05/github-agentic-token-savings/ "GitHub Slashes Agent Workflow Token Spend up to 62% with Daily Audits and MCP Pruning")

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

• Get a quick overview of content published on a variety of innovator and early adopter technologies
• Learn what you don’t know that you don’t know
• Stay up to date with the latest information from the topics you are interested in

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

**ONLINE INFOQ CERTIFICATION PROGRAM** A Cohort for Senior Engineers and Architects * **Focused on ARCHITECTURE** with Luca Mezzalira | JUNE 10 * **Focused on AI ENGINEERING** with Hien Luu | JULY 25 Bring a real architecture or AI engineering challenge from your work. Spend 5 weeks pressure-testing your approach with senior peers from other companies and experienced facilitators. Explore the upcoming cohorts. **Register Now.**

#### Events

• ##### Online InfoQ Architect Certification

June 10, 2026

• ##### Online InfoQ Organizational Architect Certification

June 19, 2026

• ##### Online InfoQ AI Engineering Certification

July 25, 2026

• ##### QCon San Francisco

November 16-20, 2026

• ##### QCon London 2027

April 13-16, 2027

#### Follow us on

Youtube 232K FollowersLinkedin 26K FollowersInstagram NewRSS 19K ReadersX 57.1k FollowersFacebook 21K LikesBluesky New

#### Stay in the know

The InfoQ PodcastEngineering Culture PodcastThe Software Architects' Newsletter

General Feedback [feedback@infoq.com](mailto:feedback@infoq.com) Advertising [sales@infoq.com](mailto:sales@infoq.com) Editorial [editors@infoq.com](mailto:editors@infoq.com) Marketing [marketing@infoq.com](mailto:marketing@infoq.com)

InfoQ.com and all content copyright © 2006-2026 C4Media Inc.

Privacy Notice, Terms And Conditions, Cookie Policy

Close

[BT](https://www.infoq.com/int/bt/ "bt")