How Uber Builds Zone-Failure-Resilient OpenSearch Clusters

How Uber Builds Zone-Failure-Resilient OpenSearch Clusters - InfoQ

Your choice regarding cookies on this site

We use cookies to optimise site functionality and give you the best possible experience.

I Accept I Do Not Accept Settings

[BT](https://www.infoq.com/int/bt/ "bt")

InfoQ Software Architects' Newsletter

A monthly overview of things you need to know as an architect or aspiring architect.

View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

Close

Live Webinar and Q&A: Building AI Agent Evals for High-Stakes Incident Response (Aug 6, 2026)Save Your Seat

Close

Toggle Navigation

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

English edition

• English edition
• Chinese edition
• Japanese edition
• French edition

[Write for InfoQ](https://www.infoq.com/write-for-infoq/ "Write for InfoQ")

Search

RegisterSign in

Unlock the full InfoQ experience

Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with content, and download exclusive resources.

Log In

or

Don't have an InfoQ account?

Register

• Stay updated on topics and peers that matter to youReceive instant alerts on the latest insights and trends.
• Quickly access free resources for continuous learningMinibooks, videos with transcripts, and training materials.
• Save articles and read at anytimeBookmark articles to read whenever youre ready.

Logo - Back to homepage

NewsArticlesPresentationsPodcastsGuides

Topics

[Development](https://www.infoq.com/development/ "Development")

• [Java](https://www.infoq.com/java/ "Java")
• [Kotlin](https://www.infoq.com/kotlin/ "Kotlin")
• [.Net](https://www.infoq.com/dotnet/ ".Net")
• [C#](https://www.infoq.com/c_sharp/ "C#")
• [Swift](https://www.infoq.com/swift/ "Swift")
• [Go](https://www.infoq.com/golang/ "Go")
• [Rust](https://www.infoq.com/rust/ "Rust")
• [JavaScript](https://www.infoq.com/javascript/ "JavaScript")

Featured in Development

• #### The Rust High Performance Talk You Did Not Expect

Ruth Linehan explains how migrating high-performance caching services from Kotlin to Rust shattered internal preconceptions around delivery velocity and engineering overhead. She discusses the ergonomics of the Rust borrow checker, shares how compile-time safety shortens the developer feedback loop, and profiles how tools like Criterion and flamegraphs optimize concurrent code paths.


All in developmentFollow Topic

[Architecture & Design](https://www.infoq.com/architecture-design/ "Architecture & Design")

• [Architecture](https://www.infoq.com/architecture/ "Architecture")
• [Enterprise Architecture](https://www.infoq.com/enterprise-architecture/ "Enterprise Architecture")
• [Scalability/Performance](https://www.infoq.com/performance-scalability/ "Scalability/Performance")
• [Design](https://www.infoq.com/design/ "Design")
• [Case Studies](https://www.infoq.com/Case_Study/ "Case Studies")
• [Microservices](https://www.infoq.com/microservices/ "Microservices")
• [Service Mesh](https://www.infoq.com/servicemesh/ "Service Mesh")
• [Patterns](https://www.infoq.com/DesignPattern/ "Patterns")
• [Security](https://www.infoq.com/Security/ "Security")

Featured in Architecture & Design

• #### Comprehension at AI Speed: Building a Context Store for Evolutionary Architecture

AI makes the first 80% of development feel fast, but hides architectural complexity until it's too late. To prevent system instability, engineering leaders must shift from raw throughput to systemic comprehension. By unifying spec-anchored SDD, TDD, and automated fitness functions into a repo-bound "Context Store," teams can ensure AI agents and human reviewers evolve code safely.


All in architecture-designFollow Topic

[AI Infrastructure](https://www.infoq.com/ai-ml-data-eng/ "AI Infrastructure")

• [Big Data](https://www.infoq.com/bigdata/ "Big Data")
• [Machine Learning](https://www.infoq.com/machinelearning/ "Machine Learning")
• [NoSQL](https://www.infoq.com/nosql/ "NoSQL")
• [Database](https://www.infoq.com/database/ "Database")
• [Data Analytics](https://www.infoq.com/data-analytics/ "Data Analytics")
• [Streaming](https://www.infoq.com/streaming/ "Streaming")

Featured in AI, ML & Data Engineering

• #### From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry

Ben O'Mahony discusses building custom AI-powered Language Server Protocols (LSPs) that go beyond standard rule-based checkers. He explains how to instrument AI agents natively with OpenTelemetry to track concrete user actions (accepting, dismissing, or regenerating code fixes) as implicit labels, creating a continuous data flywheel to distill frontier capabilities into cheaper, local SLMs.


All in ai-ml-data-engFollow Topic

[Culture & Methods](https://www.infoq.com/culture-methods/ "Culture & Methods")

• [Agile](https://www.infoq.com/agile/ "Agile")
• [Diversity](https://www.infoq.com/diversity/ "Diversity")
• [Leadership](https://www.infoq.com/leadership/ "Leadership")
• [Lean/Kanban](https://www.infoq.com/lean/ "Lean/Kanban")
• [Personal Growth](https://www.infoq.com/personal-growth/ "Personal Growth")
• [Scrum](https://www.infoq.com/scrum/ "Scrum")
• [Sociocracy](https://www.infoq.com/sociocracy/ "Sociocracy")
• [Software Craftmanship](https://www.infoq.com/software_craftsmanship/ "Software Craftmanship")
• [Team Collaboration](https://www.infoq.com/team-collaboration/ "Team Collaboration")
• [Testing](https://www.infoq.com/testing/ "Testing")
• [UX](https://www.infoq.com/ux/ "UX")

Featured in Culture & Methods

• #### Road to Compliance: Will Your Internal Users Hate Your Platform Team?

Davide de Paolis discusses the realities of rolling out cloud infrastructure compliance without fracturing developer relations. Drawing from a real-world platform team reboot at Sevdesk, he explains how to implement "minimum viable governance" on AWS, utilize event-driven Slack alerting to automate policy feedback, and shift from rigid enforcement to high-empathy, data-driven collaboration.


All in culture-methodsFollow Topic

DevOps

• [Infrastructure](https://www.infoq.com/infrastructure/ "Infrastructure")
• [Continuous Delivery](https://www.infoq.com/continuous_delivery/ "Continuous Delivery")
• [Automation](https://www.infoq.com/automation/ "Automation")
• [Containers](https://www.infoq.com/containers/ "Containers")
• [Cloud](https://www.infoq.com/cloud-computing/ "Cloud")
• [Observability](https://www.infoq.com/observability/ "Observability")

Featured in DevOps

• #### Chaos Engineering GPU Clusters

Bryan Oliver discusses the frontier of AI infrastructure: chaos engineering for large-scale GPU clusters. He shares how engineering leaders can handle complex topologies, network protocols like RDMA, and NUMA misalignments. Discover seven practical fault-injection strategies to maximize multi-million dollar hardware efficiency and build robust observability loops.


All in devopsFollow Topic

[Events](https://events.infoq.com/ "Events")

Helpful links

• [About InfoQ](https://www.infoq.com/about-infoq "About InfoQ")
• [InfoQ Editors](https://www.infoq.com/infoq-editors "InfoQ Editors")
• [Write for InfoQ](https://www.infoq.com/write-for-infoq "Write for InfoQ")
• [About C4Media](https://c4media.com/ "About C4Media")
• [Diversity](https://c4media.com/diversity "Diversity")

Choose your language

• [En](https://www.infoq.com/news/2026/07/uber-opensearch-zone-failure/# "InfoQ English")
• 中文
• 日本
• Fr

July 25,2026 AI Engineering Certification Production AI calls on retrieval, agents, evals, and infrastructure, checked with peers. **Online. Register Now.**

Aug 21,2026 Engineering Leadership Certification Work through leadership decisions with senior peers facing similar technical trade-offs. **Online. Register Now.**

Nov 16-20,2026 QCon San Francisco What's working across AI, architecture, and leadership, from the teams doing it. **Register. Early bird ends July 14.**

Apr 13-16,2027 QCon London What early-adopter teams have proven in production, across 15 engineering tracks. **Register. Early bird ends July 14.**

[InfoQ Homepage](https://www.infoq.com/ "InfoQ Homepage")[News](https://www.infoq.com/news "News")How Uber Builds Zone-Failure-Resilient OpenSearch Clusters

[DevOps](https://www.infoq.com/Devops/ "DevOps")

Building AI Agent Evals for High-Stakes Incident Response (Webinar Aug 6th)

How Uber Builds Zone-Failure-Resilient OpenSearch Clusters

Jul 17, 2026 3 min read

by

• Claudio Masolo

Follow Senior DevOps Engineer

#### Follow us on

Youtube 232K FollowersLinkedin 26K FollowersInstagram NewRSS 19K ReadersX 57.1k FollowersFacebook 21K LikesBluesky New

Log in to listen to this article

Loading audio

Audio 2

0:00 0:00

Normal 1.25x 1.5x

Like

• Reading list

Uber explained how it keeps its OpenSearch deployments running during a zone outage. It does this by using OpenSearch's built-in shard allocation and its own isolation-group system, which relies on the Odin container orchestration platform. This way, it maintains both query and ingestion capabilities. The design tolerates a full zone failure plus one additional node failure while preserving quorum and avoiding data loss.

Uber aimed to solve a problem: physical zones often lack balanced node counts. This issue messed up OpenSearch's allocation-awareness logic. It typically left clusters in a yellow state with unassigned shards. Mapping shard placement to physical zones led to disk skew and hot nodes when zone capacity was uneven. This happens because OpenSearch's rebalancing expects an even pool of nodes for each awareness attribute.

Uber's fix introduces isolation groups (IGs) as a logical layer between physical failure domains and OpenSearch's placement logic. Every IG is guaranteed an equal number of nodes, regardless of how many physical zones are underneath, and a node retains its IG membership through hardware replacements. Most Uber services, including OpenSearch, run 3 IGs, so a single zone maps to exactly one IG, and a zone failure removes at most about a third of capacity. Each index runs with a minimum of 2 replicas (3 total shard copies), one per IG, giving baseline resilience against a single-group loss.

The harder problem is what happens during the failure itself. By default, OpenSearch responds to missing shard copies by rebalancing quickly across the remaining nodes. This can overload disk I/O, CPU, and network, risking instability in healthy areas. Uber tackles this with forced shard allocation awareness. The cluster is set up with all expected IG attribute values from the start, not just the ones visible now. When an IG goes missing, OpenSearch spots the gap. It won’t over-allocate to the other groups, so affected shards stay unassigned. This causes the cluster to turn yellow instead of starting a rebalancing storm. Shards only get reassigned once the failed IG's nodes return or an operator explicitly updates the awareness configuration.

/filters:no_upscale()/news/2026/07/uber-opensearch-zone-failure/en/resources/1srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy81ODY3Y2FhNS1hMTUwLTQxNTItYTk1Mi01OWQxNTQ5MjI1ZjIucG5n-1784274628644.jpeg)Shard allocation awareness Uber uses 5 cluster manager nodes for quorum, instead of the usual 3. This works with OpenSearch's cluster.auto_shrink_voting_configuration setting. A zone failure can take out up to 2 manager nodes, leaving 3; the voting configuration automatically shrinks to those 3, electing a new primary with a 2-of-3 quorum. A subsequent single-node failure still leaves 2 nodes, enough to retain quorum and keep the cluster writable, which a standard 3-manager setup could not survive.

Uber reports that the IG abstraction fixed yellow-state and shard-assignment failures. These issues came from uneven physical zone sizes. Now, there's 100% shard assignment and consistently green cluster health. It also addresses disk skew and hot-node effects linked to zone asymmetry. This approach works for all Tier 3 and higher OpenSearch and Elasticsearchclusters at Uber. It builds on OpenSearch features such as shard allocation awareness and voting configuration shrinkage. It doesn't need a custom search engine or a forked version.

Engineers running multi-zone OpenSearch or Elasticsearch clusters can adopt the forced-awareness pattern without Uber's Odin-specific tooling. The key requirements are an even node distribution across a fixed, explicitly declared set of awareness attribute values, at least 3 shard copies mapped one-to-one to failure domains, and an odd number of cluster manager nodes (5 rather than 3) with auto-shrink voting enabled. This is to survive a zone-plus-node failure sequence.

Uber's method reflects a wider trend in the industry. It focuses on embedding failure-domain awareness in the data layer rather than relying solely on infrastructure-level failover. Uber used a similar isolation-group model for Apache Pinot. They mapped isolation-group IDs to replica-group pools. This setup ensures that segment replicas can survive when a zone completely fails.

About the Author

#### Claudio Masolo

Claudio is a Senior DevOps Engineer at Nearform. In his spare time, he likes running, reading, and playing old video games.

Show more Show less

#### This content is in the DevOps topic

Follow Topic

##### Related Topics:

• DevOps### DevOps

Followers: 5099

Follow Topic

• Architecture### Architecture

Followers: 3351

Follow Topic

• Reliability### Reliability

Followers: 22

Follow Topic

• OpenSearch### OpenSearch

Followers: 11

Follow Topic

• Uber### Uber

Followers: 24

Follow Topic

* #### Popular in DevOps

• ##### Cloudflare Identifies Race Condition in hyper’s HTTP/1 Implementation

• ##### The Rust High Performance Talk You Did Not Expect

• ##### AWS Ships Claude Apps Gateway as Self-Hosted Control Plane for Claude Code and Claude Desktop

• ##### AWS Details How One Customer Scaled to One Million Lambda Functions

• ##### How Datadog Used Claude and Cursor for Test-Driven Production Migration

* #### Related Sponsors

• ##### Deliver Apps right from UI. Try WaveMaker Design-to-Code. Scale production with your own Design Systems.

• ##### Leverage AI to Protect Your Data Assets... from AI

• ##### How an AI Agent Deleted Production Data and Its Backups at a Company (and How to Protect Yours)

• ##### Below the Framework: Why Agent Context Is an Infrastructure Problem (Live Webinar August 27, 2026) - Save Your Seat

• ##### From Demo to Production: Why Agentic AI Systems Fail—and How to Fix Them

* #### Related Sponsor


• August 6, 2026, 1 PM EDT

##### Building AI Agent Evals for High-Stakes Incident Response

Presented by: Brianne Bujnowski - Senior Product Marketing Manager, AI at Datadog, and Benjamin Barton - Senior Software Engineer at Datadog

SPONSORED BY DATADOG Save your seat

Related Content

• ##### When a Cloud Region Fails: Rethinking High Availability in a Geopolitically Unstable World

Apr 22, 2026 

• ##### Trade-Offs in Multi-Region Architectures: Latency vs. Cost

Jul 10, 2026 

• ##### Airbnb Shares Architecture behind Sitar-Agent Dynamic Configuration Sidecar for Kubernetes Services

Jul 08, 2026

• Icon##### Designing AI Platforms for Reliability: Tools for Certainty, Agents for Discovery

Jul 07, 2026 

• ##### Netflix Cuts Cassandra Read Latency from Seconds to Milliseconds with Dynamic Partition Splitting

Jul 06, 2026

• Icon##### Practical Robustness: Going beyond Memory Safety in Rust

Jul 06, 2026 

• Icon##### Architecting a Centralized Platform for Data Deletion at Netflix

Jun 04, 2026 

• ##### Stragglers, Not Failures: How Adaptive Hedged Requests Reduce p99 Latency by 74 Percent

May 28, 2026 

• Icon##### Designing AI Platforms for Reliability: Tools for Certainty, Agents for Discovery

May 27, 2026 

Related Sponsors

• #### Building AI Agent Evals for High-Stakes Incident Response (Live Webinar August 6, 2026) - Save Your Seat

AI agents are transforming incident response, but their non-deterministic behavior makes quality hard to measure. Learn how Datadog built an eval platform to assess reasoning, tool use, and outcomes in production for high-stakes agents at scale.

• #### The Observability Migration Playbook

Observability migrations aren't like other platform moves—you can't lift-and-shift AppDynamics, New Relic, or Splunk into Datadog. This playbook from NoBS distills 10 proven principles and a 5-phase framework from hundreds of migrations.

• Sponsored by


Related Content

• Icon##### Security and Architecture: to Betray One is to Destroy Both

Mar 27, 2026 

• ##### Architecting Portable Systems on Open Standards for Digital Sovereignty

Mar 23, 2026 

• ##### Read-Copy-Update (RCU): the Secret to Lock-Free Performance

Mar 06, 2026 

• ##### Borrowing from Kotlin/Android to Architect Scalable iOS Apps in SwiftUI

Feb 26, 2026 

• ##### You’ve Generated Your MVP Using AI. What Does That Mean for Your Software Architecture?

Feb 12, 2026 

• ##### Working with Code Assistants: the Skeleton Architecture

Feb 03, 2026 

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

• Development

• ##### [The Rust High Performance Talk You Did Not Expect](https://www.infoq.com/presentations/rust-tps-service/ "The Rust High Performance Talk You Did Not Expect")

• ##### [How to Build More Resilient Local-First Applications with AT Protocol Infrastructure](https://www.infoq.com/news/2026/07/atproto-webapp/ "How to Build More Resilient Local-First Applications with AT Protocol Infrastructure")

• ##### [Cloudflare Identifies Race Condition in hyper’s HTTP/1 Implementation](https://www.infoq.com/news/2026/07/cloudflare-hyper-bug-fix/ "Cloudflare Identifies Race Condition in hyper’s HTTP/1 Implementation")

• Architecture & Design

• ##### [Stripe Benchmark Shows AI Agents Build Integrations but Struggle with Validation](https://www.infoq.com/news/2026/07/stripe-ai-agents-benchmark/ "Stripe Benchmark Shows AI Agents Build Integrations but Struggle with Validation")

• ##### [Google and Industry Partners Announce Agentic Resource Discovery Specification for AI Agents](https://www.infoq.com/news/2026/07/agentic-resource-discovery-spec/ "Google and Industry Partners Announce Agentic Resource Discovery Specification for AI Agents")

• ##### [Comprehension at AI Speed: Building a Context Store for Evolutionary Architecture](https://www.infoq.com/articles/ai-speed-context-store-architecture/ "Comprehension at AI Speed: Building a Context Store for Evolutionary Architecture")

• Culture & Methods

• ##### [Developing and Deploying a Platform that the Business Understands and Developers Actually Want](https://www.infoq.com/news/2026/07/platform-business-users/ "Developing and Deploying a Platform that the Business Understands and Developers Actually Want")

• ##### [Road to Compliance: Will Your Internal Users Hate Your Platform Team?](https://www.infoq.com/presentations/platform-engineering-team-compliance/ "Road to Compliance: Will Your Internal Users Hate Your Platform Team?")

• ##### [Formal Methods for Every Engineer in an AI-Powered Future](https://www.infoq.com/podcasts/formal-methods-ai-powered-future/ "Formal Methods for Every Engineer in an AI-Powered Future")

• AI, ML & Data Engineering

• ##### [Pinecone Introduces Nexus Engine for Compiling Business Context into Structured Data for AI Agents](https://www.infoq.com/news/2026/07/pinecon-nexus-knowledge-engine/ "Pinecone Introduces Nexus Engine for Compiling Business Context into Structured Data for AI Agents")

• ##### [Version Controlled SQL Database Dolt Releases 2.0 with Automatic Storage Cleanup and Compression](https://www.infoq.com/news/2026/07/dolt-version-control/ "Version Controlled SQL Database Dolt Releases 2.0 with Automatic Storage Cleanup and Compression")

• ##### [From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry](https://www.infoq.com/presentations/otel-slm-ai/ "From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry")

• DevOps

• ##### [Cloud Native Infrastructure Emerges as the Foundation for Trustworthy Agentic AI](https://www.infoq.com/news/2026/07/cncf-trustworthy-agentic-ai/ "Cloud Native Infrastructure Emerges as the Foundation for Trustworthy Agentic AI")

• ##### [How Uber Builds Zone-Failure-Resilient OpenSearch Clusters](https://www.infoq.com/news/2026/07/uber-opensearch-zone-failure/ "How Uber Builds Zone-Failure-Resilient OpenSearch Clusters")

• ##### [Linkerd 2.20 Delivers Smarter Traffic Management and Dramatic Efficiency Gains](https://www.infoq.com/news/2026/07/linkerd-2-20-improvements/ "Linkerd 2.20 Delivers Smarter Traffic Management and Dramatic Efficiency Gains")

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

• Get a quick overview of content published on a variety of innovator and early adopter technologies
• Learn what you don’t know that you don’t know
• Stay up to date with the latest information from the topics you are interested in

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

**InfoQ Online Certification Programs** For Senior Engineers, Architects, and Technical Leaders * **AI Engineering** with Hien Luu | July 25 * **Architect** with Luca Mezzalira | August 13 * **AI Security & Privacy Engineering** with Katharine Jarmul | August 26 As your role becomes more senior, the work changes. You're no longer just implementing decisions; you're shaping the systems, trade-offs, and technical direction other teams depend on. These 5-week online programs give you a structured way to work through real decisions from your current role with senior peers from other companies. **RESERVE YOUR PLACE** Live online. 4 hours a week, for 5 weeks.

#### Events

• ##### Online InfoQ AI Engineering Certification

July 25, 2026

• ##### Online InfoQ Architect Certification

August 13, 2026

• ##### Online InfoQ Engineering Leadership Certification

August 21, 2026

• ##### Online InfoQ AI Security & Privacy Engineering Program

August 26, 2026

• ##### QCon San Francisco

November 16-20, 2026

• ##### QCon London 2027

April 13-16, 2027

#### Follow us on

Youtube 232K FollowersLinkedin 26K FollowersInstagram NewRSS 19K ReadersX 57.1k FollowersFacebook 21K LikesBluesky New

#### Stay in the know

The InfoQ PodcastEngineering Culture PodcastThe Software Architects' Newsletter

General Feedback [[email protected]](mailto:[email protected]) Advertising [[email protected]](mailto:[email protected]) Editorial [[email protected]](mailto:[email protected]) Marketing [[email protected]](mailto:[email protected])

InfoQ.com and all content copyright © 2006-2026 C4Media Inc.

Privacy Notice, Terms And Conditions, Cookie Policy

Close

[BT](https://www.infoq.com/int/bt/ "bt")