Cloudflare Builds High-Performance Infrastructure for Running LLMs

Cloudflare Builds High-Performance Infrastructure for Running LLMs - InfoQ

[BT](http://www.infoq.com/int/bt/ "bt")

InfoQ Software Architects' Newsletter

A monthly overview of things you need to know as an architect or aspiring architect.

View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

Close

QCon San Francisco (Nov 16-20): What's next in AI? What's next in software? Learn from the teams already doing it.Register Now

Close

Toggle Navigation

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

English edition

• English edition
• Chinese edition
• Japanese edition
• French edition

[Write for InfoQ](http://www.infoq.com/write-for-infoq/ "Write for InfoQ")

Search

RegisterSign in

Unlock the full InfoQ experience

Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with content, and download exclusive resources.

Log In

or

Don't have an InfoQ account?

Register

• Stay updated on topics and peers that matter to youReceive instant alerts on the latest insights and trends.
• Quickly access free resources for continuous learningMinibooks, videos with transcripts, and training materials.
• Save articles and read at anytimeBookmark articles to read whenever youre ready.

Logo - Back to homepage

NewsArticlesPresentationsPodcastsGuides

Topics

[Development](http://www.infoq.com/development/ "Development")

• [Java](http://www.infoq.com/java/ "Java")
• [Kotlin](http://www.infoq.com/kotlin/ "Kotlin")
• [.Net](http://www.infoq.com/dotnet/ ".Net")
• [C#](http://www.infoq.com/c_sharp/ "C#")
• [Swift](http://www.infoq.com/swift/ "Swift")
• [Go](http://www.infoq.com/golang/ "Go")
• [Rust](http://www.infoq.com/rust/ "Rust")
• [JavaScript](http://www.infoq.com/javascript/ "JavaScript")

Featured in Development

• #### From VR to Flat Screens: Bridging the Input and Immersion Gap

Dany Lepage discusses the architectural journey of porting a hit VR title to seven non-VR platforms. He explains how his team solved the challenges of cross-progression, diverse input paradigms, and maintaining release velocity across Steam, iOS, and PlayStation. Beyond the tech, he shares candid lessons on the "product fit" gap when translating immersive social presence to 2D screens.


All in developmentFollow Topic

[Architecture & Design](http://www.infoq.com/architecture-design/ "Architecture & Design")

• [Architecture](http://www.infoq.com/architecture/ "Architecture")
• [Enterprise Architecture](http://www.infoq.com/enterprise-architecture/ "Enterprise Architecture")
• [Scalability/Performance](http://www.infoq.com/performance-scalability/ "Scalability/Performance")
• [Design](http://www.infoq.com/design/ "Design")
• [Case Studies](http://www.infoq.com/Case_Study/ "Case Studies")
• [Microservices](http://www.infoq.com/microservices/ "Microservices")
• [Service Mesh](http://www.infoq.com/servicemesh/ "Service Mesh")
• [Patterns](http://www.infoq.com/DesignPattern/ "Patterns")
• [Security](http://www.infoq.com/Security/ "Security")

Featured in Architecture & Design

• #### Stripe’s Docdb: How Zero-Downtime Data Movement Powers Trillion-Dollar Payment Processing

Jimmy Morzaria discusses the evolution of Stripe’s database tier to support 5 million QPS with 5.5 nines of reliability. He explains the architecture of DocDB and shares how Stripe leverages a custom zero-downtime data movement platform to perform horizontal sharding, version upgrades, and multi-tenant migrations - all while maintaining the strict consistency required for global commerce.


All in architecture-designFollow Topic

[AI Infrastructure](http://www.infoq.com/ai-ml-data-eng/ "AI Infrastructure")

• [Big Data](http://www.infoq.com/bigdata/ "Big Data")
• [Machine Learning](http://www.infoq.com/machinelearning/ "Machine Learning")
• [NoSQL](http://www.infoq.com/nosql/ "NoSQL")
• [Database](http://www.infoq.com/database/ "Database")
• [Data Analytics](http://www.infoq.com/data-analytics/ "Data Analytics")
• [Streaming](http://www.infoq.com/streaming/ "Streaming")

Featured in AI, ML & Data Engineering

• #### The Next Generation of AI Products

Hilary Mason shares her journey from academia to building AI products at scale. She discusses the shift from discrete engineering to probabilistic mindsets, explaining why managing "human considerations" is the hardest part of the stack. She explains the "existential crisis" for engineers, arguing that great architecture today is about context management, systems thinking, and good taste.


All in ai-ml-data-engFollow Topic

[Culture & Methods](http://www.infoq.com/culture-methods/ "Culture & Methods")

• [Agile](http://www.infoq.com/agile/ "Agile")
• [Diversity](http://www.infoq.com/diversity/ "Diversity")
• [Leadership](http://www.infoq.com/leadership/ "Leadership")
• [Lean/Kanban](http://www.infoq.com/lean/ "Lean/Kanban")
• [Personal Growth](http://www.infoq.com/personal-growth/ "Personal Growth")
• [Scrum](http://www.infoq.com/scrum/ "Scrum")
• [Sociocracy](http://www.infoq.com/sociocracy/ "Sociocracy")
• [Software Craftmanship](http://www.infoq.com/software_craftsmanship/ "Software Craftmanship")
• [Team Collaboration](http://www.infoq.com/team-collaboration/ "Team Collaboration")
• [Testing](http://www.infoq.com/testing/ "Testing")
• [UX](http://www.infoq.com/ux/ "UX")

Featured in Culture & Methods

• #### Panel: Building a Culture that Works

The panelists share insights on evolving company culture. They discuss leveraging feedback loops, lending social capital, and the friction between legacy bureaucracy and agile engineering. The panel explains how to maintain cohesion in remote teams and use interviews to uncover the true "unmanicured" culture of a firm.


All in culture-methodsFollow Topic

DevOps

• [Infrastructure](http://www.infoq.com/infrastructure/ "Infrastructure")
• [Continuous Delivery](http://www.infoq.com/continuous_delivery/ "Continuous Delivery")
• [Automation](http://www.infoq.com/automation/ "Automation")
• [Containers](http://www.infoq.com/containers/ "Containers")
• [Cloud](http://www.infoq.com/cloud-computing/ "Cloud")
• [Observability](http://www.infoq.com/observability/ "Observability")

Featured in DevOps

• #### AI-Powered SRE for Autonomous Incident Response

The presenters discuss incident response, how AI-enhanced SRE platforms connect signals from logs, metrics, traces, and historical incidents to enable autonomous decisions.


All in devopsFollow Topic

[Events](https://events.infoq.com/ "Events")

Helpful links

• [About InfoQ](http://www.infoq.com/about-infoq "About InfoQ")
• [InfoQ Editors](http://www.infoq.com/infoq-editors "InfoQ Editors")
• [Write for InfoQ](http://www.infoq.com/write-for-infoq "Write for InfoQ")
• [About C4Media](https://c4media.com/ "About C4Media")
• [Diversity](https://c4media.com/diversity "Diversity")

Choose your language

• [En](http://www.infoq.com/news/2026/05/cloudflare-llm-infrastructure/# "InfoQ English")
• 中文
• 日本
• Fr


[InfoQ Homepage](http://www.infoq.com/ "InfoQ Homepage")[News](http://www.infoq.com/news "News")Cloudflare Builds High-Performance Infrastructure for Running LLMs

[AI, ML & Data Engineering](http://www.infoq.com/ai-ml-data-eng/ "AI, ML & Data Engineering")

Portable by Design: Data Mobility & Recovery Patterns for Multi-Cloud Systems (Webinar May 21st)

Cloudflare Builds High-Performance Infrastructure for Running LLMs

May 03, 2026 2 min read

by

• Renato Losio

Follow Cloud Expert | AWS Data Hero

#### Write for InfoQ

Feed your curiosity.Help 550k+ global

senior developers

each month stay ahead.Get in touch

Log in to listen to this article

Audio ready to play

Your browser does not support the audio element.

0:00 0:00

Normal 1.25x 1.5x

Like

• Reading list

Cloudflare has recently announced new infrastructure designed to run large AI language models across its global network. As these models rely on costly hardware and must handle large volumes of incoming and outgoing text, Cloudflare separates the model's input processing and output generation onto different optimized systems and uses a custom inference engine to manage GPUs more efficiently.

According to the Cloudflare team, one key improvement is splitting model processing into two stages, each handled by a different machine: one stage reads and prepares the input text, while the other generates the output. Michelle Chen, principal product manager at Cloudflare, Kevin Flansburg, senior engineering manager at Cloudflare, and Vlad Krasnov, principal systems engineer at Cloudflare, write:

One hardware configuration that we use to improve performance and efficiency is disaggregated prefill. There are two stages to processing an LLM request: prefill, which processes the input tokens and populates the KV cache, and decode, which generates output tokens. Prefill is usually compute bound, while decode is memory bound.

Cloudflare also created a custom AI inference engine called Infire. Announced during Cloudflare Birthday Week 2025, Infire runs large language models across multiple GPUs more efficiently, reduces memory usage, and starts models more quickly, delivering faster responses.

Large language models such as Kimi K2.5 are so large (over 1 trillion parameters and about 560GB in size) that they must be split across multiple GPUs, requiring at least eight H100s just to load the model into memory, before accounting for additional memory used during processing. Explaining why Infire and the hardware optimizations help run huge models more efficiently and deliver faster responses to users, Chen, Flansburg, and Krasnov add:

For pipeline parallelism, Infire attempts to properly load balance all stages of the pipeline, in order to prevent the GPUs of one stage from starving while other stages are executing. On the other hand, for tensor parallelism, Infire optimizes for reducing cross-GPU communication, making it as fast as possible. For most models, utilizing both pipeline parallelism and tensor parallelism in tandem provides the best balance of throughput and latency.

In a previous article, Cloudflare explained how to run open-source models on its AI inference platform, starting with Moonshot AI’s Kimi K2.5 model on Workers AI, and highlighted how the team is using a variety of hardware configurations to best serve models.

/filters:no_upscale()/news/2026/05/cloudflare-llm-infrastructure/en/resources/1BLOG-3266_2-1776661409906.png)

_Source: Cloudflare blog_

According to Cloudflare, the team further optimized Infire to reduce GPU memory usage for internal processes, allowing it to run Llama 4 Scout on just two H200 GPUs with large capacity for context tokens and Kimi K2.5 on eight H100 GPUs, while still leaving memory for the KV cache.

Cloudflare also recently introduced Unweight, a system the company claims compresses large language model weights by about 15–22% without losing accuracy, reducing the amount of data GPUs need to load and move during inference, so models can run faster and more efficiently.

Cloudflare is not the only provider highlighting the challenges of running LLMs in production. Cockroach Labs' recent State of AI Infrastructure report states that as companies move AI systems into everyday use, many are finding their current infrastructure is not ready to handle the scale and reliability these workloads require. Cockroach Labs' analysis summarizes:

Legacy infrastructure, built around episodic human interaction, simply wasn’t designed for this kind of pressure. To handle the pace and unpredictability of AI, companies need more than performance upgrades. They need a fundamental shift in how systems are architected.

Cloudflare has separetely discussed how it optimized for efficient prompt caching.

About the Author

#### Renato Losio

Renato has extensive experience as a cloud architect, tech lead, and cloud services specialist. Currently, he lives in Berlin and works remotely as a principal cloud architect. His primary areas of interest include cloud services and relational databases. He is an editor at InfoQ and a recognized AWS Data Hero. You can connect with him on LinkedIn.

Show more Show less

#### This content is in the AI, ML & Data Engineering topic

Follow Topic

##### Related Topics:

• Development### Development

Followers: 4095

Follow Topic

• AI, ML & Data Engineering### AI, ML & Data Engineering

Followers: 5889

Follow Topic

• Optimization### Optimization

Followers: 9

Follow Topic

• Big Data Infrastructure### Big Data Infrastructure

Followers: 36

Follow Topic

• AI Architecture### AI Architecture

Followers: 34

Follow Topic

• Cloudflare### Cloudflare

Followers: 16

Follow Topic

• Large language models### Large language models

Followers: 141

Follow Topic

• GPU### GPU

Followers: 7

Follow Topic

* #### Related Editorial

• ##### Cloudflare Announces Agent Memory, a Managed Persistent Memory Service for AI Agents

• ##### Cloudflare's Matrix Homeserver Demo Sparks Debate over AI-Generated Code Claims

• ##### Cloudflare Introduces Automated Scoring for Shadow AI Risk Assessment

• ##### Dynamic Moments: Weaving LLMs into Deep Personalization at DoorDash

• ##### Anthropic Paper Examines Behavioral Impact of Emotion-Like Mechanisms in LLMs

* #### Related Sponsors

• ##### Deliver Apps right from UI. Try WaveMaker Design-to-Code. Scale production with your own Design Systems.

• ##### Why APIs Can’t Trust Clients—and How to Bridge the Gap

• ##### [[Webinar] Creating Certainty in the Age of Agentic AI. Watch On-Demand.](http://www.infoq.com/vendorcontent/show.action?vcr=531d8edd-4f74-486b-aaca-10058c609c1c&primaryTopicId=4523&vcrPlace=BOTTOM&pageType=NEWS_PAGE&vcrReferrer=https%3A%2F%2Fwww.infoq.com%2Fnews%2F2026%2F05%2Fcloudflare-llm-infrastructure%2F)

• ##### Does your AI stack need a session layer? A maturity framework for teams building AI agents

• ##### Understanding AI Agent Security

* #### Related Sponsor


• May 12, 2026, 1:30 PM EDT

##### Designing Data Layers for Agentic AI: Patterns for State, Memory, and Coordination at Scale

Presented by: Karthik Ranganathan - Co-CEO & Co-Founder at YugabyteDB, and Aditi Gupta - Snr. GenAI/ML Specialist Solutions Architect

SPONSORED BY YUGABYTEDB Save your seat

Related Content

• ##### Cloudflare Announces Agent Memory, a Managed Persistent Memory Service for AI Agents

Apr 30, 2026

• ##### Cloudflare Optimizes Edge Stack for High-Core CPUs instead of Large Cache

Apr 25, 2026

• ##### Cloudflare Sandboxes Reach General Availability, Giving AI Agents Persistent Isolated Environments

Apr 22, 2026

• ##### Cloudflare Outlines MCP Architecture as Enterprises Confront Security and Governance Risks

Apr 22, 2026

• ##### Cloudflare Introduces Project Think: a Durable Runtime for AI Agents

Apr 21, 2026

• ##### Anthropic Paper Examines Behavioral Impact of Emotion-Like Mechanisms in LLMs

Apr 14, 2026

• ##### Cloudflare Introduces EmDash: TypeScript CMS Positioned as WordPress Successor

Apr 09, 2026

• Icon##### Dynamic Moments: Weaving LLMs into Deep Personalization at DoorDash

Apr 21, 2026 

• ##### Building LLMs in Resource-Constrained Environments: a Hands-On Perspective

Feb 09, 2026 

Related Sponsors

• #### Designing Data Layers for Agentic AI: Patterns for State, Memory, and Coordination at Scale (Live Webinar May 12, 2026) - Save Your Seat

AI agents create new architectural challenges: shared memory, cross-agent state, and auditability. This session explores data layer patterns—conversation state, knowledge persistence, coordination—and tradeoffs in consistency, latency, and cost at scale using AWS and YugabyteDB.

• Sponsored by


Related Content

• ##### NVIDIA Launches Ising Open Models for Quantum Computing

Apr 30, 2026

• ##### Anthropic Introduces Managed Agents to Simplify AI Agent Deployment

Apr 21, 2026

• ##### Legare Kerrison and Cedric Clyburn on LLM Performance and Evaluations

Apr 28, 2026

• ##### Designing Memory for AI Agents: inside Linkedin’s Cognitive Memory Agent

Apr 20, 2026

• ##### How Slack Manages Context in Long-Running Multi-agent Systems

Apr 28, 2026

• ##### Cloudflare Launches Code Mode MCP Server to Optimize Token Usage for AI Agents

Apr 16, 2026

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

• Development

• ##### [QCon San Francisco 2026: 12 Tracks Announced](http://www.infoq.com/news/2026/04/qconsf-2026-tracks-announced/ "QCon San Francisco 2026: 12 Tracks Announced")

• ##### [Microsoft's Russinovich and Hanselman Warn AI Is Hollowing out the Junior Developer Pipeline](http://www.infoq.com/news/2026/04/junior-developer-pipeline-crisis/ "Microsoft's Russinovich and Hanselman Warn AI Is Hollowing out the Junior Developer Pipeline")

• ##### [C++26: Reflection, Memory Safety, Contracts, and a New Async Model](http://www.infoq.com/news/2026/04/cpp-26-reflection-safety-async/ "C++26: Reflection, Memory Safety, Contracts, and a New Async Model")

• Architecture & Design

• ##### [Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance](http://www.infoq.com/news/2026/05/confluent-kafka-header-schema-id/ "Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance")

• ##### [Meta's Approach to Migrating their Systems to Post-Quantum Cryptography](http://www.infoq.com/news/2026/04/meta-quantum-crypto-migration/ "Meta's Approach to Migrating their Systems to Post-Quantum Cryptography")

• ##### [Stripe’s Docdb: How Zero-Downtime Data Movement Powers Trillion-Dollar Payment Processing](http://www.infoq.com/presentations/docdb-online-database/ "Stripe’s Docdb: How Zero-Downtime Data Movement Powers Trillion-Dollar Payment Processing")

• Culture & Methods

• ##### [Driving and Measuring the Impact of Platform Engineering](http://www.infoq.com/news/2026/04/measure-platform-engineering/ "Driving and Measuring the Impact of Platform Engineering")

• ##### [How Observability and Telemetry Can Enhance the Practice of Software Engineering](http://www.infoq.com/news/2026/04/observability-telemetry/ "How Observability and Telemetry Can Enhance the Practice of Software Engineering")

• ##### [Panel: Building a Culture that Works](http://www.infoq.com/presentations/panel-positive-culture/ "Panel: Building a Culture that Works")

• AI, ML & Data Engineering

• ##### [Cloudflare Builds High-Performance Infrastructure for Running LLMs](http://www.infoq.com/news/2026/05/cloudflare-llm-infrastructure/ "Cloudflare Builds High-Performance Infrastructure for Running LLMs")

• ##### [DuckLake 1.0: Data Lake Format with SQL Catalog Metadata](http://www.infoq.com/news/2026/05/ducklake-sql-catalog/ "DuckLake 1.0: Data Lake Format with SQL Catalog Metadata")

• ##### [The Next Generation of AI Products](http://www.infoq.com/presentations/ai-products/ "The Next Generation of AI Products")

• DevOps

• ##### [Meta Deploys Unified AI Agents to Automate Performance Optimization at Hyperscale](http://www.infoq.com/news/2026/05/meta-ai-agents-hyperscale/ "Meta Deploys Unified AI Agents to Automate Performance Optimization at Hyperscale")

• ##### [Broadcom Donates Velero to CNCF, Shifting Kubernetes Backup to Community Governance](http://www.infoq.com/news/2026/05/broadcom-velero-cncf/ "Broadcom Donates Velero to CNCF, Shifting Kubernetes Backup to Community Governance")

• ##### [Dropbox Redesigns Compaction to Reclaim Space from Underfilled Storage Volumes](http://www.infoq.com/news/2026/04/dropbox-tiered-compaction/ "Dropbox Redesigns Compaction to Reclaim Space from Underfilled Storage Volumes")

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

• Get a quick overview of content published on a variety of innovator and early adopter technologies
• Learn what you don’t know that you don’t know
• Stay up to date with the latest information from the topics you are interested in

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

**May 7 | June 10, 2026 | Online** Architecture decisions are hard to validate while shipping. Join a **5-week online cohort** for **senior engineers, architects, and team leads** to pressure-test real decisions, apply practical frameworks, and work through challenges with a confidential peer group. Facilitated by Luca Mezzalira, Principal Architect at AWS, this cohort helps you: * Pressure-test real decisions. * Apply frameworks to real problems. * Publish on InfoQ.com and earn your certification. **RESERVE YOUR PLACE**

#### Events

• ##### Online InfoQ Architect Certification

May 7, 2026

• ##### QCon AI Boston

June 1-2, 2026

• ##### Online InfoQ Architect Certification

June 10, 2026

• ##### QCon San Francisco

November 16-20, 2026

#### Follow us on

Youtube 232K FollowersLinkedin 26K FollowersInstagram NewRSS 19K ReadersX 57.1k FollowersFacebook 21K LikesBluesky New

#### Stay in the know

The InfoQ PodcastEngineering Culture PodcastThe Software Architects' Newsletter

General Feedback [[email protected]](mailto:[email protected]) Advertising [[email protected]](mailto:[email protected]) Editorial [[email protected]](mailto:[email protected]) Marketing [[email protected]](mailto:[email protected])

InfoQ.com and all content copyright © 2006-2026 C4Media Inc.

Privacy Notice, Terms And Conditions, Cookie Policy

Close

[BT](http://www.infoq.com/int/bt/ "bt")