Meta Reports 4x Higher Bug Detection with Just-in-Time Testing

Meta Reports 4x Higher Bug Detection with Just-in-Time Testing - InfoQ

[BT](http://www.infoq.com/int/bt/ "bt")

InfoQ Software Architects' Newsletter

A monthly overview of things you need to know as an architect or aspiring architect.

View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

Close

Live Webinar and Q&A: Shipping Faster, Breaking More: Rethinking Delivery Systems in the Age of AI (May 28, 2026)Save Your Seat

Close

Toggle Navigation

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

English edition

• English edition
• Chinese edition
• Japanese edition
• French edition

[Write for InfoQ](http://www.infoq.com/write-for-infoq/ "Write for InfoQ")

Search

RegisterSign in

Unlock the full InfoQ experience

Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with content, and download exclusive resources.

Log In

or

Don't have an InfoQ account?

Register

• **Stay updated on topics and peers that matter to you**Receive instant alerts on the latest insights and trends.
• **Quickly access free resources for continuous learning**Minibooks, videos with transcripts, and training materials.
• **Save articles and read at anytime**Bookmark articles to read whenever youre ready.

Logo - Back to homepage

NewsArticlesPresentationsPodcastsGuides

Topics

[Development](http://www.infoq.com/development/ "Development")

• [Java](http://www.infoq.com/java/ "Java")
• [Kotlin](http://www.infoq.com/kotlin/ "Kotlin")
• [.Net](http://www.infoq.com/dotnet/ ".Net")
• [C#](http://www.infoq.com/c_sharp/ "C#")
• [Swift](http://www.infoq.com/swift/ "Swift")
• [Go](http://www.infoq.com/golang/ "Go")
• [Rust](http://www.infoq.com/rust/ "Rust")
• [JavaScript](http://www.infoq.com/javascript/ "JavaScript")

Featured in Development

• #### From VR to Flat Screens: Bridging the Input and Immersion Gap

Dany Lepage discusses the architectural journey of porting a hit VR title to seven non-VR platforms. He explains how his team solved the challenges of cross-progression, diverse input paradigms, and maintaining release velocity across Steam, iOS, and PlayStation. Beyond the tech, he shares candid lessons on the "product fit" gap when translating immersive social presence to 2D screens.


All in developmentFollow Topic

[Architecture & Design](http://www.infoq.com/architecture-design/ "Architecture & Design")

• [Architecture](http://www.infoq.com/architecture/ "Architecture")
• [Enterprise Architecture](http://www.infoq.com/enterprise-architecture/ "Enterprise Architecture")
• [Scalability/Performance](http://www.infoq.com/performance-scalability/ "Scalability/Performance")
• [Design](http://www.infoq.com/design/ "Design")
• [Case Studies](http://www.infoq.com/Case_Study/ "Case Studies")
• [Microservices](http://www.infoq.com/microservices/ "Microservices")
• [Service Mesh](http://www.infoq.com/servicemesh/ "Service Mesh")
• [Patterns](http://www.infoq.com/DesignPattern/ "Patterns")
• [Security](http://www.infoq.com/Security/ "Security")

Featured in Architecture & Design

• #### Platform Engineering: Lessons from the Rise and Fall of eBay Velocity

Randy Shoup discusses the "Velocity Initiative," a transformation that doubled engineering productivity and modernized eBay’s DORA metrics. He shares the technical playbook used to scale 4,500 services while explaining why even elite engineering execution can’t save a company hampered by waterfall planning, risk aversion, and a "pathological" culture of fear.


All in architecture-designFollow Topic

[AI Infrastructure](http://www.infoq.com/ai-ml-data-eng/ "AI Infrastructure")

• [Big Data](http://www.infoq.com/bigdata/ "Big Data")
• [Machine Learning](http://www.infoq.com/machinelearning/ "Machine Learning")
• [NoSQL](http://www.infoq.com/nosql/ "NoSQL")
• [Database](http://www.infoq.com/database/ "Database")
• [Data Analytics](http://www.infoq.com/data-analytics/ "Data Analytics")
• [Streaming](http://www.infoq.com/streaming/ "Streaming")

Featured in AI, ML & Data Engineering

• #### Lakehouse Tower of Babel: Handling Identifier Resolution Rules Across Database Engines

Lakehouse architectures enable multiple engines to operate on shared data using open table formats such as Apache Iceberg. However, differences in SQL identifier resolution and catalog naming rules create interoperability failures. This article examines these behaviors and explains why enforcing consistent naming conventions and cross-engine validation is critical.


All in ai-ml-data-engFollow Topic

[Culture & Methods](http://www.infoq.com/culture-methods/ "Culture & Methods")

• [Agile](http://www.infoq.com/agile/ "Agile")
• [Diversity](http://www.infoq.com/diversity/ "Diversity")
• [Leadership](http://www.infoq.com/leadership/ "Leadership")
• [Lean/Kanban](http://www.infoq.com/lean/ "Lean/Kanban")
• [Personal Growth](http://www.infoq.com/personal-growth/ "Personal Growth")
• [Scrum](http://www.infoq.com/scrum/ "Scrum")
• [Sociocracy](http://www.infoq.com/sociocracy/ "Sociocracy")
• [Software Craftmanship](http://www.infoq.com/software_craftsmanship/ "Software Craftmanship")
• [Team Collaboration](http://www.infoq.com/team-collaboration/ "Team Collaboration")
• [Testing](http://www.infoq.com/testing/ "Testing")
• [UX](http://www.infoq.com/ux/ "UX")

Featured in Culture & Methods

• #### Empower Your Developers: How Open Source Dependencies Risk Management Can Unlock Innovation

Celine Pypaert discusses the ubiquitous nature of open-source software and shares a blueprint for securing modern applications. She explains how to prioritize high-risk vulnerabilities using exploitability data, the role of Software Bill of Materials (SBOM), and the importance of bridging the gap between DevOps and Security through clear accountability and automated governance.


All in culture-methodsFollow Topic

DevOps

• [Infrastructure](http://www.infoq.com/infrastructure/ "Infrastructure")
• [Continuous Delivery](http://www.infoq.com/continuous_delivery/ "Continuous Delivery")
• [Automation](http://www.infoq.com/automation/ "Automation")
• [Containers](http://www.infoq.com/containers/ "Containers")
• [Cloud](http://www.infoq.com/cloud-computing/ "Cloud")
• [Observability](http://www.infoq.com/observability/ "Observability")

Featured in DevOps

• #### Beyond One-Click: Designing an Enterprise-Grade Observability Extension for Docker

Docker Extensions boost developer speed but create a "visibility gap" by isolating telemetry. To meet enterprise needs, extensions must act as bridges to centralized platforms. This article details how to use OpenTelemetry, policy-as-code, and encryption to build secure pipelines. Learn to balance developer productivity with the governance required for scalable, compliant observability.


All in devopsFollow Topic

[Events](https://events.infoq.com/ "Events")

Helpful links

• [About InfoQ](http://www.infoq.com/about-infoq "About InfoQ")
• [InfoQ Editors](http://www.infoq.com/infoq-editors "InfoQ Editors")
• [Write for InfoQ](http://www.infoq.com/write-for-infoq "Write for InfoQ")
• [About C4Media](https://c4media.com/ "About C4Media")
• [Diversity](https://c4media.com/diversity "Diversity")

Choose your language

• [En](http://www.infoq.com/news/2026/04/meta-jit-testing-ai-detection/# "InfoQ English")
• 中文
• 日本
• Fr


[InfoQ Homepage](http://www.infoq.com/ "InfoQ Homepage")[News](http://www.infoq.com/news "News")Meta Reports 4x Higher Bug Detection with Just-in-Time Testing

[Architecture & Design](http://www.infoq.com/architecture-design/ "Architecture & Design")

Designing Data Layers for Agentic AI: Patterns for State, Memory, and Coordination at Scale (Webinar May 12th)

Meta Reports 4x Higher Bug Detection with Just-in-Time Testing

Apr 17, 2026 2 min read

by

• Leela Kumili

Follow Lead Engineer

#### Write for InfoQ

**Feed your curiosity.**Help 550k+ global

senior developers

each month stay ahead.Get in touch

Log in to listen to this article

Loading audio

Your browser does not support the audio element.

0:00 0:00

Normal 1.25x 1.5x

Like

• Reading list

Meta has reported improved software quality using a Just-in-Time (JiT) testing approach that dynamically generates tests during code review instead of relying on long-lived, manually maintained test suites. According to Meta’s engineering blog and accompanying research, the approach improves bug detection by approximately 4x in AI-assisted development environments.

The shift is driven by agentic workflows where AI systems increasingly generate or modify large portions of code. In this environment, traditional test suites face higher maintenance overhead and reduced effectiveness, as brittle assertions and outdated coverage struggle to keep up with rapid changes.

AsAnkit K., ICT Systems Test Engineer, observes:

AI generating code and tests faster than humans can maintain them makes JiT testing almost inevitable.

JiT testing addresses this by generating tests at pull request time based on the specific code diff. Instead of static validation, the system infers developer intent, identifies potential failure modes, and constructs targeted tests designed to fail when regressions exist. It targets regression-catching tests that fail on the proposed changes but pass on the parent revision. This is achieved through a pipeline combining large language models, program analysis, and mutation testing, where synthetic defects are injected to validate whether generated tests detect them.

As Mark Harman, Research Scientist at Meta, notes:

This work represents a fundamental shift from ‘hardening’ tests that pass today to ‘catching’ tests that find tomorrow’s bugs.

A key component is the Dodgy Diff and intent-aware workflow architecture, which reframes a code change as a semantic signal rather than a textual diff. The system analyzes the diff to extract behavioral intent and risk areas, then performs intent reconstruction and change-risk modeling to understand what could break as a result. These signals feed into a mutation engine that generates dodgy; variants of the code, simulating realistic failure scenarios. An LLM-based test synthesis layer then generates tests aligned with inferred intent, followed by filtering to remove noisy or low-value tests before surfacing results in the pull request.

!Image 12/filters:no_upscale()/news/2026/04/meta-jit-testing-ai-detection/en/resources/3doggydiff-1776257581823.jpeg)

_Architecture of ‘Dodgy diff’ and Intent-Aware Workflows for generating Just-in-Time Catches (Source: Meta Research Paper)_

Meta reports that the system was evaluated on over 22,000 generated tests. Results show a 4x improvement in bug detection over baseline-generated tests and up to 20x improvement in detecting meaningful failures compared to coincidental outcomes. In one evaluation subset, 41 issues were identified, of which 8 were confirmed as real defects, including several with potential production impact.

Mark Harman, in another LinkedIn post, emphasized

Mutation testing, after decades of purely intellectual impact, confined to academic circles, is finally breaking out into industry and transforming practical, scalable Software Testing 2.0.

Catching JiT tests are designed for AI-driven development, generated per change to detect serious, unexpected bugs without ongoing maintenance. They reduce brittle test suites by adapting automatically as code evolves and shifting effort from humans to machines. Human review is required only when meaningful issues are surfaced. This reframes testing toward change-specific fault detection rather than static correctness validation.

About the Author


#### **Leela Kumili**

Leela is a Lead Software Engineer at Starbucks with deep expertise in building scalable, cloud-native systems and distributed platforms. She drives architecture, delivery, and operational excellence across the Rewards Platform, leading efforts to modernize systems, improve scalability, and enhance reliability. In addition to her technical leadership, Leela serves as an AI Champion for the organization, identifying opportunities to improve developer productivity and workflows using LLM-based tools and establishing best practices for AI adoption. She is passionate about building production-ready systems, enhancing developer experience, and mentoring engineers to grow in both technical and strategic impact. Her interests include platform engineering, distributed systems, developer productivity, and bridging technical solutions with business and product goals.

Show more Show less

#### This content is in the Large language models topic

Follow Topic

##### Related Topics:

• Development### Development

Followers: 4085

Follow Topic

• Architecture & Design### Architecture & Design

Followers: 10198

Follow Topic

• AI, ML & Data Engineering### AI, ML & Data Engineering

Followers: 5862

Follow Topic

• Code Reviews### Code Reviews

Followers: 11

Follow Topic

• Agents### Agents

Followers: 39

Follow Topic

• Test Automation### Test Automation

Followers: 61

Follow Topic

• Testing### Testing

Followers: 443

Follow Topic

• Automated testing### Automated testing

Followers: 66

Follow Topic

• Software Testing### Software Testing

Followers: 57

Follow Topic

• Automation### Automation

Followers: 268

Follow Topic

• Large language models### Large language models

Followers: 137

Follow Topic

• AI Assisted Coding### AI Assisted Coding

Followers: 2

Follow Topic

* #### Popular in Architecture & Design

• ##### Google Open Sources Experimental Multi-Agent Orchestration Testbed Scion

• ##### Java News Roundup: JDK 27 Release Schedule, Hibernate, LangChain4j, Keycloak, Helidon, Junie CLI

• ##### Dynamic Languages Faster and Cheaper in 13-Language Claude Code Benchmark

• ##### Cloudflare Introduces EmDash: TypeScript CMS Positioned as WordPress Successor

• ##### Building Hierarchical Agentic RAG Systems: Multi-Modal Reasoning with Autonomous Error Recovery

* #### Related Sponsors

• ##### Stateful agents, stateful infra: The transport gap AI teams are patching by hand

• ##### The Rise of Client-Side Risk and the Trust Gap

• ##### Shipping Faster, Breaking More: Rethinking Delivery Systems in the Age of AI (Live Webinar May 28th) - Save Your Seat

• ##### Beyond Scripts: Why IaC is Actually an Architecture Problem. Join the Discussion at IaCConf 2026 (May 14th).

• ##### Understanding AI Agent Security

• #### Related Sponsor

Confidently test, evaluate, and red-team your LLM apps with **Promptfoo** — catch regressions, benchmark models, and ship high-quality AI features faster; start testing your prompts today. **Learn More.**

Related Content

• Icon##### No QA Environment? No Problem: How Classpass Enables Testing on a Single Environment in ECS

Jan 19, 2026 

• ##### Anthropic Introduces Agent-Based Code Review for Claude Code

Apr 17, 2026

• ##### CNCF Warns Kubernetes Alone Is Not Enough to Secure LLM Workloads

Apr 17, 2026

• ##### Cloudflare Launches Code Mode MCP Server to Optimize Token Usage for AI Agents

Apr 16, 2026

• ##### Google Opens Gemma 4 Under Apache 2.0 with Multimodal and Agentic Capabilities

Apr 16, 2026

• ##### Cursor 3 Introduces Agent-First Interface, Moving beyond the IDE Model

Apr 16, 2026

• ##### Google’s TurboQuant Compression May Support Faster Inference, Same Accuracy on Less Capable Hardware

Apr 15, 2026

• ##### Google Released Gemma 4 with a Focus on Local-First, On-Device AI Inference

Apr 13, 2026

• ##### Anthropic Paper Examines Behavioral Impact of Emotion-Like Mechanisms in LLMs

Apr 14, 2026

Related Sponsors

• #### Inside MCP: A Protocol for AI Integration

The Model Context Protocol (MCP) defines a standard way for AI systems to interact with tools, data, and services. This article explains MCP’s architecture—hosts, clients, and servers—and how it enables structured, secure integrations between AI models and external systems.

• #### Harder, Better, Prompter, Stronger: AI system prompt hardening

System prompts define how LLM applications behave—but they are vulnerable to manipulation. This article explores prompt hardening techniques such as instruction shielding, syntax reinforcement, and layered prompting to defend AI systems against prompt injection and override attacks.

• Sponsored by


Related Content

• ##### Lyft Scales Global Localization Using AI and Human-in-the-Loop Review

Apr 13, 2026

• ##### Google Open Sources Experimental Multi-Agent Orchestration Testbed Scion

Apr 07, 2026

• ##### Sonatype Launches Guide to Enhance Safety in AI-Assisted Code Generation

Mar 20, 2026

• ##### QCon London 2026: Refreshing Stale Code Intelligence

Mar 19, 2026

• ##### OpenAI Extends the Responses API to Serve as a Foundation for Autonomous Agents

Mar 27, 2026

• ##### Green IT: How to Reduce the Impact of AI on the Environment

Mar 26, 2026

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

• Development

• ##### [C++26: Reflection, Memory Safety, Contracts, and a New Async Model](http://www.infoq.com/news/2026/04/cpp-26-reflection-safety-async/ "C++26: Reflection, Memory Safety, Contracts, and a New Async Model")

• ##### [From VR to Flat Screens: Bridging the Input and Immersion Gap](http://www.infoq.com/presentations/game-vr-flat-screens/ "From VR to Flat Screens: Bridging the Input and Immersion Gap")

• ##### [Cursor 3 Introduces Agent-First Interface, Moving beyond the IDE Model](http://www.infoq.com/news/2026/04/cursor-3-agent-first-interface/ "Cursor 3 Introduces Agent-First Interface, Moving beyond the IDE Model")

• Architecture & Design

• ##### [Meta Reports 4x Higher Bug Detection with Just-in-Time Testing](http://www.infoq.com/news/2026/04/meta-jit-testing-ai-detection/ "Meta Reports 4x Higher Bug Detection with Just-in-Time Testing")

• ##### [Cloudflare Launches Code Mode MCP Server to Optimize Token Usage for AI Agents](http://www.infoq.com/news/2026/04/cloudflare-code-mode-mcp-server/ "Cloudflare Launches Code Mode MCP Server to Optimize Token Usage for AI Agents")

• ##### [Zendesk Says AI Makes Code Abundant, Shifting the Bottleneck to “Absorption Capacity”](http://www.infoq.com/news/2026/04/zendesk-absorption-capacity/ "Zendesk Says AI Makes Code Abundant, Shifting the Bottleneck to “Absorption Capacity”")

• Culture & Methods

• ##### [Platform as a Product: Delivering Value While Balancing Competing Priorities](http://www.infoq.com/news/2026/04/platform-product-deliver-value/ "Platform as a Product: Delivering Value While Balancing Competing Priorities")

• ##### [Empower Your Developers: How Open Source Dependencies Risk Management Can Unlock Innovation](http://www.infoq.com/presentations/open-source-dependencies/ "Empower Your Developers: How Open Source Dependencies Risk Management Can Unlock Innovation")

• ##### [Tiger Teams, Evals and Agents: The New AI Engineering Playbook](http://www.infoq.com/podcasts/tiger-teams-evals-agents/ "Tiger Teams, Evals and Agents: The New AI Engineering Playbook")

• AI, ML & Data Engineering

• ##### [Anthropic Introduces Agent-Based Code Review for Claude Code](http://www.infoq.com/news/2026/04/claude-code-review/ "Anthropic Introduces Agent-Based Code Review for Claude Code")

• ##### [Lakehouse Tower of Babel: Handling Identifier Resolution Rules Across Database Engines](http://www.infoq.com/articles/lakehouse-sql-identifier-rules/ "Lakehouse Tower of Babel: Handling Identifier Resolution Rules Across Database Engines")

• ##### [Google Opens Gemma 4 Under Apache 2.0 with Multimodal and Agentic Capabilities](http://www.infoq.com/news/2026/04/google-gemm4/ "Google Opens Gemma 4 Under Apache 2.0 with Multimodal and Agentic Capabilities")

• DevOps

• ##### [CNCF Warns Kubernetes Alone Is Not Enough to Secure LLM Workloads](http://www.infoq.com/news/2026/04/kubernetes-secure-workloads/ "CNCF Warns Kubernetes Alone Is Not Enough to Secure LLM Workloads")

• ##### [OpenTelemetry Declarative Configuration Reaches Stability Milestone](http://www.infoq.com/news/2026/04/opentelemetry-declarative-config/ "OpenTelemetry Declarative Configuration Reaches Stability Milestone")

• ##### [New Rowhammer Attacks on NVIDIA GPUs Enable Full System Takeover](http://www.infoq.com/news/2026/04/rowhammer-attacks-nvidia/ "New Rowhammer Attacks on NVIDIA GPUs Enable Full System Takeover")

**The InfoQ** Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

• Get a quick overview of content published on a variety of innovator and early adopter technologies
• Learn what you don’t know that you don’t know
• Stay up to date with the latest information from the topics you are interested in

Enter your e-mail address

Select your country - [x] I consent to InfoQ.com handling my data as explained in this Privacy Notice.

We protect your privacy.

**May 7 | June 10, 2026 | Online** Architecture decisions are hard to validate while shipping. Join a **5-week online cohort** for **senior engineers, architects, and team leads** to pressure-test real decisions, apply practical frameworks, and work through challenges with a confidential peer group. Facilitated by Luca Mezzalira, Principal Architect at AWS, this cohort helps you: * Pressure-test real decisions. * Apply frameworks to real problems. * Publish on InfoQ.com and earn your certification. **RESERVE YOUR PLACE**

[Home](http://www.infoq.com/ "Home")[Create account](http://www.infoq.com/reginit.action "Create account")Log In[QCon Conferences](http://qconferences.com/ "QCon Conferences")Events[Write for InfoQ](http://www.infoq.com/write-for-infoq/ "Write for InfoQ")[InfoQ Editors](http://www.infoq.com/infoq-editors/ "InfoQ Editors")[About InfoQ](http://www.infoq.com/about-infoq/ "About InfoQ")[About C4Media](https://c4media.com/ "About C4Media")[Media Kit](https://get.infoq.com/infoq-mediakit/ "Media Kit")[InfoQ Developer Marketing Blog](https://devmarketing.c4media.com/?utm_source=infoq "InfoQ Developer Marketing Blog")[Diversity](https://c4media.com/diversity "Diversity")

#### Events

• ##### Online InfoQ Architect Certification

May 7, 2026

• ##### QCon AI Boston

June 1-2, 2026

• ##### Online InfoQ Architect Certification

June 10, 2026

• ##### QCon San Francisco

November 16-20, 2026

#### Follow us on

Youtube 232K FollowersLinkedin 26K FollowersRSS 19K ReadersX 57.1k FollowersFacebook 21K LikesBluesky New

#### Stay in the know

The InfoQ PodcastEngineering Culture PodcastThe Software Architects' Newsletter

General Feedback [feedback@infoq.com](mailto:feedback@infoq.com) Advertising [sales@infoq.com](mailto:sales@infoq.com) Editorial [editors@infoq.com](mailto:editors@infoq.com) Marketing [marketing@infoq.com](mailto:marketing@infoq.com)

InfoQ.com and all content copyright © 2006-2026 C4Media Inc.

Privacy Notice, Terms And Conditions, Cookie Policy

Close

[BT](http://www.infoq.com/int/bt/ "bt")