Better speech synthesis for audiobooks

Better speech synthesis for audiobooks - Amazon Science

Conversational AI

Better speech synthesis for audiobooks

“Controllable text to speech” makes speech more expressive, character voices more distinct and consistent.

By

Arent van Korlaar

October 23, 2026

6 min read

Share

• Copy link

• Email

• X

• LinkedIn

• Facebook

• Line

• Reddit

• QZone

• Sina Weibo

• WeChat

• WhatsApp

分享到微信

x

Listen • 8:53 min

Key takeaways

• Audible's controllable text-to-speech (TTS) system profiles book characters and attributes dialogue with 97% accuracy in English (94% in Spanish).

• Rendering distinct character voices improved dialogue differentiation by 3.7 points on a MUSHRA test (a 0–100 listener-rating scale), with statistically significant listening-experience gains for 8 of 10 voices across both English and Spanish.

• The system controls delivery via natural-language captions (e.g., "whispering nervously", "shouting in anger") and fixes each character's voice profile once at the book level, so the model renders that character consistently from beginning to end.

Was this answer helpful?

Text-to-speech systems — which convert written text into synthetic speech — display impressive naturalness in applications such as chatbots and virtual assistants, where interactions last seconds. But they tend not to fare as well with longer-form content.

When applied to audiobooks, for instance, conventional text-to-speech (TTS) models fail to reliably change speaking style between characters, as a human narrator would. This creates two problems for listeners: (1) distinguishing narration from dialogue and (2) identifying which character speaks which lines.

To address this problem, the Audible technology team is using controllable TTS, meaning the ability to control dialogue delivery with natural-language speech descriptions (e.g., "speak in an enthusiastic, high-pitched voice"). This allows the TTS model to leverage the deep textual understanding of large language models (LLMs) to synthesize high-quality speech.

Our approach involves three key innovations. First, we developed a novel data synthesis pipeline to produce speech data captioned with high-quality descriptions of speech characteristics. Training on this data enables our model to understand captions and produce audio consistent with them at inference time.

Second, we created a system that automatically profiles characters in books and identifies their dialogue with 97% accuracy. It also provides captions for that dialogue, which act as stage directions for the model.

Third, we used the first two techniques to synthesize audiobooks featuring distinct character voices, improving dialogue differentiation by 3.7 points on a MUSHRA (multiple stimuli with hidden reference and anchor) test, a standardized method where listeners rate audio quality on a scale from 0 to 100. Across tests in both English and Spanish, we observe statistically significant improvement in listening experience for eight out of ten voices.

Controllable text to speech assigns more self-consistent and distinct speaking styles to different characters in an audiobook — officers on the bridge of a spaceship, for example.

Spaceship dialogue

TTS for audiobooks

Subtle defects in synthetic speech, such as unnatural prosody or monotone delivery, are tolerable for short stretches but become increasingly grating over time — and audiobook listeners engage with the same voice for 10 to 15 hours per book. This is why off-the-shelf TTS models often fail to produce enjoyable, natural audiobook narrations.

Furthermore, general-purpose TTS models are designed to process individual utterances with minimal context. That’s a reasonable approach when responding to questions like "What's the weather today?", but it’s inadequate for audiobooks, where narrative coherence depends on understanding the surrounding story context. A TTS model reading dialogue needs to know whether the speaker is the protagonist confronting an antagonist or a minor character making small talk, information that may only be available in earlier chapters.

Finally, general-purpose models struggle to produce character voices in a consistent and natural way across the span of an entire book, often voicing a character in a markedly different way in chapter 1 than in chapter 10.

Controllable text-to-speech

I. The data synthesis pipeline

Whereas conventional TTS models are trained on text-speech pairs, a controllable-TTS model also requires annotations that describe how the speech sounds. The speech in the training set also needs to be highly expressive, covering the wide variety of dimensions — pitch, speaking rate, character voices, emotions, speaking styles, etc. — the model must control. Previous TTS models couldn't handle that kind of expressivity, so most existing training data is neutral in tone. We expanded the training sets through two approaches: new recordings and human captioning of existing audio data.

While there are some datasets that include descriptions of prosodic features, each has its own description format: some are long, some are short, and some use single tags. This limits the flexibility of model inputs and tends to make the model overfit the formats of specific datasets. To improve generalization, we use LLMs to rewrite the original prompts into different formats, varying length, verbosity, and structure (single tags vs. free-form natural language).

Second, the human-in-the-loop data collection procedures do not scale to the size of the datasets we need. Therefore, we complement the collected data with a large amount of synthetic data. We use two approaches:

1. Expert-model pipeline. This approach uses expert models and signal-processing algorithms such as pitch extraction, speaking rate extraction, and emotion detection to extract numerical features from audio data. We then feed these features and classification labels into an LLM to generate speech captions.

1. Speech captioner. We developed a model that performs a task opposite that of controllable TTS: it takes speech and outputs a description of prosodic features. Using this speech captioner, we can annotate larger amounts of data. We use the annotated data as seed data to train the initial controllable-TTS model, and we also use it to continue training the speech captioner itself. We then use these models to generate synthetic data, and we improve them both iteratively to create a synthetic-data flywheel.

Anxious speech

Anxious

More anxious

Sophisticated speech

Sophisticated

More sophisticated

Shouting

Shouting

Shouting louder

II. BookLLM: Adapting to the audiobook domain

The audiobook domain presents additional challenges for a controllable-TTS system, since different characters need to have distinct but consistent voices. For this, we developed BookLLM, a separate component that leverages off-the-shelf LLMs to identify unique characters and uses information in the book — for instance, age, gender, personality, and role — to define their voice characteristics. Then the model determines which character speaks where and captions dialogue with descriptions of how it should be spoken.

We evaluated BookLLM on 25 English and Spanish books in the mystery, sci fi, and romance genres. We collected ground truth annotations using two human annotators. In English and Spanish, the system achieved 97% and 94% accuracy, respectively, in identifying which character speaks where, including distinguishing between narration and dialogue

III. Bringing it all together

Bringing the two components together yields a complete audiobook-synthesis pipeline. Given a new book, BookLLM first reads the entire text to build a cast list: it determines the unique characters, ranks them by importance, and derives a voice description for each. Rather than inventing an unbounded set of voices, it assigns each character a distinct profile from a curated voice catalogue, deliberately maximizing the contrast between the most prominent characters so that listeners can tell them apart. Minor and one-off characters fall back to the default narrator, avoiding a proliferation of barely used voices.

BookLLM then walks through the book and attributes every passage to a speaker — the narrator or a specific character — and, for each line of dialogue, writes a caption describing how it should be delivered, such as "whispering nervously" or "shouting in anger". These captions are exactly the kind of natural-language speech description that the controllable-TTS model was trained to follow.

At synthesis time, each unit of text is passed to the TTS model, together with its speaker's voice profile and, if necessary, its stage direction caption. Because a character's voice profile is fixed once at the book level, the model renders that character consistently from beginning to end, avoiding general-purpose models’ tendency to drift, or voice a character differently in chapter one than in chapter ten. The per-line captions layer moment-to-moment expressivity on top. The result is a full-length audiobook in which characters sound distinct, consistent, and expressive that ultimately provides a more delightful listening experience.

Controllable-TTS demo

Research areas

• Conversational AI

Tags

• Text-to-speech (TTS)

About the Author

Arent van Korlaar is an applied scientist with Audible.