https://t.co/n9ddlrBfZA

Fish Audio on X: "https://t.co/n9ddlrBfZA" / X

Fish Audio

@FishAudio

$

We Made Our Best Voice Model Free. Here's the GPU Math Behind It.

Fish Audio S2.1 Pro can clone a voice from 5 seconds of audio and gives developers word-level control over emotion, intonation, and pacing.

Most voice AI developers are paying for GPUs that are waiting.

That sounds absurd. But a naively served TTS endpoint handling one request at a time can sit below 10% GPU utilization. Much of the remaining capacity is lost to memory stalls and scheduling overhead—and it still shows up in the API bill.

So we rebuilt the serving stack behind S2.1 Pro.

The result:

The same request volume that previously needed four H200s now runs on one.

At launch, S2.1 Pro cost roughly one-sixth as much as ElevenLabs' comparable API tier.*

In our launch benchmark, it ran 2× faster than Cartesia.*

Our custom kernels run 2.1–4.3× faster than a torch.compile-fused cuBLAS baseline at decode shapes.

Continuous batching scales aggregate throughput by roughly 52× from concurrency 1 to 64.

Cluster-wide GPU utilization moved from roughly 50% to 90%+.

*Price and performance comparisons refer to Fish Audio's launch benchmark and public pricing at that time.

Free access to S2.1 Pro is now extended through November 2026.

That change in unit economics is what lets us make S2.1 Pro available free to developers—not a distilled or stripped-down version.

Here’s what changed under the hood.

The GPU problem you’re paying for

Every TTS API call runs an autoregressive decode loop: generate an audio token, read from the KV cache, then repeat.

Each step involves a small matrix multiply—typically M ≤ 128—followed by a memory read. Those shapes are too small to keep the GPU saturated. The CUDA cores finish quickly, then wait for the memory subsystem to catch up.

At low concurrency, the loop runs in short bursts separated by scheduling overhead. The hardware is there. The work simply is not filling it.

This is a structural serving problem, not a pricing-page problem. The fix is to make the hardware spend less time waiting.

We optimized the entire stack

There was no single 4× breakthrough. The gains came from fixing several layers and letting them compound:

Custom CUDA kernels to remove per-token compute waste

FP8 quantization to reduce memory-bandwidth pressure

Continuous batching to keep the decode loop full

GPU scheduling to turn idle capacity into useful work

Owned network and storage infrastructure to remove cloud markup and extra latency hops

Custom kernels for the shapes TTS actually uses

Standard GEMM libraries are excellent at training and large-batch prefill. TTS decode is different.

In a standard SwiGLU MLP layer, activation, gate multiplication, and down projection can become separate kernel launches. Intermediate results are written to HBM and read back between operations. At M=1, that round trip costs more than the compute itself.

We built fish-scales-ops to target these decode shapes directly:

bsgemm: 128×128 block-scaled FP8 for NVIDIA Hopper/H200

mxfp8: 1×32 microscaling with OCP UE8M0 for Blackwell

Custom FlashAttention paths for the same serving workload

In our internal benchmarks on equivalent hardware, with the full methodology published in the repository, the kernels run 2.1–4.3× faster than a torch.compile-fused cuBLAS baseline at decode shapes.

The important detail is not just lower precision. It is keeping intermediate traffic off the memory bus and fusing the work around the shapes that production decode actually produces.

FP8 only works if quality survives

Quantization can make a benchmark look great while quietly damaging the output.

We learned that the hard way. During mxfp8 development, a weight-layout bug produced roughly 53% relative logit error while kernel latency still looked normal. A performance benchmark alone would have passed it.

Every inference-stack change now goes through two gates:

A correctness gate: quantized greedy decode must match the bf16 baseline token-for-token on the first five audio frames of a fixed reference input.

A performance gate: per-request decode throughput must remain within 5% of frozen baselines at concurrency 1, 16, and 64.

Both must pass before deployment.

S2.1 Pro Free uses the same model weights as paid traffic. Quantization happens in the serving layer, with correctness checked against the unquantized baseline.

Continuous batching: 52× throughput for 37ms more latency

Static batching waits for a batch boundary. Continuous batching lets new requests join the active decode batch as slots become available.

On one H200 with bsgemm FP8:

At concurrency 1: 154 tokens/s aggregate throughput, 73.2ms p50 time-to-first-byte

At concurrency 64: 8,006 tokens/s aggregate throughput, 109.8ms p50 time-to-first-byte

That is roughly 52× more throughput for about 37ms more TTFB.

Batching usually means a painful latency trade-off. Here, most of the gain comes from filling capacity that was already sitting idle.

Latency-sensitive voice agents use a separate real-time path optimized for time-to-first-audio rather than maximum aggregate throughput.

The rest of the GPU should not sit idle either

Inference traffic is bursty. Even after optimizing each request, an inference-only cluster can spend long periods underfilled.

We run inference through Kubernetes and training through Slurm, with elastic data-cleaning and preprocessing jobs as the lowest-priority fill work. They use spare capacity, checkpoint safely, and are preempted as soon as real traffic or training needs the GPU.

That scheduling layer moved utilization from roughly 50% to 90%+.

We also own more of the infrastructure around the GPUs: our ASN, 300Gbps connectivity, Cloudflare peering, and tiered storage. Removing cloud egress markup also removes a network hop from the user’s request.

Four H200s became one

Put the layers together and the same request volume that previously needed four H200s now runs on one.

This is the part that changes the economics. We did not eliminate inference cost. We reduced it structurally—enough to make high-quality free access viable without swapping in a smaller model.

The free API uses the same S2.1 Pro model weights, serving stack, and endpoint as paid traffic. Developers get voice cloning from a five-second reference, 83 languages, and natural-language control over emotion, pacing, and intonation.

We open-sourced the kernels

The core kernel library is available now: fish-scales-ops.

It includes production-grade FP8 GEMM and FlashAttention paths for NVIDIA Hopper and Blackwell, with CUDA Graph capture support. The underlying foundation model, Fish Speech, is also open-weight.

If you serve an autoregressive model—not only TTS—the small-M decode problem is likely familiar. The kernels and benchmark methodology are there to inspect, reproduce, and build on.

Start building with S2.1 Pro Free

Use the same Fish Audio API endpoint and set:

model: s2.1-pro-free

No credit card. Voice cloning included. No hard character cap under Fair Use.

Get started now.

Free access is now extended through November 2026. The free tier is best-effort and does not include SLA or latency guarantees; some commercial scenarios have restrictions.

Check current availability and production options.

/$

7:19 PM · Aug 27, 2026

11.8K

Views

17

222

198