Shopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens

Shopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens - InfoQ

InfoQ Homepage News Shopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens

AI, ML & Data Engineering

When AI Accelerates Development, Can Your CI Pipeline Keep Up? (Webinar Oct 8th)

Shopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens

Sep 03, 2026 2 min read

by

• Sergio De Simone

#### Follow us on

Youtube

232K Followers

Linkedin

26K Followers

Instagram

New

RSS

19K Readers

X

57.1k Followers

Facebook

21K Likes

Bluesky

Listen to this article -

0:00

Audio ready to play

Your browser does not support the audio element.

Normal

1.25x

1.5x

Like

new dropdown rading list

• Reading list

Shopify's engineering introduced Gisting, a novel technique for compressing long LLM prompts into a smaller set of learned "gist" tokens, improving throughput and reducing inference cost.

Spotify emphasizes that replacing lengthy text for concise gist tokens at inference time reduces end-to-end latency, drops infrastructure costs, and boosts token throughput without modifying the model's core weights.

The company says that gisting reduced the Sidekick GraphQL agent’s system prompt from about 6000 tokens to 1500 gist tokens without sacrificing prediction quality. This implies a 4:1 reduction in context size:

At 350 requests per minute (RPM), the median time to first token (TTFT) dropped from 438ms to 354ms, the median end-to-end request latency dropped from 6.8s to 4.2s, and throughput rose from 20.2 to 23.4 queries per second (QPS).

These improved metrics allowed Spotify to reduce the number of allocated GPUs.

Gisting is based on a technique pioneered in a 2022 paper, " Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models " , and consists of a two-step process to learn the embeddings of the new compressed gist token. In a first pass, the teacher pass , the model is run with the real prompt to derive the teacher logits of the response. In the student pass , the model is run with the gist tokens to derive the student logits. Finally, the gist are trained to minimize the KL divergence between the teacher logits and the student logits, that is until the student's predictions closely match the teacher's.

When training finishes, we write the gist embeddings straight into the model's embedding matrix, and register the new gist tokens as special tokens in the model’s tokenizer. The model loads and runs like any other at inference time: no custom attention mask, extra encoder, or special serving path.

The key advantage of gisting is that the model does not process a conventional summary of the original prompt, but rather a learned representation designed to make the LLM to behave as close as possible to how it would if it had seen the original prompt.

Gisting can reduce latency and increase throughput. In Shopify's case, Time to First Token (TTFT) dropped from 438ms to 354ms, and end-to-end latency fell from 6.8s to 4.2s. At the same time, queries per second (QPS) increased from 20.2 to 23.4, allowing engineering teams to scale down overall GPU allocation.

As a final note, Shopify also emphasizes that gisting is complementary to other optimization techniques, such as prefix caching . Prefix caching avoids recomputing the KV tensors for cached prompt sequences, but the model must still process those cached tensors during the decoding phase. Gisting further reduces this overhead by replacing a long prompt with a shorter sequence of learned gist tokens. The two optimizations therefore compound, and Shopify uses them together.

There is much more to gisting than can be covered here. Make sure to read the original article if you are interested in the full details, which covers topics such as the role of autosearch in tuning the Gisting process and other implementation details that significantly affect performance.

main wrapper for authors section

About the Author

section title

main wrapper for each author

#### Sergio De Simone

Show more

Show less

#### This content is in the AI, ML & Data Engineering topic

##### Related Topics:

• Development

• Architecture & Design

• AI, ML & Data Engineering

• Model Inference

• Large language models

• Agents

• Performance

• Related Editorial

• Related Sponsors The Practical Agentic Coding Playbook

• Related Sponsor Automated code transformation at scale. Fix vulnerabilities, migrate frameworks, and modernize legacy code across every repo. Try Moderne Free .

The InfoQ Newsletter

A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example

We protect your privacy.