Zstandard (Zstd): Modern Compression for the Web and Why Cloudflare Chose It

Zstandard, commonly shortened to Zstd, is a modern data compression algorithm developed by Facebook (now Meta). The algorithm is designed to provide an optimal combination of compression speed and high compression ratio — two things that traditionally often had to be sacrificed one for the other. In the web world, Zstd is increasingly popular especially in CDN services like Cloudflare because of its efficient compression capability and its support for modern web protocols. This article discusses Zstd’s basic concepts, its comparison with gzip and Brotli, how Content-Encoding works in HTTP, and the technical reasons behind Cloudflare’s decision to adopt this algorithm widely.

What Is Zstandard (Zstd)?

The main goal of compression algorithms like Zstd is reducing the size of data sent over the network, without sacrificing the speed of the compression or decompression process. Compared to gzip — the compression algorithm that has been the web standard for years — Zstd offers three main advantages: a higher compression ratio (smaller compressed file sizes), faster compression and decompression processes, and streaming support making it suitable for large files or dynamic content.

Zstd isn’t an algorithm exclusive to the web. Its uses span various contexts, including file storage for compressing long-term stored data, backup processes needing fast compression for large amounts of data, and HTTP response compression in web servers and CDNs — the context that’s the main focus of this article.

Zstd was released as open source by Facebook in 2016, and since then has been widely adopted across various systems — from the Linux kernel, the ZFS and Btrfs file systems, to various modern databases and CDNs. This wide adoption is largely driven by the balance between performance and compression ratio that previous-generation algorithms find hard to match.

One unique feature distinguishing Zstd from most other compression algorithms is its support for dictionary compression — the ability to train a “dictionary” from a representative set of data, then use that dictionary to compress similar small files with a far better ratio than compression without a dictionary. This is very useful for scenarios like compressing many small JSON responses with similar structures — conventional compression algorithms struggle to achieve high ratios on small data because there aren’t enough repeating patterns within a single file to exploit, while dictionary compression allows those patterns to be “remembered” in advance from a reference dataset.

Adjustable Compression Levels

Zstd provides a wide range of compression levels — generally from level 1 (fastest, lowest compression ratio) to level 22 (slowest, highest compression ratio). This flexibility lets users adjust the trade-off between speed and compressed output size per their specific needs — low levels suit real-time scenarios very sensitive to latency, while high levels suit archival data compression done once but read many times, where compression time isn’t as critical as the resulting final size.

This characteristic is what makes Zstd flexible enough for various contexts at once — from real-time HTTP response compression needing low levels for minimal latency, to large-scale backup data compression that can use high levels because its processing time isn’t urgent.

Content-Encoding and HTTP

In the HTTP protocol, the Content-Encoding header tells the client (browser or crawler) how the sent data has been compressed, so the client knows which decompression method to use before being able to read the data’s contents. An example response header showing Zstd usage:

Content-Type: application/xml
Content-Encoding: zstd

This header means the sent XML file has been compressed with Zstd, and the client must decompress it first before being able to parse the XML contents. This process usually runs automatically at the browser or HTTP client level, without requiring manual intervention from the end user.

Several Content-Encoding values commonly found on the web:

• gzip — classic compression, broadly compatible across almost all browsers and clients
• br (Brotli) — modern efficient compression, especially for text-based content like HTML and CSS
• zstd — very fast with a high compression ratio, suitable for CDNs and large files

All three algorithms can coexist in the same web infrastructure — the server or CDN usually picks the algorithm used based on what the client supports, via negotiation using the Accept-Encoding header the client sends on every request.

Zstd vs gzip Comparison

To understand Zstd’s position compared to older compression approaches, here’s a direct comparison with gzip on several key aspects:

The most striking difference is in the combination of compression ratio and speed. In general, compression algorithms have a classic trade-off: higher compression ratios usually require longer processing time. Zstd is specifically designed to shift this trade-off — providing a compression ratio equal to or better than gzip, while maintaining high processing speed, even at aggressive compression levels.

flowchart LR
    A[Raw data] --> B{Choose an algorithm}
    B -- gzip --> C[Medium compression, fast]
    B -- zstd --> D[High compression, very fast]
    C --> E[Larger transfer size]
    D --> F[Smaller transfer size, lower latency]

Compression Supported by Modern Browsers

Modern browsers generally support several HTTP compression methods at once, and will tell the server which methods they support via a request header. Here’s a summary of the most commonly encountered compression methods:

Safari and some older browser versions may not support Zstd natively yet. Because of this, CDNs applying Zstd usually still provide a fallback to gzip or Brotli based on the Accept-Encoding header the browser sends, so content remains accessible without errors even if the client doesn’t support Zstd.

This fallback mechanism is important to understand because it means developers and site owners don’t need to worry about Zstd making their site inaccessible to some visitors — Accept-Encoding negotiation at the HTTP level automatically handles this compatibility without additional configuration on the application side.

Why Zstd Is Used at Cloudflare

Cloudflare is a Content Delivery Network (CDN) distributing content to edge servers spread around the world, with the goal of bringing content closer to visitor locations for lower latency. Several main reasons Cloudflare adopted Zstd widely:

1. High speed — faster compression and decompression than gzip directly reduces latency, because the time needed to compress content at the edge server and decompress it on the client side becomes shorter.
2. High compression ratio — smaller data transfer sizes mean significant bandwidth savings, both for Cloudflare as the infrastructure operator and for visitors accessing from limited-bandwidth connections.
3. Streaming-friendly — support for large and dynamic content makes Zstd suitable for videos, constantly updated content feeds, or XML sitemap files that can be large on sites with many pages.
4. Broad modern support — the majority of modern browsers and clients already support Zstd, so its adoption doesn’t sacrifice compatibility for most users.

With Zstd, Cloudflare can speed up content delivery to visitors while simultaneously reducing bandwidth consumption across its entire edge network — a combination that directly impacts the performance of sites served through its infrastructure.

Impact for Users and Developers

For developers managing a website or web application, there are several practical implications to understand about Zstd:

Zstd generally doesn’t need to be manually enabled at the application level or in static site generators like Hugo. This compression is usually configured and handled at the CDN level (like Cloudflare) or the web server, not at the application code level — meaning developers don’t need to write additional logic to enable Zstd, just make sure the CDN or web server being used already supports it.

Various types of web assets — XML sitemaps, RSS feed files, CSS, JavaScript, and other static assets — can be compressed with Zstd by the CDN, while maintaining the correct Content-Type per the original content type (for example application/xml for sitemaps, regardless of the compression algorithm used to send them).

Modern clients will automatically decompress received files based on the included Content-Encoding header, without requiring manual intervention from users or developers of applications running on the client side.

Another implication often overlooked is Zstd’s impact on infrastructure operational costs at large scale. For CDN operators serving very high traffic volumes, bandwidth savings of even a few percent can mean significant absolute cost savings, given the scale of traffic handled. The combination of a better compression ratio with still-high processing speed makes Zstd economically attractive, not just from a technical performance standpoint — something that also helps explain why large CDN operators like Cloudflare are willing to invest in adopting this algorithm comprehensively across their entire edge network.

For developers wanting to verify whether Zstd is truly used for the content they manage, the simplest way is checking the response headers via browser developer tools or via command-line tools like curl -I with an explicit Accept-Encoding: zstd header included. If the server or CDN supports it, a Content-Encoding: zstd header will appear in the response, indicating the received content has been compressed with this algorithm.

Summary

• Zstd (Zstandard) is a modern compression algorithm from Facebook that shifts the classic trade-off between compression ratio and speed — both high at the same time.
• Compared to gzip, Zstd offers a higher compression ratio, faster compression and decompression, and streaming support suitable for large files or dynamic content.
• The HTTP Content-Encoding: zstd header tells the client the data has been compressed with Zstd, and compatibility negotiation is handled automatically via Accept-Encoding.
• Modern browsers (Chrome, Firefox, Edge, Opera) support Zstd, while browsers that don’t yet support it still automatically get a fallback to gzip or Brotli.
• Cloudflare adopted Zstd because of its high speed, large compression ratio, streaming-friendly nature, and already-broad modern support.
• For developers, Zstd doesn’t need to be manually enabled at the application level — it’s usually already handled at the CDN or web server level.
• Zstd is an ideal choice for modern HTTP compression, including XML sitemaps, RSS feeds, and various other static assets.