Elasticsearch EventLog compression

Nowadays, everybody uses compression, deduplication, aggregation, etc., to reduce the pressure on storage and save the company money. In the EventLog world, you cannot delete or in any way modify the older data if you need it for compliance reasons. So, the only things left are compression and hot/warm architecture on the database side. This quick blog will concentrate on compression and testing it throughout the Elasticsearch versions.

Elasticsearch has two types of compression: default (lz4) and deflate in version 7 and zstd in version 8. Lz4 is usually the default compression in many industries because of the compression speed, and zstd has been an industry standard in the last few years when it comes to compressing data. You can find it almost anywhere (Proxmox VM compression, for example). The compression size is usually really good, but the decompression speed is excellent. It gives you an intangible performance drop when used in production.

Let's get down to business.

Our dataset is the daily index syslog_message_20250203, which has 129 318 135 messages inside it. These messages are usually company-running messages, such as routers, Linux, Windows machines, AD logs, Kubernetes logs, etc. When you export the index with programs like Elasticdump, its size in JSON is around 41GB!

In the table below, we can see the data we’ve been testing and its performance.

As we can see, zstd is a clear winner, with a bit of a performance penalty. We were baffled by Elastic for waiting to implement this compression for this long.

Overall, when using Elastic, it is recommended that you go to the Elasticsearch 8 version and use zstd compression by default. If you are upgrading from version 7 to version 8, don’t forget to reindex the data with new mapping so that you can enjoy space reduction right away. The mapping configuration is “codec”:”best_compression” inside data template.