How to Measure Your SIEM Compression Ratio (and Why the Published Figure Does Not Apply)

You have been told to measure your own compression ratio. Here is how, where the numbers live, and what to do with the answer once you have it.

Why a published ratio cannot tell you yours

A compression ratio is not a property of the datastore. It is a property of your data sitting in that datastore. Two organisations running the same software on the same version will land on different ratios, because the ratio depends on your schema, the cardinality of your fields and the codec applied to each column.

That is why no published figure, including any in a vendor datasheet, can tell you what you will see. Treat a quoted ratio as a starting assumption to be replaced by your own measurement, not as a number to size against.

Where the compressed and uncompressed sizes live

You do not have to estimate either side of the ratio. A columnar datastore already tracks both, because it needs them to manage its own storage. In ClickHouse, which nPro uses as its datastore, the figures are reported per data part and per column.

Per table

The system.parts table reports data_compressed_bytes and data_uncompressed_bytes for every active data part. Summing both and dividing gives the ratio for a table.

SELECT
  table,
  sum(data_uncompressed_bytes) / sum(data_compressed_bytes) AS ratio
FROM system.parts
WHERE active
GROUP BY table
ORDER BY ratio DESC

Per column

The system.columns table reports the same two figures per column. This is the more useful view when you are deciding where to change a codec, because it shows which fields are carrying the storage and which are already compressing well. The codecs themselves are set at table creation, in the column compression codecs section of the CREATE TABLE syntax.

How to run the measurement properly

The measurement is easy. Making it representative is the part people get wrong.

Use a full week, not a quiet weekend

Ingest a week of real logs before you read the numbers. A weekend sample is dominated by low-volume periods and heartbeat traffic, which compress unusually well and will flatter your ratio. A full week catches the weekday peaks that actually drive your storage.

Let the merges settle

Read the figures after the data has been through its normal merge cycle rather than immediately after ingest. Parts are written small and merged later, and the ratio you measure before that has happened is not the ratio you will live with.

Measure your log sources separately

Do not average the ratio across all your sources. A single blended figure hides the answer, because the sources behave nothing alike. Verbose, highly repetitive sources with low field cardinality compress far better than sparse, high-cardinality ones, and a blended number will be wrong for both.

Measure each major source on its own: firewall, endpoint, authentication, cloud workload. You will then know which source is driving your storage bill, which is the only way to decide whether dropping or trimming one is worth the detections you would lose.

When to measure again

Re-measure when the inputs change, not on a calendar. Onboarding a new log source, raising logging verbosity, changing a parser or adding sites all move the number. A ratio measured once at procurement and never revisited is the one that runs out of disk.

What to do with the number

Your measured ratio is the divisor in the sizing chain: daily raw volume, divided by the ratio, multiplied by retention days, plus headroom. Every other input is one you already know or can decide, which is what makes this the one worth measuring properly. The full method is in SIEM storage sizing.

If you are earlier in the process, start with what is a self-hosted SIEM, or compare the storage models in ClickHouse versus Elasticsearch for security logs.