SIEM Log Retention Best Practices: How Long to Keep Security Logs

Log retention is one of the most consequential — and most frequently mishandled — decisions in running a SIEM. Retain too little and you cannot investigate an incident or satisfy an auditor; retain too much on the wrong architecture and storage costs spiral. This guide covers how long to keep security logs, what drives those decisions, and how to architect retention so it is both compliant and affordable.

Why Retention Matters More Than People Think

The single most important statistic in incident response is attacker dwell time — how long an intruder is in your environment before detection. Industry data consistently shows median dwell times measured in weeks to months. If your logs only go back 30 days but the attacker was present for 90, your forensic investigation hits a wall exactly where it matters most. Retention length directly determines how far back you can reconstruct an attack.

What Drives Your Retention Period

1. Compliance requirements

Many frameworks mandate minimum retention periods. PCI-DSS requires at least one year of log retention, with three months immediately available for analysis. ISO 27001 expects retention aligned to a documented risk assessment. Regional data protection regimes — Malaysia's PDPA, Singapore's PDPA, Indonesia's UU PDP — expect you to be able to evidence security controls and reconstruct breaches, which implies meaningful retention. Financial sector rules (Bank Negara's RMiT, MAS TRM) have their own expectations.

2. Forensic and threat-hunting needs

Beyond the compliance minimum, retention should cover realistic attacker dwell times. Many mature security teams retain 12 months or more specifically so that when a breach is discovered, they can trace it back to the true point of initial compromise.

3. Operational and cost constraints

Retention is bounded by what you can afford to store and query. This is where SIEM architecture becomes decisive — and where many teams are forced into shorter retention than they would like.

A Sensible Default Retention Policy

These are starting points, not absolutes — your actual policy should be driven by your specific regulatory obligations and risk assessment.

Hot, Warm, and Cold Tiering

The key to affordable long retention is tiering. Not all logs need to be instantly queryable:

• Hot tier — recent logs (typically 30–90 days) kept in fast storage for real-time detection and active investigation
• Warm tier — older logs still queryable but on cheaper storage, accepting slightly slower queries
• Cold tier / archive — compressed long-term storage for compliance and deep forensics, retrievable when needed

This is where storage architecture makes or breaks your retention budget.

Why Compression Is the Deciding Factor

The economics of long retention come down to compression. A SIEM built on a columnar store like ClickHouse — which nPro uses — typically achieves around 10:1 compression on security log data. That means a year of logs that would occupy 10TB raw fits in roughly 1TB. On platforms with high per-GB storage overhead, the same retention is often financially prohibitive, forcing teams to cut retention precisely when they should be extending it. We cover this architectural difference in detail in our ClickHouse vs Elasticsearch comparison.

Practical Recommendations

1. Start from your compliance floor — identify the minimum retention your regulations require, then extend based on realistic dwell times
2. Tier aggressively — keep 90 days hot, archive the rest compressed; most queries hit recent data
3. Protect integrity — store logs in an append-only, tamper-evident manner so they hold up as evidence
4. Retain identity logs longest — authentication data is the highest-value forensic source and compresses well
5. Document your policy — a written, risk-based retention policy is itself a compliance artifact auditors expect

How nPro Handles Retention

nPro's ClickHouse foundation makes long retention affordable through high compression, and its configurable retention policies let you set different periods per log type and tier. Because nPro is self-hosted, your retention is bounded only by your own storage, never by per-GB cloud pricing — so you can keep the year-plus of history that real incident response demands.