Log compaction is one of those Kafka features that makes sense once you've used it, but can be confusing when you first read about it. Learn how it works and what you need to know for the CCDAK exam.
Log compaction is one of those Kafka features that makes sense once you've used it, but can be confusing when you first read about it. It sits slightly outside the "Kafka is a stream of immutable events" that most introductions focus on. In simple terms, compaction exists for situations where Kafka is being used to represent state. But the state can and does change – so how does Kafka address this?
If you are preparing for the CCDAK, the exam will not just ask questions about configurations, but also test your understanding of concepts like log compaction and its guarantees.
Kafka logs are immutable which is to say records are written once and never changed. This eliminates coordination and locking and at the same time allows Kafka to scale horizontally and tolerate failures. The tradeoff is that any concept of "update" has to be expressed as a new record rather than a change in an existing one.
When records represent state, this tradeoff creates problems. A topic keyed by user ID might contain thousands of updates for a single user, but only a few of the updates are still meaningful. Other examples of state-based data include:
In these cases, consumers care about the current value, not the entire history. A service restarting does not need every past configuration change – it needs the latest one.
If Kafka only supported time-based retention, these topics would grow forever or force consumers to replay long histories that no longer matter. That is inefficient, and in some cases, actively harmful.
Kafka uses log compaction to keep only the latest record for each key. A background thread scans log segments and rewrites them, removing older records with duplicate keys. Unlike the delete policy that removes segments based on time or size, compaction triggers based on a dirty ratio and works by preserving only the newest value per key.
When log compaction is enabled on a topic, Kafka guarantees that at least the most recent record for each key will be retained.
Older records with the same key become eligible for removal during compaction. The important word here is "eligible". Compaction does not happen immediately, and Kafka makes no promise about when older records will disappear.
Here are a few constraints to remember:
From the perspective of a consumer reading the log from beginning to end, a compacted topic still looks like a normal Kafka log. You will see duplicates. You may see keys appear many times. Over time, those older entries will be removed, but consumers should not depend on that cleanup happening at any specific moment.
TIP – This "eventual cleanup" behavior shows up frequently in CCDAK exam questions to see if you understand the timing and guarantees of compaction.
One of the most common sources of confusion is mixing up retention-based deletion and log compaction. They are separate mechanisms, and Kafka can apply both to the same topic.
A topic can use both. For example, a compacted topic might still have a retention period that limits how long the latest values are kept. In that case, once the retention window expires, Kafka will delete the segment regardless of compaction rules.
TIP – For the CCDAK, it is important to remember that compaction does not override retention. Both policies apply independently.
Log compaction only works when records have keys. Kafka uses the key to determine which records are related and which can be discarded. If a record has no key, Kafka cannot decide whether it supersedes another record, so it is never compacted.
Enabling compaction on a topic that has unkeyed records does nothing. Those records behave exactly like they would in any append-only topic.
TIP – For the CCDAK exam, this may show up as a design question. A system that depends on compaction must choose keys deliberately and use them consistently.
Compaction is handled by background cleaner threads running on the broker. These threads scan closed log segments, identify records that have been superseded by newer records with the same key, and rewrite segments without those older entries. Compaction triggers based on a dirty ratio and also respects min.compaction.lag.ms, which prevents compaction of recent records. The active segment of a partition is never compacted. Only closed segments are eligible. Compaction rewrites data, so disk usage can temporarily increase. Cleaner threads work continuously but at a controlled rate.
Because this process works on closed segments and runs asynchronously, it is normal to see duplicate keys for long periods of time. This is expected and correct. Consumers should always be written to tolerate the duplicates.
Log compaction keeps the latest value for each key, but that still leaves an open question. How do you remove a key entirely? Kafka's answer is the tombstone message.
A tombstone is simply a record with a key and a null value. There is no special delete API and no separate concept of a delete operation. When Kafka encounters a tombstone for a key in a compacted topic, it treats it as a deletion marker. The tombstone itself is retained for delete.retention.ms (default 24 hours) so consumers can see that the key was deleted. After this period passes and compaction runs again, Kafka removes the tombstone too.
The tombstone itself sticks around for a while so consumers can see that the key was deleted. After enough time passes and compaction runs again, Kafka removes the tombstone too.
Kafka does not guess at your intent. If a producer stops sending updates for a key, Kafka assumes the last value is still valid. Tombstones exist because they make deletion explicit.
Consider a topic that tracks user accounts. If an account gets deleted, the producer must send a tombstone. Without it, any consumer rebuilding state from the log will see the last known value and think the account still exists.
Ignoring tombstones is a classic mistake. Developers often write consumers that only handle non-null values, then they wonder why their local state records should have been deleted.
When a consumer reads null value, it should treat that as a delete and remove any local state for that key. This applies both when processing live updates and when rebuilding state by replaying the topic from the beginning. Ignoring tombstones leads to stale data and incorrect state.
Kafka provides specific guarantees for log compaction, but it is just as important to understand what it does not guarantee.
Kafka guarantees that if any record for a key exists after compaction, it will be the most recent one. Consumers can reconstruct the latest state by reading the full log. Ordering is preserved within a partition.
Kafka does not guarantee that old records will be removed immediately, that consumers will see exactly one record per key, or that compaction will complete within a fixed time window.
TIP – These constraints will influence how you build systems and may appear in CCDAK questions to test your understanding of the delay between writing records and removing old ones.
If consumers care about the latest value per key and not the full history of changes, log compaction is probably the best choice.
| Good use cases | Bad uses cases |
|---|---|
| User profile updates | Financial transactions |
| Configuration changes | Audit logs |
| Product catalog updates | Order events |
| Feature flags | Click streams |
| Service discovery metadata |
That does not mean compacted topics replace databases or state stores. Kafka is still a log. You can't query it like a database. Compaction just reduces how much old data you have to read through.
TIP – For CCDAK scenarios, think about what consumers actually need. Ask what they need to know and whether the full history adds value.
Compaction is resource-intensive, using CPU, disk IO, and memory. On large clusters, aggressive compaction settings can slow down brokers.
Kafka lets you control how compaction behaves through configuration options. You can limit IO throughput and set thresholds for when segments get cleaned.
TIP – CCDAK requires memorizing these settings. If you'd like to learn more about configuration options, check out the Confluence docs on this topic.
Log compaction solves a specific problem. It helps to maintain the current state in Kafka without accumulating endless history. It's not complicated, but you do need to understand the tradeoffs:
For the CCDAK, the key is understanding what compaction guarantees and what it doesn't. If you built that mental model then you'll be able to navigate the exam scenarios.
This guide covered the fundamentals, but passing requires mastering the edge cases, configuration nuances, and real-world scenarios that trip up most candidates.
150+ pages covering every exam objective with real-world examples and zero filler content.
300+ exam-style scenarios with detailed explanations that teach you the why behind every answer.
Authentic exam questions covering all domains with detailed explanations.
Quick-reference guides for configurations, APIs, and patterns—perfect for last-minute revision.
Stay current with updates delivered whenever new content and features are released.
No 40-hour video courses. Focused content that gets you exam-ready in weeks, not months.
150+ pages, 600+ practice questions, 300+ exam questions, 5+ cheat sheets. One-time payment, lifetime value.
Published on February 13, 2025
Back to All Articles