CCDAK Study Guide

Core Concepts

• Kafka: distributed event streaming platform for real-time data pipelines
• Ordering: guaranteed only within partitions, not across topics
• Replication: uses leader replica for writes and followers that replicate
• Safe pattern: replication.factor=3 with min.insync.replicas=2 and producer acks=all

Critical Producer Configs

Common Retryable Errors

Schema Registry Essentials

• Kafka stores bytes only - objects must be serialized
• Avro: language-neutral with JSON schemas, supports schema evolution
• schema.registry.url: must be configured on both producer and consumer

AdminClient Basics

• Async API: all operations return KafkaFuture, are eventually consistent
• Topic Management: createTopics(), deleteTopics(), describeTopics()
• Key Config: request.timeout.ms controls max wait time (default 30s)

KRaft Architecture

• Active Controller: manages all cluster metadata and elects leaders
• Raft Quorum: 3 or 5 nodes elect active Controller leader
• No Split Brain: Raft epochs prevent zombie controllers
• Metadata: brokers pull updates from controller via __cluster_metadata topic

Exactly Once Semantics

• Scope: atomic consume-process-produce-commit within Kafka only
• Transactional ID: unique persistent ID for zombie fencing
• Performance: overhead per transaction not per message
• Consumer Isolation: read_committed filters aborted transactions

Idempotent Producer

• Mechanism: PID and sequence numbers prevent duplicates from retries
• Configuration: enable.idempotence=true implies acks=all, retries>0, max.in.flight≤5
• Limitation: does not handle application restarts or multiple producers

Connect Architecture

• Connector: logical job definition that creates tasks
• Task: execution unit that moves data in or out of Kafka
• Worker: JVM process that runs connectors and tasks
• Offset storage: tracks progress in Kafka topics for recovery

Single Message Transforms

• Purpose: simple stateless message transformations
• Common SMTs: MaskField, Filter, RegexRouter, TimestampConverter
• Usage: configured in connector config with transforms property

Internal Topics

• config.storage.topic: stores connector and task configs
• status.storage.topic: stores connector and task status
• offset.storage.topic: stores source offsets for exactly-once delivery

Security Protocols

• PLAINTEXT: no encryption, no auth (internal only)
• SASL_SSL: recommended production standard with auth + encryption
• ACLs: Principal P is Allowed/Denied Operation O on Resource R
• Super Users: defined in server.properties bypass ACL checks

Critical Metrics to Monitor

Monitoring Priority

• Critical: UnderReplicatedPartitions, ActiveControllerCount, OfflinePartitionsCount
• High: RequestHandlerAvgIdlePercent, EventQueueSize, records-lag-max
• Medium: All throttle and throughput metrics

Stream Processing Basics

• Kafka Streams: embedded Java library, no separate cluster needed
• Fault Tolerance: uses local state stores backed by Kafka changelog topics
• Processing Guarantees: at_least_once vs exactly_once_v2
• Scaling: limited by number of input topic partitions

KStream vs KTable

• KStream: immutable append-only stream of events
• KTable: table of latest value per key, built from compacted topic
• GlobalKTable: fully copied to every instance for reference data

Windowing & Joins

• Window Types: Tumbling, Hopping, Sliding, Session
• KStream-KStream Join: requires window, joins events close in time
• KStream-KTable Join: no window needed, enriches with latest table value