Apache Kafka
A distributed event streaming platform used to move large volumes of real-time data between services — the backbone of data pipelines at most large-scale tech companies.
Apache Kafka is the dominant distributed event streaming platform, used to build real-time data pipelines, event-driven microservices architectures, and high-throughput messaging systems. It decouples data producers from consumers through a durable, ordered log of events (topics and partitions), enabling systems to process millions of events per second with replay capability. Kafka is operated by LinkedIn, Uber, Netflix, Airbnb, and most major technology companies, and it appears in senior backend engineer, data engineer, and platform engineer job descriptions at scale-focused organizations.
Typical time to job-readiness: ~6 weeks.
Learning Apache Kafka
Beginner
Understand the core concepts: topics, partitions, producers, consumers, and consumer groups. Run Kafka locally with Docker Compose and write a simple producer/consumer in Python or Java. Grasp why ordering guarantees are per-partition, not per-topic.
Intermediate
Kafka Streams for stateful stream processing, consumer group rebalancing and offset management, schema management with Avro and Schema Registry, and Kafka Connect for integrating databases and external systems without custom code.
Advanced
Partition design for throughput and ordering, exactly-once semantics, consumer lag monitoring, tiered storage for cost optimization, and Kafka cluster sizing and tuning. Senior interviews with Kafka involve system design: design a real-time leaderboard, fraud detection pipeline, or event sourcing system.
Key concepts
- Topics and partitions — data is organized by topic; each topic is split into partitions for parallelism
- Producers write messages, consumers read them — producers don't know who's consuming
- Consumer groups — multiple consumers share the load; each partition is assigned to one consumer in the group
- Offsets — each message has a position in the partition; consumers track their position independently
- Retention: Kafka durably stores messages for a configurable period — consumers can re-read past events
- Exactly-once vs at-least-once semantics — trade-off between correctness and performance
Common interview topics
- Explain how Kafka topics, partitions, and consumer groups work together
- What is the difference between Kafka and a traditional message queue like RabbitMQ
- How does Kafka guarantee ordering of messages
- What is consumer lag and how do you monitor it
- Design a system to process 1 million events per second using Kafka