AiVibeApache Kafka
Apache Kafka is an open-source distributed event streaming platform that stores streams of records in partitioned, replicated, append-only logs, letting many applications publish and consume data in real time at high throughput.
Apache Kafka is an open-source distributed event streaming platform, originally developed at LinkedIn and now an Apache Software Foundation project. Producers write records to named topics, which are split into partitions and stored as append-only logs across a cluster of brokers. Each partition is replicated to other brokers for fault tolerance, and consumers read records by tracking their position, called an offset, so many independent applications can read the same data.
Kafka is used for real-time data pipelines, log and metrics aggregation, event-driven microservices, change data capture and stream processing. Kafka Connect provides connectors for moving data between Kafka and databases, object stores and other systems, and the Kafka Streams library processes data within applications. In manufacturing, Kafka is often used between edge or MQTT layers and enterprise analytics. Managed services include Amazon MSK and Confluent Cloud, and Azure Event Hubs offers a Kafka-compatible endpoint.
Running Kafka well requires planning for partition counts, replication factor, retention, disk throughput and consumer group behaviour. Ordering is guaranteed only within a partition, so the choice of record key matters. Recent versions use the built-in KRaft consensus protocol for cluster metadata instead of Apache ZooKeeper. Security features include TLS encryption, SASL authentication and access control lists, which need to be configured because they are not enabled by default.
Key points
- Originally developed at LinkedIn; now an Apache Software Foundation project
- Topics are split into partitions; ordering is guaranteed only within a partition
- Consumers track offsets, so many applications can read and replay the same data
- TLS, SASL and ACLs must be configured, as security is not enabled by default
Where AiVibe comes in
AiVibe Software Services delivers cloud solutions on AWS, Microsoft Azure, Google Cloud or on-premise, together with cloud security, legacy modernisation, data analytics and AI and machine learning services.