Cloud message queues and event-driven architectures enable asynchronous communication between distributed services by decoupling producers and consumers through message brokers. These systems form the backbone of modern microservices, supporting patterns from simple point-to-point messaging to event streaming with exactly-once delivery guarantees and sophisticated routing. Kafka's evolution to KRaft mode in version 4.0 (March 2025) — removing ZooKeeper entirely — alongside the CloudEvents CNCF standard and a growing ecosystem of alternatives (Pulsar, NATS JetStream, Redpanda) makes broker selection and schema management more impactful than ever. Understanding tradeoffs between delivery models, queue types, and messaging patterns is essential for building scalable, resilient distributed systems that handle failures gracefully while maintaining data consistency.
What This Cheat Sheet Covers
This topic spans 16 focused tables and 160 indexed concepts. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
Table 1: Major Cloud Message Queue and Streaming Services
The broker or service you choose defines your latency floor, throughput ceiling, operational model, and available delivery semantics. Compare services by their use-case fit — ordered streaming, fan-out notifications, managed queues, or lightweight edge messaging — before optimizing configuration.
| Service | Example | Description |
|---|---|---|
kafka-topics.sh --create --topic orders --partitions 12 | • Distributed append-only log • high-throughput ordered streaming • as of Kafka 4.0 (Mar 2025) ZooKeeper fully removed — KRaft mode only • current stable: 4.3 (May 2026) | |
aws sqs send-message --queue-url $URL --message-body "order-123" | • Fully managed queue • Standard (at-least-once, best-effort order) or FIFO (exactly-once, strict order) • up to 256 KB per message • serverless scaling | |
aws sns publish --topic-arn $ARN --message "event" | • Managed pub/sub fanout • pushes to SQS, Lambda, HTTP, email, SMS • FIFO topics preserve order + deduplication | |
PutRecords(StreamName="clickstream", Records=[...]) | • Shard-based streaming for high-volume real-time data (clickstream, logs, telemetry) • multiple independent consumers • 24 h–365 d retention • ~200 ms latency | |
aws kafka create-cluster --cluster-name prod-kafka | • Fully managed Kafka on AWS • supports Kafka 4.1+ (Oct 2025), KRaft mode • MSK Express Brokers (2024+): 3× throughput, 20× faster scaling • MSK Serverless, MSK Connect, MSK Replicator for cross-region | |
sender.send_messages(ServiceBusMessage("payload")) | • Enterprise queue/topic with sessions, transactions, dead-letter queues • Queues (point-to-point) or Topics+Subscriptions (pub/sub) • 256 KB–100 MB messages | |
producer.send_batch(event_data_batch) | • Kafka-compatible managed streaming • partitioned consumer groups • Capture writes to Blob/ADLS • up to 90-day retention |