Learn how to use RabbitMQ Streams using both Core and Plugin in this tutorial
Recently I published an article about RabbitMQ and how to use it in Go to build a Event Driven Architecture.
All Images in this article is made by Percy Bolmér. Gopher by Takuya Ueda, Original Go Gopher by Renée French (CC BY 3.0)
That article covers the basics of RabbitMQ, this time, however, we will increase our knowledge and learn about Streams. This article expects that you have a basic understanding of RabbitMQ Queues; If you don’t I highly recommend my previous article.
I have always relied on RabbitMQ and Kafka combined. RabbitMQ for most stuff and the times when I needed replayability, time traveling, or events that persisted then I used Kafka.
Time traveling is when you can replay events from a certain time in your system
Using two brokers became more complex since I now needed to deploy Kafka and handle both brokers.
RabbitMQ has released support for a feature that they call Streams in 2021. It’s taken time for me to test it, but I finally did and am glad that I did.
Streams are designed to specifically help us with the problems mentioned above, which the industry usually uses Kafka for.
Each event sent on the RabbitMQ streams is written to the file system. and stored forever unless a specific TTL is configured. Each event is indexed which allows us to easily replay events, and trigger time traveling.
RabbitMQ Streams can be used by companies to build robust event-driven data pipelines that ensure reliable and timely data delivery with high integrity. The ability to replay and view events afterward can also be used for audibility and reproducibility.
All the code that will be used can be found on my Github
Another reason for streams was that each consumer had to create their Queue in fan-out…