Redis is the most popular key-value database store. Redis is used in the majority of large-scale applications. In fact, we at Codeforgeek use Redis to handle frequent caching. You might be reading this article from the Redis cache.
You can learn more about Redis and Database caching here.
While building large-scale distributed systems, we often end up in a scenario where we have to perform a huge amount of data ingestion or data transfer from one system to another. The only feasible way to perform this is through streaming. Distributed engineers generally use streaming software such as Apache Kafka to perform data streaming and ingestion.
Redis with the support of version 5.0 offers an innovative way to manage streams. Redis Stream is a built-in feature and it offers a data structure that developers can use to perform data ingestion, data consumption, creates a data channel between producer and consumers, and so on.
In this tutorial, we are going to explore how to use Redis streams and how you as a distributed systems engineer can incorporate Redis streams in your architecture.
Redis streams capabilities
Redis is used in various use cases across the system. This includes content caching like we do at Codeforgeek, session stores ( Learn about using Redis as session store ), real-time analytics, message brokers, etc.
While Redis already provides publish/subscribe feature, with the release of Redis streams, Redis offers the following capabilities:
- Collect large amounts of data.
- Create a data channel between many consumers and producers.
- Data ordering is effectively managed even though the rate of data ingestion and data consumption is different.
- Offline data persistence support.
- Asynchronous communication between producer and consumer.
- Easily scalable to large number producers and consumers.
Like I mentioned earlier, Redis streams are built-in into Redis. You don’t need to perform extra installation or management of heavy software. It’s right into Redis.
Let’s jump right into Redis streams and learn how to use them.
Using Redis Streams
Redis Streams is an append-only log-based data structure. Redis streams offer commands to add data in streams, consume streams and manage how data is consumed.
Redis streams can have one-to-one communication or one to many or many to many communication streams between producers and consumers. Redis streams also offer consumer groups so that they can consume only the information that they are supposed to receive.
Let’s use Redis streams commands. I am assuming that you have Redis version 5.0 or above already installed in your system. If not, please navigate to the Redis download page to download the latest version of Redis.
Alternatively, you can also use cloud based service like RedisGreen to use Redis and run commands without having to install and manage it in your system. Redisgreen also offers encryption support using TLS.
Once you have Redis installation sorted out, start the Redis server if you are using it in your system using the following command.
Some installers already run the Redis server in the background, so if you get some error, don’t worry about it.
Now, let’s start the Redis command line shell using the following command.
Let’s add some data in Redis streams. The default and easy way is to run this command.
Here, mystream is a name of the stream. Name is the key and Shahid (My name) is the value.
When you run this command, Redis will generate the ID for your stream data.
We can also add our own ID while adding the records. For example,
However, you should rely on a system generated unique ID.
Let’s consume data from the Redis streams.
To read everything from start of the stream to the end, we can use the following command.
Here we are giving the name of the stream and start index i.e 0. This command will return all the records present in the stream.
We can also limit the records by providing the record count.
We can also iterate through the stream using the XRANGE command.
For example,
The command shown above will read the record from the stream between the ID mentioned in the command.
If we don’t know the ID of the records, we can pick the number of records using the COUNT operator.
The command shown above will return the record from 10 records iterating from the beginning of the stream.
Redis streams offer more commands and ways to manage streams. You can learn more about it in the documentation page here.
