Email Queue Design

Emails are very important part of any web/mobile based company. Whether you are a SaaS-based business or standalone mobile app, you might be sending a huge number of emails to your customers.

Generally, we classify those emails into two categories:

1. Transactional email.
2. Promotional email.

Transactional emails are a key part of almost any software I can think of. Transactional emails are those which are used to fulfill certain action taken by the user. For example, account verification, password reset and other depending on your application.

Transactional emails are generally sent as a synchronous email i.e the request is dependent on the email.

Promotional emails can be anything related to updates about your product, new launch, new releases etc sent to users. Promotional emails are sent in bulk to thousands or in some case millions of email account holder or subscribers.

I have been wondering since long about how to design such as a system which handles this volume of data in form of emails.

In this article, we are going to do that and I need your inputs as well about my thought of the design.

Let’s begin.

Promotional emails can be sent asynchronously so that allows us to decouple the application in order to handle large payload.

I think queues will come handy in such cases. So here is my design.

Let’s look at each step.

Step 1

There has to be a source from where we will pull out the email list and content of the email to send to that list. Email list consists of email accounts and extra details if applicable such as name, age etc.

Most probably source would be any tables or collections in the database. We need to pull those emails and email content.

Step one can be either trigger by the user or can run a cron job. Cron can look for a specific table (say outbox table) and look for entries which are not processed based on some status field.

The most important job in this step is a formation of email. We need to create a message in form of JSON which looks similar to this.

Note: JSON is for just an example purpose.

Once the message is ready, final job of the step is to push it to event/message queue on specific topic or queue.

Recap:

• Extract list of emails from the database.
• Extract email content.
• Replace the fields of email content which matches the value of the field such as name etc.
• Form the message.
• Push it to the queue.

Step 2 is the message queue and there is nothing much we can do on that part except some configuration which is not in the scope of this article.

Step 3

This is the interesting part where we need to consume those messages from the queue and send an email. Here comes the interesting part, we can attach multiple listeners to the queue to increase the message consumption process.

For example. If we add 4 consumers or listeners to the queue and they are configured to receive 100 messages at a time.

So 4 consumers and 100 messages at a time i.e 400 messages at a time. We can increase the count to meet needs.

Step 4

This is the part where we will send an email. Consumers will invoke the SMTP Server to send the email. Most of the SMTP server’s has throttle rate limit say 100 emails per second etc so all that needs to be configured.

So this is what I propose to design email queue. Let me know your thoughts.