Hire the author: Mcdavid E
Image Source: https://commons.wikimedia.org/wiki/File:Direct_model.png
Introduction
APIs request that deals with email services can take a while to process, hence putting your server performance into questions. One way of solving this is by using a message queue service like RabbitMQ.
Message Queue is a form of asynchronous service-to-service communication used in server-less and micro-services architectures. Messages are stored in the queue until they are processed and deleted. Each message is processed only once, by a single consumer. Message queues act as a middleman for various services (e.g. a web application, as in this example). They can be used to reduce loads and delivery times of web application servers by delegating tasks that would normally take up a lot of time or resources to a third party that has no other job.
Glossary
- Message Queue – A message queue is a queue of messages sent between applications. It includes a sequence of work objects that are waiting to be processed.
- Worker – A worker is something you give a task and continue in your process, while the worker (or multiple workers) process the task on a different thread.
- Process – A process is the instance of a computer program that is being executed by one or many threads. It contains the program code and its activity.
What we’ll be doing.
In this article, we’ll be using RabbitMQ a message queuing software to build an email service with AWS SES. We would also be using PM2 to manage our processes. You can find the full project on Github
Prerequisites.
You need to have a basic understanding of the following to follow along:
- JavaScript server-side with NodeJs and Express
- Have RabbitMQ installed on your local machine
- Process Management with PM2
Step 1 – Install our dependencies
First, we initialize our npm app and install the dependencies needed to make our service function. Then, run the command below on your terminal to install the packages from NPM
npm init -y && npm i -S express dotenv amqplib make-runnable aws-sdk
Above we installed four npm packages we’d be needing to make our application run. First is expressjs which we would be using to run and set up our NodeJS server. Then, we install make-runnable
which helps us run our javascript modules as executables on the terminal. Lastly, we have amqplib
which we would use to connect to our RabbitMQ server.
Step 2 – Create our Express server
We’ll begin by creating our express server. The server would be in our index.js
file under our src
directory in the root folder.
Paste the code below into the index.js
file.
const express = require('express'); | |
const { json, urlencoded } = express; | |
const app = express(); | |
app.use(json()); | |
app.use(urlencoded({ extended: true })); | |
app.get('/', (req, res) => { | |
return res.status(200).send({ | |
message: 'Welcome to our API' | |
}) | |
}); | |
app.listen(5000, () => { | |
console.log(`app running on port: 5000`); | |
}); |
./src/index.js
Navigate to your project root directory and start your server with node src/index.js
. You should get the message below logged to your terminal
app running on port: 5000
Step 3 – Setting up AWS SES.
We’ll be making use of Amazon SES to send our emails. To set it up you’ll need to create an Amazon AWS account and register for a free tier plan.
- Follow the steps here to set up your Amazon SES platform.
- Get your Access Key Id and Secret Key attached to your AWS account. For help getting your Access Key Id and Secret Key follow this link
- Create a
.env
file in your project root directory and add your AWS credentials as see in the code snippet below.
AWS_ACCESS_KEY_ID='YOUR_AWS_ID' | |
AWS_SECRET_ACCESS_KEY='YOUR_AWS_SECRET_KEY' |
- Next, we create a config file to house our AWS configurations, this is the first time we make use of the
aws-sdk
we installed. Then, create anawsConfig.js
file in the./src
directory and add the code below to it.
const AWS = require('aws-sdk'); | |
const dotenv = require('dotenv'); | |
dotenv.config(); | |
AWS.config.update({ | |
accessKeyId: process.env.AWS_ACCESS_KEY_ID, | |
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY | |
}); | |
module.exports = { | |
key: process.env.AWS_ACCESS_KEY_ID, | |
secret: process.env.AWS_SECRET_ACCESS_KEY, | |
ses: { | |
from: { | |
// replace with actual email address | |
default: '"Company Admin" <admin@company-name.co>', | |
}, | |
// e.g. us-west-2 | |
region: 'us-east-1' | |
} | |
}; |
./src/awsConfig.js
In the config above we use the aws-sdk to set up our config to connect to aws. This is where we reference the Access Key Id and Secret Key from our AWS console.
Step 4 – Creating our Email Service to Connect to AWS
In our email service, we set up the necessary configurations needed to send our mail via Amazon SES. So, create an email.service.js
file and add the code below.
const AWS = require('aws-sdk'); | |
const awsConfig = require('./awsConfig'); | |
AWS.config.update({ | |
accessKeyId: awsConfig.key, | |
secretAccessKey: awsConfig.secret, | |
region: awsConfig.ses.region | |
}); | |
const ses = new AWS.SES({ apiVersion: '2010-12-01' }); | |
module.exports = { | |
/** | |
* @method sendMail | |
* @param {Array} to array of mails to send content to | |
* @param {String} subject Subject of mail to be sent | |
* @param {String} message content of message in html template format | |
* @param {String} from not required: mail to send email from | |
* @returns {Promise} Promise | |
*/ | |
sendMail(to, subject, message, from) { | |
const params = { | |
Destination: { | |
ToAddresses: to | |
}, | |
Message: { | |
Body: { | |
Html: { | |
Charset: 'UTF-8', | |
Data: message | |
}, | |
/* replace Html attribute with the following if you want to send plain text emails. | |
Text: { | |
Charset: "UTF-8", | |
Data: message | |
} | |
*/ | |
}, | |
Subject: { | |
Charset: 'UTF-8', | |
Data: subject | |
} | |
}, | |
ReturnPath: from || awsConfig.ses.from.default, | |
Source: from || awsConfig.ses.from.default | |
}; | |
return new Promise((resolve, reject) => { | |
ses.sendEmail(params, (err, data) => { | |
if (err) { | |
reject(err, err.stack); | |
} else { | |
resolve(data); | |
} | |
}); | |
}); | |
} | |
}; |
./src/email.service.js
Step 5 – Creating our email worker with RabbitMQ
Next, we create an email worker. We’ll be making use of the amqplib package to connect to our RabbitMQ server. If you haven’t yet, follow this link to install RabbitMQ on your machine, or you can set up a free RabbitMQ cloud instance on https://www.cloudamqp.com/ and add your URL to your .env
file.
const dotenv = require('dotenv'); | |
const EmailService = require('./email.service'); | |
dotenv.config(); | |
const queue = 'email-task'; | |
const open = require('amqplib').connect(process.env.AMQP_SERVER); | |
// Publisher | |
const publishMessage = payload => open.then(connection => connection.createChannel()) | |
.then(channel => channel.assertQueue(queue) | |
.then(() => channel.sendToQueue(queue, Buffer.from(JSON.stringify(payload))))) | |
.catch(error => console.warn(error)); | |
// Consumer | |
const consumeMessage = () => { | |
open.then(connection => connection.createChannel()).then(channel => channel.assertQueue(queue).then(() => { | |
console.log(' [*] Waiting for messages in %s. To exit press CTRL+C', queue); | |
return channel.consume(queue, (msg) => { | |
if (msg !== null) { | |
const { mail, subject, template } = JSON.parse(msg.content.toString()); | |
console.log(' [x] Received %s', mail); | |
// send email via aws ses | |
EmailService.sendMail(mail, subject, template).then(() => { | |
channel.ack(msg); | |
}); | |
} | |
}); | |
})).catch(error => console.warn(error)); | |
}; | |
module.exports = { | |
publishMessage, | |
consumeMessage | |
} | |
require('make-runnable'); |
./src/emailWorker.js
In the email worker file, we connected to our RabbitMQ server using the amqplib package and we add the URL to our RabbitMQ instance. Then we created two functions that help run our message queue.
The publishMessage function is responsible for adding our emails to the queue using the sendToQueue method to set up a task queue. It adds our message to the email-task queue.
Then, consumeMessage is the function that starts up our worker in the background to listen for incoming messages added to our email-task queue.
We import the EmailService module in this file and call the consumeMessage function.
When we add a new mail to the email-task queue, the RabbitMQ server takes responsibility for calling the sendmail method, taking the load time from your API server.
Next, we create our endpoint to send emails. In the index.js file add another endpoint, this time a post request to send emails via our service. Add the code below under the app.get
function.
//const express = require('express'); | |
// import publicMessage from Email service | |
const { publishMessage } = require('./emailWorker') | |
// ......................... | |
// ...app.get | |
/** | |
* @post sensend email | |
*/ | |
app.post('/email', (req, res) => { | |
const { body: { email } } = req; | |
const emailOptions = { | |
mail: [email], | |
subject: 'Email confirmed', | |
template: ` | |
<body> | |
<p>Hi,</p> | |
<p>Thanks for your submission, your email address has been recorder successfully</p> | |
</body> | |
` | |
} | |
// call rabbitmq service to app mail to queue | |
publishMessage(emailOptions); | |
return res.status(202).send({ | |
message: 'Email sent successfully' | |
}) | |
}) | |
// app.listen |
./src/index.js
A post request is created for us to send our emails, in the callback we are calling the publishMessage method from our emailWorker
file and passing in the object containing our email parameters. This is the point where we add our email to the email-task queue.
Step 6 – Running our App
To run our app we will need to open two instances of our terminal. The first terminal would run our node express server, then second would run our email worker process.
node src/index.js

Then, we start the email worker running on RabbitMQ by running the command below in the second terminal.
node src/emailWorker.js consumeMessage

Let’s test the email endpoint to see if it works. Open any HTTP client of your choice and make a POST request to localhost:5000/email
in the body pass in the email address as shown in the image below.

Note: Make sure to use the email you verified on your Amazon SES settings to send a request if your account is still in sandbox mode. Follow this link for more information
Step 7 – Setting up PM2.
In a production environment, it won’t be easy for us to run both our server process and email worker at the same time. That is why we need a process manager.
PM2 is a production process manager for Node.js applications with a built-in load balancer and graphical dashboard on the terminal which helps you monitor all apps running on it. It allows you to keep applications alive forever, therefore reloading them without any downtime and to facilitate common system admin tasks. You can read more about PM2 on the documentation.
To use PM2 we would need to install it globally with npm. Then, run the following command on your terminal to install PM2.
npm install -g pm2
After installing PM2, we need to set up our ecosystem config file. This is the file where we declare all our processes to be run by PM2. So, on the root directory of the project create an ecosystem.config.js file and add the code below to it.
module.exports = { | |
apps: [ | |
{ | |
name: 'API', | |
script: 'src/index.js', | |
exec_mode: 'cluster_mode', | |
instances: 'max', | |
env: { | |
NODE_ENV: 'production' | |
} | |
}, | |
{ | |
name: 'emailWorker', | |
args: 'consumeMessage', | |
exec_mode: 'fork', | |
watch: false, | |
script: 'src/emailWorker.js', | |
instances: '1' | |
} | |
] | |
}; |
./ecosystem.config.js
In the config file above we are declaring two processes the first is our API which is our express server, while the second is our email worker which runs on RabbitMQ.
With PM2 you can run your applications in cluster mode, this means running multiple instances of your application based on the number of CPUs available on the machine, thereby auto load balancing your application and preventing downtime. Our API is set to run in cluster mode and set to use the maximum number of available CPUs.
To start our application we only need one command. Run the command below on your terminal to start the app.
pm2 start ecosystem.config.js
We should see the pm2 table logged in the terminal like this:

Lastly, we test our API on our HTTP client like earlier to make sure it works.
Learning Tools
There is a lot that we can gain performance-wise from using queues and different applications that it can fit in. So, to learn more, find below useful links to learn more about RabbitMQ, Amazon SES and PM2.
- RabbitMQ Tutorials – https://www.rabbitmq.com/getstarted.html
- PM2 Docs – https://pm2.keymetrics.io/docs/usage/pm2-doc-single-page/
- Amazon SES – https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/ses-examples.html
Learning Strategy
To get the most out of this project, I had to have basic knowledge of building an API server. Apart from using the learning materials available on the internet, I also had to think of a real-world scenario to apply what I was learning too, in this case, an email service.
Reflective Analysis
The benefits that come from using a message queue cannot be overly emphasized. Working with queues in the past few months has helped me see the importance of decoupling your app into smaller units. Therefore, making it scale each one independently. With this, you get faster load time on our web applications.
Conclusion
There is much more to gain from RabbitMQ apart from just queuing, you can read more on all it’s available features here. Also, if you want to delve into advanced use cases like filtering emails by source and dividing them into channels you can pick from where we stopped in this tutorial. Here is the link to the Github repositiory to get started.
Lastly here is another fun article on confirming emails with Django/GraphQL API. How do you think you can implement a queuing service with this? Give it a shot. You could share links to your implementation in the comment section.
Very brilliant and quite useful tutorial. This can be used by learners and companies looking to send emails to customers or users. Future articles can show how to send to multiple emails.
Thanks for the feedback Jerry. As for the multiple emails you can to that with this same implementation. If you check the
app.post
method in the the index.js file the mail key in the emailOptions object is expecting an array of emails, so you can pass in more than one email address to it.Awesome tutorial!
Really useful tutorial – got me up and running super quickly.
I did wonder why the SES apiVersion was set to ‘2010-12-01’? Seemed odd to use a version from 10 years ago.
I used it without this setting and it worked fine
Thanks for writing this 🙂
Very Useful and transferable tutorial.
Thanks