Hire the author: Mcdavid E

Image Source: https://commons.wikimedia.org/wiki/File:Direct_model.png

Introduction

APIs request that deals with email services can take a while to process, hence putting your server performance into questions. One way of solving this is by using a message queue service like RabbitMQ.

Message Queue is a form of asynchronous service-to-service communication used in server-less and micro-services architectures. Messages are stored in the queue until they are processed and deleted. Each message is processed only once, by a single consumer. Message queues act as a middleman for various services (e.g. a web application, as in this example). They can be used to reduce loads and delivery times of web application servers by delegating tasks that would normally take up a lot of time or resources to a third party that has no other job.

Glossary

  • Message Queue – A message queue is a queue of messages sent between applications. It includes a sequence of work objects that are waiting to be processed.
  • Worker – A worker is something you give a task and continue in your process, while the worker (or multiple workers) process the task on a different thread.
  • Process – A process is the instance of a computer program that is being executed by one or many threads. It contains the program code and its activity.

What we’ll be doing.

In this article, we’ll be using RabbitMQ a message queuing software to build an email service with AWS SES. We would also be using PM2 to manage our processes. You can find the full project on Github

Prerequisites.

You need to have a basic understanding of the following to follow along:

  • JavaScript server-side with NodeJs and Express
  • Have RabbitMQ installed on your local machine
  • Process Management with PM2

Step 1 – Install our dependencies

First, we initialize our npm app and install the dependencies needed to make our service function. Then, run the command below on your terminal to install the packages from NPM

npm init -y && npm i -S express dotenv amqplib make-runnable aws-sdk

Above we installed four npm packages we’d be needing to make our application run. First is expressjs which we would be using to run and set up our NodeJS server. Then, we install make-runnable which helps us run our javascript modules as executables on the terminal. Lastly, we have amqplib which we would use to connect to our RabbitMQ server.

Step 2 – Create our Express server

We’ll begin by creating our express server. The server would be in our index.js file under our src directory in the root folder.

Paste the code below into the index.js file.

const express = require('express');
const { json, urlencoded } = express;
const app = express();
app.use(json());
app.use(urlencoded({ extended: true }));
app.get('/', (req, res) => {
return res.status(200).send({
message: 'Welcome to our API'
})
});
app.listen(5000, () => {
console.log(`app running on port: 5000`);
});
view raw index-0.js hosted with ❤ by GitHub

./src/index.js

Navigate to your project root directory and start your server with node src/index.js. You should get the message below logged to your terminal

app running on port: 5000

Step 3 – Setting up AWS SES.

We’ll be making use of Amazon SES to send our emails. To set it up you’ll need to create an Amazon AWS account and register for a free tier plan.

  • Follow the steps here to set up your Amazon SES platform.
  • Get your Access Key Id and Secret Key attached to your AWS account. For help getting your Access Key Id and Secret Key follow this link
  • Create a .env file in your project root directory and add your AWS credentials as see in the code snippet below.
AWS_ACCESS_KEY_ID='YOUR_AWS_ID'
AWS_SECRET_ACCESS_KEY='YOUR_AWS_SECRET_KEY'
view raw .env hosted with ❤ by GitHub
  • Next, we create a config file to house our AWS configurations, this is the first time we make use of the aws-sdk we installed. Then, create an awsConfig.js file in the./src directory and add the code below to it.
const AWS = require('aws-sdk');
const dotenv = require('dotenv');
dotenv.config();
AWS.config.update({
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
});
module.exports = {
key: process.env.AWS_ACCESS_KEY_ID,
secret: process.env.AWS_SECRET_ACCESS_KEY,
ses: {
from: {
// replace with actual email address
default: '"Company Admin" <admin@company-name.co>',
},
// e.g. us-west-2
region: 'us-east-1'
}
};
view raw awsConfig.js hosted with ❤ by GitHub

./src/awsConfig.js

In the config above we use the aws-sdk to set up our config to connect to aws. This is where we reference the Access Key Id and Secret Key from our AWS console.

Step 4 – Creating our Email Service to Connect to AWS

In our email service, we set up the necessary configurations needed to send our mail via Amazon SES. So, create an email.service.js file and add the code below.

const AWS = require('aws-sdk');
const awsConfig = require('./awsConfig');
AWS.config.update({
accessKeyId: awsConfig.key,
secretAccessKey: awsConfig.secret,
region: awsConfig.ses.region
});
const ses = new AWS.SES({ apiVersion: '2010-12-01' });
module.exports = {
/**
* @method sendMail
* @param {Array} to array of mails to send content to
* @param {String} subject Subject of mail to be sent
* @param {String} message content of message in html template format
* @param {String} from not required: mail to send email from
* @returns {Promise} Promise
*/
sendMail(to, subject, message, from) {
const params = {
Destination: {
ToAddresses: to
},
Message: {
Body: {
Html: {
Charset: 'UTF-8',
Data: message
},
/* replace Html attribute with the following if you want to send plain text emails.
Text: {
Charset: "UTF-8",
Data: message
}
*/
},
Subject: {
Charset: 'UTF-8',
Data: subject
}
},
ReturnPath: from || awsConfig.ses.from.default,
Source: from || awsConfig.ses.from.default
};
return new Promise((resolve, reject) => {
ses.sendEmail(params, (err, data) => {
if (err) {
reject(err, err.stack);
} else {
resolve(data);
}
});
});
}
};
view raw email.service.js hosted with ❤ by GitHub

./src/email.service.js

Step 5 – Creating our email worker with RabbitMQ

Next, we create an email worker. We’ll be making use of the amqplib package to connect to our RabbitMQ server. If you haven’t yet, follow this link to install RabbitMQ on your machine, or you can set up a free RabbitMQ cloud instance on https://www.cloudamqp.com/ and add your URL to your .env file.

const dotenv = require('dotenv');
const EmailService = require('./email.service');
dotenv.config();
const queue = 'email-task';
const open = require('amqplib').connect(process.env.AMQP_SERVER);
// Publisher
const publishMessage = payload => open.then(connection => connection.createChannel())
.then(channel => channel.assertQueue(queue)
.then(() => channel.sendToQueue(queue, Buffer.from(JSON.stringify(payload)))))
.catch(error => console.warn(error));
// Consumer
const consumeMessage = () => {
open.then(connection => connection.createChannel()).then(channel => channel.assertQueue(queue).then(() => {
console.log(' [*] Waiting for messages in %s. To exit press CTRL+C', queue);
return channel.consume(queue, (msg) => {
if (msg !== null) {
const { mail, subject, template } = JSON.parse(msg.content.toString());
console.log(' [x] Received %s', mail);
// send email via aws ses
EmailService.sendMail(mail, subject, template).then(() => {
channel.ack(msg);
});
}
});
})).catch(error => console.warn(error));
};
module.exports = {
publishMessage,
consumeMessage
}
require('make-runnable');
view raw emailWorker.js hosted with ❤ by GitHub

./src/emailWorker.js

In the email worker file, we connected to our RabbitMQ server using the amqplib package and we add the URL to our RabbitMQ instance. Then we created two functions that help run our message queue.

The publishMessage function is responsible for adding our emails to the queue using the sendToQueue method to set up a task queue. It adds our message to the email-task queue.

Then, consumeMessage is the function that starts up our worker in the background to listen for incoming messages added to our email-task queue.

We import the EmailService module in this file and call the consumeMessage function.

When we add a new mail to the email-task queue, the RabbitMQ server takes responsibility for calling the sendmail method, taking the load time from your API server.

Next, we create our endpoint to send emails. In the index.js file add another endpoint, this time a post request to send emails via our service. Add the code below under the app.get function.

//const express = require('express');
// import publicMessage from Email service
const { publishMessage } = require('./emailWorker')
// .........................
// ...app.get
/**
* @post sensend email
*/
app.post('/email', (req, res) => {
const { body: { email } } = req;
const emailOptions = {
mail: [email],
subject: 'Email confirmed',
template: `
<body>
<p>Hi,</p>
<p>Thanks for your submission, your email address has been recorder successfully</p>
</body>
`
}
// call rabbitmq service to app mail to queue
publishMessage(emailOptions);
return res.status(202).send({
message: 'Email sent successfully'
})
})
// app.listen
view raw index-1.js hosted with ❤ by GitHub

./src/index.js

A post request is created for us to send our emails, in the callback we are calling the publishMessage method from our emailWorker file and passing in the object containing our email parameters. This is the point where we add our email to the email-task queue.

Step 6 – Running our App

To run our app we will need to open two instances of our terminal. The first terminal would run our node express server, then second would run our email worker process.

node src/index.js
express server output
start express server

Then, we start the email worker running on RabbitMQ by running the command below in the second terminal.

node src/emailWorker.js consumeMessage
email worker queue output rabbitmq
start email worker

Let’s test the email endpoint to see if it works. Open any HTTP client of your choice and make a POST request to localhost:5000/email in the body pass in the email address as shown in the image below.

rabbitmq postman aws ses

Note: Make sure to use the email you verified on your Amazon SES settings to send a request if your account is still in sandbox mode. Follow this link for more information

Step 7 – Setting up PM2.

In a production environment, it won’t be easy for us to run both our server process and email worker at the same time. That is why we need a process manager.

PM2 is a production process manager for Node.js applications with a built-in load balancer and graphical dashboard on the terminal which helps you monitor all apps running on it. It allows you to keep applications alive forever, therefore reloading them without any downtime and to facilitate common system admin tasks. You can read more about PM2 on the documentation.

To use PM2 we would need to install it globally with npm. Then, run the following command on your terminal to install PM2.

npm install -g pm2

After installing PM2, we need to set up our ecosystem config file. This is the file where we declare all our processes to be run by PM2. So, on the root directory of the project create an ecosystem.config.js file and add the code below to it.

module.exports = {
apps: [
{
name: 'API',
script: 'src/index.js',
exec_mode: 'cluster_mode',
instances: 'max',
env: {
NODE_ENV: 'production'
}
},
{
name: 'emailWorker',
args: 'consumeMessage',
exec_mode: 'fork',
watch: false,
script: 'src/emailWorker.js',
instances: '1'
}
]
};
view raw ecosystem.config.js hosted with ❤ by GitHub

./ecosystem.config.js

In the config file above we are declaring two processes the first is our API which is our express server, while the second is our email worker which runs on RabbitMQ.

With PM2 you can run your applications in cluster mode, this means running multiple instances of your application based on the number of CPUs available on the machine, thereby auto load balancing your application and preventing downtime. Our API is set to run in cluster mode and set to use the maximum number of available CPUs.

To start our application we only need one command. Run the command below on your terminal to start the app.

pm2 start ecosystem.config.js

We should see the pm2 table logged in the terminal like this:

pm2 queue service output
PM2 table in the terminal

Lastly, we test our API on our HTTP client like earlier to make sure it works.

Learning Tools

There is a lot that we can gain performance-wise from using queues and different applications that it can fit in. So, to learn more, find below useful links to learn more about RabbitMQ, Amazon SES and PM2.

Learning Strategy

To get the most out of this project, I had to have basic knowledge of building an API server. Apart from using the learning materials available on the internet, I also had to think of a real-world scenario to apply what I was learning too, in this case, an email service.

Reflective Analysis

The benefits that come from using a message queue cannot be overly emphasized. Working with queues in the past few months has helped me see the importance of decoupling your app into smaller units. Therefore, making it scale each one independently. With this, you get faster load time on our web applications.

Conclusion

There is much more to gain from RabbitMQ apart from just queuing, you can read more on all it’s available features here. Also, if you want to delve into advanced use cases like filtering emails by source and dividing them into channels you can pick from where we stopped in this tutorial. Here is the link to the Github repositiory to get started.

Lastly here is another fun article on confirming emails with Django/GraphQL API. How do you think you can implement a queuing service with this? Give it a shot. You could share links to your implementation in the comment section.

Hire the author: Mcdavid E