MongoDB and ElasticSearch Synchronization

MongoDB and ElasticSearch is an amazing pair to develop a NoSQL application with a powerful search. To develop a polyglot database application, we need to perform the data synchronization. In this article, we will learn how to synchronize the data from MongoDB to ElasticSearch on an on-going basis.

Polyglot persistence is the concept of using different data storage technologies to handle different data storage needs within a given software application. – Wikipedia

MongoDB uses the database and collection to store the data. ElasticSearch uses the index and type to store and map the data for effective search.

Let’s learn how we can perform the synchronization.

Basics

We are assuming that MongoDB is our primary database and ElasticSearch is our secondary database. This means that we will add/delete/update the data in MongoDB first and then ElasticSearch.

MongoDB database will be mapped to the ElasticSearch index. MongoDB collection will be mapped to ElasticSearch type.

Getting Started

You need to install MongoDB and ElasticSearch in your system before proceeding ahead.

Link to download MongoDB
Link to download ElasticSearch

Run both the database engines after installation.

Performing Synchronization

We are using mongoosastic node module to perform the synchronization. Let’s create a Node project and code basic synchronization.

npm init --y

Install dependencies.

npm install --save mongoose mongoosastic

Here is the code to perform the synchronization.

const mongoose = require('mongoose');
const mongoosastic = require('mongoosastic');

mongoose.connect('mongodb://localhost:27017/mongosync');

var UserSchema = new mongoose.Schema({
name: String
, email: String
, city: String
});

UserSchema.plugin(mongoosastic, {
"host": "localhost",
"port": 9200
});

var User = mongoose.model('user', UserSchema);

User.createMapping((err, mapping) => {
console.log('mapping created');
});

var newUser = new User({
name: 'Shahid',
email: '[email protected]',
city: 'mumbai'
});

newUser.save((err) => {
if(err) {
console.log(err);
}
console.log('user added in both the databases');
})

newUser.on('es-indexed', (err, result) => {
console.log('indexed to elastic search');
});

Let me explain.

First, we are connecting to MongoDB engine.

mongoose.connect('mongodb://localhost:27017/mongosync');

Then, we are creating a MongoDB schema. We are using simple fields of users.

var UserSchema = new mongoose.Schema({
name: String
, email: String
, city: String
});

Then, we are connecting to ElasticSearch using the mongoosastic plugin.

UserSchema.plugin(mongoosastic, {
"host": "localhost",
"port": 9200
});

Then, we are creating a MongoDB model and ElasticSearch mapping.

var User = mongoose.model('user', UserSchema);

User.createMapping((err, mapping) => {
console.log('mapping created');
});

Awesome. Now, we can add data in MongoDB and it should be synced to ElasticSearch.

Run this program and you should receive a message on your terminal similar to this.

Check data entry in MongoDB.

Check data in ElasticSearch. Open localhost:9200/users/_search to view the results.

And, we have data in both the databases.

Possible use cases

One of the good use cases is an E-commerce based application. You can use MongoDB to store and retrieve the product details while ElasticSearch can act as a search engine. You can use this technique to perform synchronization.

Conclusion

MongoDB and ElasticSearch can be used to develop very rich user-centric applications. In terms of the synchronization of the data, this technique can be used. We can also use a queue based system to perform the synchronization.