In a real-world scenario, most of the time we have to deal with a huge number of data. So imagine a scenario where you have 1 million records in the database and your requirement is to show it on a web page using an API.
You just can’t pick up all records and show it, it’s just not possible and feasible. To handle such cases, we use pagination i.e showing record page by page, just like the book.
There are two ways you can implement pagination.
- Client Side Pagination
- Server Side Pagination
Client side pagination can work if you are sure that record set is static i.e not growing day by day which is I believe very rare nowadays. In this approach, we load the data in the browser and paginate it. It is fast and easy but not reliable to hold large volume (Because of browser crash you know).
For a large volume of data, we use Server side pagination.
Today, we are going to learn Server Side pagination and we are storing data in MongoDB and paginating it using API’s built-in Nodejs.
Prerequisites
Make sure you have Nodejs version 6+ and MongoDB latest installed. You can refer Node and MongoDB official site for the installation instruction.
And, we need some good amount of data. If you don’t have the data, let us generate some mock one.
Generate mock data
Mockaroo is a website to generate mock data for free (1000 record at once). I am going to use that to generate 1000 records of user.
Select the fields you want in your record and download the CSV file. Here is what I have selected.
Ok, download the file. Now, let’s dump those records in our MongoDB database.
Dumping data in MongoDB
MongoDB comes with the command line program called Mongo Import to import a huge amount of data in the MongoDB collection. We are going to use the same command line tool to import our mock data.
Here is the command.
Note: the demo is the database name and users is collection name. Change it to your need.
Once the data is imported, you can check that in Mongo shell using the following command.
db.users.find({});
It should return some data which we just created.
Ok, time to build a pagination API in Node.js.
Writing Server code for pagination
Before we jump into the code, let’s understand the basic math behind the pagination.
There are two parameters involved in pagination.
- Size – How many records per page
- Page No – the number of the page
So the math is like this.
Suppose you have 1000 documents and you want 10 documents per page i.e size of the page is 10.
So a number of pages are 1000/10 = 10.
The page number can either begin with 0 or 1 depending on your choice.
So our API has to support these two parameters.
Let’s go ahead and code this.
Start a new Node project.
Install the dependencies.
Now, let’s code the MongoDB database connection and model code.
mongoose.connect('mongodb://localhost:27017/demo');
// create instance of Schema
var mongoSchema = mongoose.Schema;
// create schema
var userSchema = {
"first_name" : String,
"last_name" : String,
"email": String,
"gender": String,
"ip_address": String,
};
// create model if not exists.
module.exports = mongoose.model('users',userSchema);
Let’s create our Server code. In MongoDB find query, there is a parameter to skip the record and limit it without doing a manually filtering i.e MongoDB itself looks for the records based on this for fast performance.
We will skip the records based on the page number and limit it based on the size. So the page size will be the number of records we want i.e limit in MongoDB term and a number of records to skip would be page size multiplied by page number.
Example:
User request for page number 1 with size 10
So, in the query, we will skip no records and get the first 10 records because of its page number one.
Math would be like this.
Skip = size * (page number -1)
Limit = size
Making sure that page number never exceeds 0. Here is the code.
const mongoOp = require("./mongo")
const app = express()
const router = express.Router()
router.get('/users',(req,res) => {
var pageNo = parseInt(req.query.pageNo)
var size = parseInt(req.query.size)
var query = {}
if(pageNo < 0 || pageNo === 0) {
response = {"error" : true,"message" : "invalid page number, should start with 1"};
return res.json(response)
}
query.skip = size * (pageNo - 1)
query.limit = size
// Find some documents
mongoOp.find({},{},query,function(err,data) {
// Mongo command to fetch all data from collection.
if(err) {
response = {"error" : true,"message" : "Error fetching data"};
} else {
response = {"error" : false,"message" : data};
}
res.json(response);
});
})
app.use('/api',router)
app.listen(3000)
We are fetching all the records with the query limit and skip.
Let’s run the code and check it out.
Running the app
Run the code using the following command.
From the browser, hit this URL.
http://localhost:3000/api/users?pageNo=1&size=10
Try playing with the size and page no to get more data.
Further development
We need to also extract the total number of pages based on the size provided by the user. So for example, if user wants 10 records a time for 1000 records in total, then we need to tell the API consumer that you can traverse complete record in 10 pages.
We can do this by getting the count of records and then dividing it by the size asked by the user.
Here are the code changes we need to make.
const mongoOp = require("./mongo")
const app = express()
const router = express.Router()
router.get('/users',(req,res) => {
var pageNo = parseInt(req.query.pageNo)
var size = parseInt(req.query.size)
var query = {}
if(pageNo < 0 || pageNo === 0) {
response = {"error" : true,"message" : "invalid page number, should start with 1"};
return res.json(response)
}
query.skip = size * (pageNo - 1)
query.limit = size
// Find some documents
mongoOp.count({},function(err,totalCount) {
if(err) {
response = {"error" : true,"message" : "Error fetching data"}
}
mongoOp.find({},{},query,function(err,data) {
// Mongo command to fetch all data from collection.
if(err) {
response = {"error" : true,"message" : "Error fetching data"};
} else {
var totalPages = Math.ceil(totalCount / size)
response = {"error" : false,"message" : data,"pages": totalPages};
}
res.json(response);
});
})
})
app.use('/api',router)
app.listen(3000)
Now when you hit the API with you will get the page numbers in response.
Further Study
Build a RESTful API using Node and MongoDB
Getting Started With MongoDB Atlas with Node.js
Best Programming Tutorial of the blog
Conclusion
We have learned how to do the pagination using MongoDB, you can, however, replace it with any other database. I highly recommend you to perform the Server side pagination as it is easy to scale and fix.
