Banner image

Server Side Pagination Using Node and Mongo

In a real-world scenario, most of the time we have to deal with a huge number of data. So imagine a scenario where you have 1 million records in the database and your requirement is to show it on a web page using an API.

You just can’t pick up all records and show it, it’s just not possible and feasible. To handle such cases, we use pagination i.e showing record page by page, just like the book.

There are two ways you can implement pagination.

  1. Client Side Pagination
  2. Server Side Pagination

Client side pagination can work if you are sure that record set is static i.e not growing day by day which is I believe very rare nowadays. In this approach, we load the data in the browser and paginate it. It is fast and easy but not reliable to hold large volume (Because browser crash you know).

For a large volume of data, we use Server side pagination.

Today, we are going to learn Server Side pagination and we are storing data in MongoDB and paginating it using API’s built-in Nodejs.

Prerequisites

Make sure you have Nodejs version 6+ and MongoDB latest installed. You can refer Node and MongoDB official site for the installation instruction.

And, we need some good amount of data. If you don’t have the data, let us generate some mock one.

Generate mock data

Mockaroo is a website to generate mock data for free (1000 record at once). I am going to use that to generate 1000 records of user.

Reference: Mockaroo Site

Select the fields you want in your record and download the CSV file. Here is what I have selected.

Ok, download the file. Now, let’s dump those records in our MongoDB database.

Dumping data in MongoDB

MongoDB comes with the command line program called Mongo Import to import huge amount of data in the MongoDB collection. We are going to use the same command line tool to import our mock data.

Here is the command.

mongoimport -d demo -c users --type csv MOCK_DATA.csv --headerline

Note: the demo is the database name and users is collection name. Change it to your need.

Once the data is imported, you can check that in Mongo shell using the following command.

use demo;
db.users.find({});

It should return some data which we just created.

Ok, time to build a pagination API in Node.js.

Writing Server code for pagination

Before we jump into the code, let’s understand the basic math behind the pagination.

There are two parameters involved in pagination.

  1. Size – How many records per page
  2. Page No – the number of the page

So the math is like this.

Suppose you have 1000 documents and you want 10 documents per page i.e size of the page is 10.

So a number of pages are 1000/10 = 10.

The page number can either begin with 0 or 1 depending on your choice.

So our API has to support these two parameters.

Let’s go ahead and code this.

Start a new Node project.

npm init --y

Install the dependencies.

npm i --S express mongoose

Now, let’s code the MongoDB database connection and model code.

mongo.js
var mongoose = require("mongoose");
mongoose.connect('mongodb://localhost:27017/demo');
// create instance of Schema
var mongoSchema =   mongoose.Schema;
// create schema
var userSchema  = {
    "first_name" : String,
    "last_name" : String,
    "email": String,
    "gender": String,
    "ip_address": String,
};
// create model if not exists.
module.exports = mongoose.model('users',userSchema);

Let’s create our Server code. In MongoDB find query, there is a parameter to skip the record and limit it without doing a manually filtering i.e MongoDB itself looks for the records based on this for fast performance.

We will skip the records based on the page number and limit it based on the size. So page size will be the number of records we want i.e limit in MongoDB term and a number of records to skip would be page size multiplied by page number.

Example:

User request for page number 1 with size 10

So, in the query, we will skip no records and get first 10 records because of its page number one.

Math would be like this.

Skip = size * (page number -1)
Limit = size

Making sure that page number never exceeds 0. Here is the code.

app.js
const express = require('express')
const mongoOp = require("./mongo")
const app = express()
const router = express.Router()

router.get('/users',(req,res) => {
  var pageNo = parseInt(req.query.pageNo)
  var size = parseInt(req.query.size)
  var query = {}
  if(pageNo < 0 || pageNo === 0) {
        response = {"error" : true,"message" : "invalid page number, should start with 1"};
        return res.json(response)
  }
  query.skip = size * (pageNo - 1)
  query.limit = size
  // Find some documents
       mongoOp.find({},{},query,function(err,data) {
        // Mongo command to fetch all data from collection.
            if(err) {
                response = {"error" : true,"message" : "Error fetching data"};
            } else {
                response = {"error" : false,"message" : data};
            }
            res.json(response);
        });
})

app.use('/api',router)
app.listen(3000)

We are fetching all the records with the query limit and skip.

Let’s run the code and check it out.

Running the app

Run the code using the following command.

node app.js

From the browser, hit this URL.

http://localhost:3000/api/users?pageNo=1&size=10

Pagination API response

Try playing with the size and page no to get more data.

Further development

We need to also extract the total number of pages based on the size provided by the user. So for example, if user wants 10 records a time for 1000 records in total, then we need to tell the API consumer that you can traverse complete record in 10 pages.

We can do this by getting the count of records and then dividing it by the size asked by the user.

Here are the code changes we need to make.

app.js
const express = require('express')
const mongoOp = require("./mongo")
const app = express()
const router = express.Router()

router.get('/users',(req,res) => {
  var pageNo = parseInt(req.query.pageNo)
  var size = parseInt(req.query.size)
  var query = {}
  if(pageNo < 0 || pageNo === 0) {
        response = {"error" : true,"message" : "invalid page number, should start with 1"};
        return res.json(response)
  }
  query.skip = size * (pageNo - 1)
  query.limit = size
  // Find some documents
       mongoOp.count({},function(err,totalCount) {
             if(err) {
               response = {"error" : true,"message" : "Error fetching data"}
             }
         mongoOp.find({},{},query,function(err,data) {
              // Mongo command to fetch all data from collection.
            if(err) {
                response = {"error" : true,"message" : "Error fetching data"};
            } else {
                var totalPages = Math.ceil(totalCount / size)
                response = {"error" : false,"message" : data,"pages": totalPages};
            }
            res.json(response);
         });
       })
})

app.use('/api',router)
app.listen(3000)

Now when you hit the API with you will get the page numbers in response.

Conclusion

We have learned how to do the pagination using MongoDB, you can, however, replace it with any other database. I highly recommend you to perform the Server side pagination as it is easy to scale and fix.


Enjoyed the article? Help us grow and produce more awesome content like this by going pro!
Become a pro!

Shahid (UnixRoot) Shaikh

Hey there, This is Shahid, an Engineer and Blogger from Bombay. I am also an Author and i wrote a programming book on Sails.js, MVC framework for Node.js.

Related Posts

9 Comments

  1. That was a Wonderful Article Thank You Shahid I was Hoping that can make a blog related to MERN Stack looking Forward to it 🙂

  2. I think i am getting problem in running the application.
    Kindly tell where and how to enter the command “node app.js”
    should i first navigate to the folder where i saved both node.js and app.js and there i should run this command or in Node.js shell?

    1. You should specify the path of the code you wanna run. If you switch to the project folder where app.js is placed then running node app.js would work!

  3. I want to draw your attention to statement (copied verbatim from above):
    “Making sure that page number never exceeds 0.”

    Shouldn’t it be: “Making sure that page number never <= 0" ?

  4. I am trying to implement this at my end but it is not working.
    I am not getting any console error i did some debugging and found in app.js file mongoOp.find is not getting called.
    I have tried everthing but im not able to find the root cause. need your help on this.
    my mongodb version is 2.4.6.

Leave a Reply

Your email address will not be published. Required fields are marked *