Jenkins and Node

Continuous Integration and deployment with Jenkins and Node.js

Write code, test and deploy

This is one of the common development cycle of any developer.

Most of the time testing and deployment steps do not change frequently and in order to keep the developer focus on writing code we do the automation of testing and deployment.

This automation is called “continuous integration and deployment”.

In this tutorial, we will learn how to set up the continuous integration and deployment infrastructure for Node.js project.

Prerequistics

I am going to use Jenkins as the automation tool and Github as source code management system.

Jenkins is free for use but we require public server in order to integrate the Github repository.

For the server, I will be using $5 droplet on Digitalocean. In case you wish to buy it, you can use this link to get the $10 off for the first month.

Related learning : Hosting Node.js app to Digitalocean Server

How it’s going to work

Here is the complete flow of the system.

  • You made some changes in your project.
  • You push those changes in Github on master or any branch.
  • Github will notify Jenkins about the new push. ( We will configure it )
  • Jenkins will then run the commands you ask it to run.

Those commands will contain following.

  • Test script.
  • Deployment script.

Deployment script will be added to Project only and Jenkins will use that to communicate to Server and perform the push.

Basically, what you do manually such as running test command, login to Server, performing Git pull and then restarting build server, we will do all of them automatically by writing the script.

Let’s get down to work

Creating a project

You need a Node.js project hosted on Github to perform the testing and deployment. I am sure most of you already have projects at Github written in Node.

For those who does not have one, let’s create one together.

Go to github.com to create new repository. Give it proper name and initiate it with README file and choose Node as .gitignore file.

Devops nodejs

After creating a new repository, clone the project on your local system using following command.

git clone https://github.com/<username>/repo-name.git

For.eg

git clone https://github.com/codeforgeek/demoapp

Let’s create simple Express project to deliver “Hello World” message. Switch to the clone folder and run npm init -y command to generate the package.JSON file.

package.json
{
  "name": "sampleCode",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Install express using the following command.

npm i --save express

Create new file named “app.js” and put the following code in it.

app.js
var express = require('express');
var app = express();

app.get('/',function(req,res) {
  res.send("Hello World");
});

app.listen(4000);

Run the code using pm2 start app.js command and visit http://localhost:4000 from your browser to view the app.

PM2 is process manager and we recommend you to use it. Here is the article on same.

So we have a basic app ready to go. Let’s write one simple test case which perform the home page accesibility checking.

First, install supertest, mocha and should packages.

npm i --save-dev supertest should mocha

We need mocha globally installed too. Run this command to do so.

sudo npm i -g mocha

Create new folder named “test” and new file inside it named “test.js”.

mkdir test && cd test

Paste following code in it.

/test/test.js
var supertest = require("supertest");
var should = require("should");

// This agent refers to PORT where the program is running.

var server = supertest.agent("http://localhost:4000");

// UNIT test begin

describe("SAMPLE unit test",function(){

  // #1 should return home page
  it("should return home page",function(done){
    // calling home page
    server
    .get("/")
    .expect("Content-type",/text/)
    .expect(200) // THis is HTTP response
    .end(function(err,res){
      // HTTP status should be 200
      res.status.should.equal(200);
      done();
    });
  });

});

Run your node app then run the following command from another tab.

mocha

You should be able to see the console like this.

Mocha Devops

Ok, so we have our app running with the test. Let’s push it to Github.

First add all files.

git add .

Add new commit.

git commit -m "first push"

Push it.

git push

Adding Github webhook to push events to Jenkins

When any changes i.e commits are pushed to Github repository, we need a mechanism to notify that event to our Jenkins Server which we going to configure in next section.

Github allows us to add Webhook services to achieve same. Open GitHub repository, go to settings and click “Webhooks and Services” from side panel. Click on “Add Services” and search Jenkins, click on “Jenkins Github Plugin”.

Note: Don’t choose Git one.

Adding Github webhook

After that, add the webhook URL which is in the following format.

http://yourdomain.com-OR-IP/github-webhook

webhook screen

Click on Add Service and you are good.

Creating droplet and Installing Jenkins

I am using DigitalOcean as Web Hosting. You can use any Server as you like, in case you want to host your app on DigitalOcean and looking forward to buying it, you can use this link to get $10 discount on our side.

Click here to go to DigitalOcean droplet creation screen.

Continous integration for Node

Choose $5 plan and create the droplet.

After creating droplet wait for some time, DigitalOcean will send you an email containing the credentials.

Login to your droplet using the following command.

ssh [email protected]

Provide the password given in email and you are good to go. Once you are login, update the kernel using the following command.

sudo apt-get update

We need Java runtime system to execute Jenkins. Run the following commands to install it.

sudo apt-get install default-jre
sudo apt-get install default-jdk

Run following commands one by one to install Jenkins.

sudo wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

After installation, Jenkins will automatically start itself and you can visit the domain-name:8080 to access it.

Continous integration for Node

We have installed Jenkins, let’s add Github plugin in Jenkins in order to integrate the project.

Adding Github plugin in Jenkins

Click on “Manage Jenkins” and then click on “Manage Plugins”.

Plugin Jenkins

Now search for “GitHub” from search bar placed at right side of the screen and choose the Github plugin. There are much more with similar names but choose the one shown below.

Github plugin

Click on “Install without restart” and in few seconds plugin will be installed.

plugin installed

Configuring project with Jenkins

We have our project ready and hosted at Github and Jenkins installed, all we need now is to integrate both of them so that when we push any changes to Github, Jenkins know about it and do the testing and deployment.

Click on “New Item” and fill in the proper project name. Choose “Freestyle project” among the options.

Continious integration using Node.js

In configuration screen, check Github project and add the Github project link. Under “Source Code Management” select Git and add the Repository URL with .git extension.

You can choose which branches you want to cover up in the deployment process or leave it blank for any branch.

Under “Build Trigger” section select “Build when a change is pushed to GitHub” option. This is very important.

Under “Build” section, Click on “Add build step” button and then click “Execute shell” option.

In that for instance type “npm install” and then click on Save. Here is full-page screenshot.

Jenkins configuration

Let’s check if we have Github and Jenkins talking to each other.

Make any changes in the project and do the commit and push. As soon as you push new changes Jenkins will show you new build in the project dashboard.

Building process Jenkins

You can replace the command with mocha in order to do testing first.

Configuring deployment process

We have Github communicating to Jenkins, it’s time to perform the deployment to our Server. One thing, Jenkins will not directly communicate to your web server. All it can do is to run instruction which you want it to run.

So let’s first list down the steps we manually perform to do the deployment to any Server.

  • Login to Server using SSH.
  • Switching to project directory.
  • Pulling code from Github.
  • Restarting process manager ( say pm2 ).

We will write down all of these commands in one file and tell Jenkins to execute it.

But, what about SSH username and password ? Of course, Jenkins won’t enter it like we do. We need to make sure Jenkins can perform SSH login without using the password to our Server.

To achieve that, we will generate SSH key manually from Jenkins Server and add it into our Development Server.

Let’s do it.

Login to Jenkins Server using Terminal and switch to root user using this command.

sudo su

Jenkins automatically creates a new user after installation. Switch to it using this command.

su jenkins

Let’s generate RSA key, run following command.

ssh-keygen -t rsa

Press Enter for the location and do not type any password when it asks to, just hit enter.

Once the process is completed, print the public key information using this command.

cat ~/.ssh/id_rsa.pub

It should start with ssh-rsa and ends with [email protected]. Copy the key.

Now login to your development server and switch to ~/.ssh directory using following command.

cd ~/.ssh

If it’s not present, create one using this command.

mkdir ~/.ssh

Now open the file named authorized_keys if its present, else create one.

nano authorized_keys

Paste the key in the file, if there is already some information is present, just append the key in the new line and do not change anything else.

Once done save the file.

In order to validate whether keys are properly configured or not, switch to Jenkins Server and try to login to development server using SSH.

ssh [email protected]

If it’s not asking any password and log in successfully, you are good to go.

Now we have both the Server connecting each other. Let’s write the deployment script and finish this job.

Create a new file in your project directory with no extension. I name it deploy.

Paste following code in it.

-#!/bin/sh     
ssh [email protected] <<EOF       
 cd /your-project-path 
 git pull      
 npm install --production      
 pm2 restart all
 exit      
EOF

Save the file. Each line of the code in this file is what you actually do manually, I don’t think so I need to explain this.

Make the file executable using the following command. This is important for Shell script to execute.

chmod +x deploy

Go to Jenkins project page, from left navigation click on “Configure” and scroll down to “Build” section.

Add “./deploy” in the build commands. Here is screenshot for same.

Adding deployment script

Click on Save.

Now in the project directory, add the deploy file, commit and push it to Github.

Check the second build in Jenkins.

Build process

Refresh your project page and you shall see the changes updated in the project.

It Works. We have automated the process to do the testing and deployment every time we make some changes in the project.

Tips for integrating Github private repository or organization with Jenkins

Setting up Jenkins for the private repository is similar to what we did above. I personally configured the continuous integration and deployment environment for my organization using same steps I mentioned above except some little extra efforts due to Server configuration ( This is my guess ).

When Jenkins was trying to pull the code, it was throwing an error due to HTTPS method used by Github. I stumbled upon various methods and nothing worked for me. I changed the HTTPS origin to SSH origin and updated the repository origin too in Development server.

I also configured my account to use the SSH key in order to perform the Git operation without authentication and that you may need in order to let Jenkins do the code update.

Here is what I did.

Go to this official doc of Github to generate and add the SSH key in Github. You need to generate SSH key in your development Server and add the Public key in Github.

Link to official doc.

Once that is done, change the origin of Github repository from HTTPS to SSH.

Then log-in to development Server and switch to project directory and run this command to change the origin to SSH.

 git remote set-url origin <copy it from github>

I have solved the issue using this technique, if there is still a better way and I am sure there is, let me know in the comment or email me.

Conclusion

Continious inetegrationa and deployment using Jenkins is super easy and this approach would save lot of time of developers. They need to just focus on writing better code and Jenkins will do the rest boring stuff.

Shahid (UnixRoot) Shaikh

Hey there, This is Shahid, an Engineer and Blogger from Bombay. I am also an Author and i wrote a programming book on Sails.js, MVC framework for Node.js.

Related Posts

26 Comments

  1. Thanks for the great article. After reading many other articles, now I seems to get the understanding. I would like to get few advice on process that I would like to create in Jenkins.
    – With Jenkin Build, I would like to create tars/zip of the code for specific environments. Each build tars need to have a specific name.
    – When QA team requires, they can select one of the successful build to be deployed to a specific environment (QA/STG).

    Please let me know how it can be achieved.

    Thanks,
    Tanmay

  2. Hi,

    For second process I use the promoted builds plugin https://wiki.jenkins-ci.org/display/JENKINS/Promoted+Builds+Plugin. Our build server generates a set of build artifacts (in this case zip files) and the promoted plugin then controls the promotion step of the build. In this case you can have the a step to promote build to QA or STG. I have promotions of DevDeployed, ReadyForQA, QADeployed, PassedQA, and ReleaseCandidate and they all have dependencies to ensure that each promotion is followed. So once build promoted to ReleaseCandidate that is the build that is then deployed to Production with no rebuild.

  3. Did you forget to tell how to configure webhooks in github, github and jenkins are not talking to each other in my setup

  4. What tool is good for deploying nodejs app in a shared server, which dosent have ssh access or nodejs installed

  5. This line of the deploy script is making my build to fail:

    pm2 restart all

    Throws me this message:

    [PM2] Spawning PM2 daemon with pm2_home=/root/.pm2
    [PM2] PM2 Successfully daemonized
    Restarts are now immutable, to update environment or conf use –update-env
    [PM2][WARN] No process found

    │ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │

    Use `pm2 show ` to get more details about an app
    Build step ‘Execute shell’ marked build as failure
    Finished: FAILURE

      1. I found that the mistake is to start the app with “node app.js” as you do in the tutorial after the express code, the correct way is like your solution, starting it with pm2 start app.js, other way it will tell there’s no pm2 process.

  6. This is my Console Output in Jenkins after run the Task, I set in the Shell Script of the Task:
    SAMPLE unit test
    1) should return home page
    double callback!
    0 passing (42ms)
    1 failing
    1) SAMPLE unit test should return home page:
    Uncaught TypeError: Cannot read property ‘status’ of null

    The error is pointing me in to this line :
    res.status.should.equal(200);

    You know why is that, is that Ok?

  7. Thanks for this very nice tutorial. I have some problems in the last step:

    + ./deploy
    Pseudo-terminal will not be allocated because stdin is not a terminal.
    Host key verification failed.
    Build step ‘Execute shell’ marked build as failure
    Finished: FAILURE

    Could you help me ?. ty

  8. Hi, Great article shahid. One question if we are deploying inside a windows server what will be the content of deploy file.

    1. I would guess you could use a .bat file, you might need ssh access on the windows server for jenkins to connect also. Freessh is what I have used before. Good article too op.

  9. Hello i have one problem i added one nodejs application to jenkins from git repository but if i m trying to build that app server started but the build doesn’t stop can you please help me out how to resolve this issue..?

Leave a Reply

Your email address will not be published. Required fields are marked *