Quick summary: In this tutorial, we are going to program a voice-controlled drone using Node.js and Web speech API. We will use Parrot ARDrone 2.0 as our drone and Node.js + web speech API to build our application.
Introduction
Drones are amazing. I personally love the ability to sit down and fly the drone to either capture pictures/videos or just having fun.
Drones are not just for fun though. Drones are used heavily in the movie production industry, glacier exploration, military, agriculture and many more.
Consumer Drones are getting popular due to ease in regulation by many countries and great quality drones manufactured by DJI and other providers.
In this tutorial, we are going to create a program that will allow us to control and maneuver our drone using voice commands. That’s right, the drone will do as you command.
Before diving into development, do have a look at this one minute video where I am flying the drone in my hotel room (Not safe at all).
Let’s look at the things you require beforehand.
Source code and demo
Hardware
You need the following hardware:
You need to have a standard laptop running Windows/Mac/Ubuntu. I have tested it on Mac and Ubuntu 18.04.
Software
You must have the latest version of Node.js installed in your system. Visit this site to download Node.js.
You need to have the latest version of Google Chrome installed in your system. Visit this site to download chrome.
Understanding the drone
Let’s try to understand the built and structure of the Parrot ARDrone.
The Parrot ARDrone is a quadcopter i.e there are 4 rotating motors attached to each corner.
Each pair of opposite rotors is turning the same way. One pair is turning clockwise and the other anti-clockwise.
The drone movements are obtained by the changing pitch, roll and yaw angles.
As you can observe in the diagram above, by changing the left and right rotors speed the opposite way yields roll movement. This allows the drone to go forth and back.
By changing front and rear rotors speeds the opposite way yields pitch movement.
By changing each rotor pair speed the opposite way yields yaw movement. This allows turning left and right.
You really don’t need to learn aerodynamics to fly the drone, this is just for the basic information about the drone.
How Parrot ARDrone works
Parrot ARDrone provides its own WiFi access point. One needs to connect to it in order to send the command to the drone.
There are various mobile apps which you can use to fly the drone using your mobile. We will also use the WiFi access point to connect to the drone.
Let’s do the basic connection first. Connect the battery in the drone and it will boot the drone system.
Check your WiFi connection. You should an access point naming similar to one shown in the screenshot below.
Connect to the WiFi network. Once connected, open up the terminal and try to do a telnet to 192.168.1.1 – this is the IP address of the drone.
You should see successful telnet into the drone operating system running Linux Busybox.
The App Architecture
Our app codebase is divided into the following section:
- User interface with Web speech API to detect voice.
- Command filtering and matching.
- Sending commands to the drone.
- Live video streaming
The web speech API works with an internet connection. The reason why we are adding ethernet connection because we will connect to the drone WiFi access point and we need internet to detect our voice using web speech API.
Let’s build our app.
Coding the app
Let’s code our app. First, create a new folder and switch to it using the terminal.
Create a new Node project using the following command.
Let’s install the required dependency.
We are going to support the following commands.
- Takeoff
- Land
- Up – go up about half meter distance then stops.
- Down – go down about half meter distance then stops.
- Go left – go left about half meter distance then stops.
- Go right – go right about half meter distance then stops.
- Turn – turn clockwise 90 degrees.
- Go forward – go forward about a meter distance then stops.
- Go backward – go backward about a meter distance then stops.
- stop
Let’s code our server.
Here is our code to receive the command, filter it and control the drone.
const bodyparser = require('body-parser');
const arDrone = require('ar-drone');
const router = express.Router();
const app = express();
const commands = ['takeoff', 'land','up','down','goleft','goright','turn','goforward','gobackward','stop'];
var drone = arDrone.createClient();
// disable emergency
drone.disableEmergency();
// express
app.use(bodyparser.json());
app.use(express.static(__dirname + '/public'));
router.get('/',(req,res) => {
res.sendFile('index.html');
});
router.post('/command',(req,res) => {
console.log('command recieved ', req.body);
console.log('existing commands', commands);
let command = req.body.command.replace(/ /g,'');
if(commands.indexOf(command) !== -1) {
switch(command.toUpperCase()) {
case "TAKEOFF":
console.log('taking off the drone');
drone.takeoff();
break;
case "LAND":
console.log('landing the drone');
drone.land();
break;
case "UP":
console.log('taking the drone up half meter');
drone.up(0.2);
setTimeout(() => {
drone.stop();
clearTimeout();
},2000);
break;
case "DOWN":
console.log('taking the drone down half meter');
drone.down(0.2);
setTimeout(() => {
drone.stop();
clearTimeout();
},2000);
break;
case "GOLEFT":
console.log('taking the drone left 1 meter');
drone.left(0.1);
setTimeout(() => {
drone.stop();
clearTimeout();
},1000);
break;
case "GORIGHT":
console.log('taking the drone right 1 meter');
drone.right(0.1);
setTimeout(() => {
drone.stop();
clearTimeout();
},1000);
break;
case "TURN":
console.log('turning the drone');
drone.clockwise(0.4);
setTimeout(() => {
drone.stop();
clearTimeout();
},2000);
break;
case "GOFORWARD":
console.log('moving the drone forward by 1 meter');
drone.front(0.1);
setTimeout(() => {
drone.stop();
clearTimeout();
},2000);
break;
case "GOBACKWARD":
console.log('moving the drone backward 1 meter');
drone.back(0.1);
setTimeout(() => {
drone.stop();
clearTimeout();
},2000);
break;
case "STOP":
drone.stop();
break;
default:
break;
}
}
res.send('OK');
});
app.use('/',router);
app.listen(process.env.port || 3000);
Here is the HTML and JavaScript code to detect speech and send a command to Node server.
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Voice Controlled Notes App</title>
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/shoelace-css/1.0.0-beta16/shoelace.css">
<link rel="stylesheet" href="styles.css">
</head>
<body>
<div class="container">
<h1>Voice Controlled Drone</h1>
<p class="page-description">A tiny app that allows you to control AR drone using voice</p>
<h3 class="no-browser-support">Sorry, Your Browser Doesn't Support the Web Speech API. Try Opening This Demo In Google Chrome.</h3>
<div class="app">
<h3>Give the command</h3>
<div class="input-single">
<textarea id="note-textarea" placeholder="Create a new note by typing or using voice recognition." rows="6"></textarea>
</div>
<button id="start-record-btn" title="Start Recording">Start Recognition</button>
<button id="pause-record-btn" title="Pause Recording">Pause Recognition</button>
<p id="recording-instructions">Press the <strong>Start Recognition</strong> button and allow access.</p>
</div>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
<script src="script.js"></script>
</body>
</html>
And here is the JavaScript code to handle the voice commands and sending it back to the Node server.
var SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition;
var recognition = new SpeechRecognition();
}
catch(e) {
console.error(e);
$('.no-browser-support').show();
$('.app').hide();
}
// other code, please refer GitHub source
recognition.onresult = function(event) {
// event is a SpeechRecognitionEvent object.
// It holds all the lines we have captured so far.
// We only need the current one.
var current = event.resultIndex;
// Get a transcript of what was said.
var transcript = event.results[current][0].transcript;
// send it to the backend
$.ajax({
type: 'POST',
url: '/command/',
data: JSON.stringify({command: transcript}),
success: function(data) { console.log(data) },
contentType: "application/json",
dataType: 'json'
});
};
Let’s run the code.
Running the app
Run the app using the following command. Make sure you are connected to the drone WiFi and ethernet cable is connected to the computer as well.
Open localhost:3000 in the browser and click on Start Recognition.
Say Takeoff and see the drone taking off from the ground. Try other commands too and have fun.
Streaming live video from the drone
Create a new file inside the project folder and copy/paste this code.
const drone = require("dronestream");
const server = http.createServer(function(req, res) {
require("fs").createReadStream(__dirname + "/public/video.html").pipe(res);
});
drone.listen(server);
server.listen(4000);
Here is the HTML code. Place it inside public folder.
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<title>Stream as module</title>
<script src="/dronestream/nodecopter-client.js" type="text/javascript" charset="utf-8"></script>
</head>
<body>
<h1 id="heading">Drone video stream</h1>
<div id="droneStream" style="width: 640px; height: 360px"> </div>
<script type="text/javascript" charset="utf-8">
new NodecopterStream(document.getElementById("droneStream"));
</script>
</body>
</html>
Run the code and view localhost:8080 to view the live video feed from the front camera.
Tips
- Make sure the ground over which drone is flying is free from bug obstacles.
- Please fly this drone indoors.
- Always put the protective cover on the drone before a takeoff.
- Make sure the battery is charged.
- In case of drone acts weird, hold the drone from the bottom and flip it. It will put the drone in emergency mode and rotors will stop immediately.
Conclusion
Writing code and seeing the machine acts feels awesome and when that machine is a drone it’s extraordinary feeling. We learned how to code a program that can control the drone using our voice commands. We can code tons of awesome stuff such as face recognition using drone camera, autonomous flights, gesture recognition etc.
Let me know what you are going to build next?
