Line-by-line file reading is a common task in Node.js when processing large files or handling data in a streaming manner. It is simple for developers to read files line by line and handle the data as needed with Node.js built-in methods and modules.
By processing data progressively, developers can prevent memory problems that would emerge from putting the full file into memory at once. Processing data and carrying out procedures more effectively is made possible by reading files line by line.
In this article, we will use two different modules to read the file line by line in Node.js:
Node.js Readline Module
An interface for reading data line-by-line from a readable stream (such as a file stream or user input) is provided by the Node.js readline module. With its user-friendly interface, developers can effortlessly design a line-by-line reader and handle each line with events or callbacks. This module works well in situations like log analysis or CSV file parsing when line-by-line processing is necessary.
Using the below lines enables you to access the readline module’s functionality in your script and is the standard way to import the readline module in Node.js:
const readline = require("readline");
const fs = require("fs");
Here we have also imported the FS module.
We will first design the input and output interfaces for the interaction:
const fileStream = fs.createReadStream('input.txt');
const rl = readline.createInterface({
input: fileStream,
output: process.stdout,
crlfDelay: Infinity
});
Below attributes are present in the object that createInterface receives:
- input: Indicates the stream of text to be read. Here, it’s configured to fileStream using the FS module.
- output: Indicates which writable stream will be utilized. This instance involves the standard output stream
- crlfDelay: stands for Carriage Return Line Feed Delay and is applicable when dealing with text files that might have different line endings.
The readline interface emits several events, but two of the most commonly used events are:
rl.on('line', function (line) {
console.log('Line from file:', line);
});
rl.on('close', function () {
console.log('all done');
});
Here:
- rl.on(‘line’, …): Event listener is triggered for each line read from the file. It logs each line to the console.
- rl.on(‘close’, …): Event listener is triggered when the file reading is complete. It logs a message to the console indicating that all lines have been read.
Using Readline Module for Reading a File Line by Line in Node.js
Let’s see an example based on the above approach to read a file line by line using readline module.
const fs = require('fs');
const readline = require('readline');
const fileStream = fs.createReadStream('input.txt');
const rl = readline.createInterface({
input: fileStream,
output: process.stdout,
crlfDelay: Infinity
});
rl.on('line', function (line) {
console.log('Line from file:', line);
});
rl.on('close', function () {
console.log('all done');
});
In this example, the script uses the FS (File System) and readline modules to read a file (input.txt) line by line.
First, it utilizes FS to build a readable stream using the fs.createReadStream(‘input.txt’), and the outcome is kept in the fileStream variable. After that, readline is used to establish an interface using createInterface() method. process.stdout is selected as the output, while the filestream variable is selected as the input option. Configuring crlfDelay to Infinity will recognize that the readline module will treat any sequence of “\r ” or “\n” characters as the end of a line. It delays emitting the ‘line’ event until it encounters the end of the line.
The script also configures the readline interface event listeners. Every time a new line is read from the file, the ‘line’ event is triggered. The line is logged to the console by the associated callback function. When the full file has been read, the callback function logs “all done” to the console and initiates the “close” event.
Output:
Run the Node.js script by running the below command.
node index.js
This script is going to display every line from “input.txt” to the console as it runs, and it will terminate by printing “all done”.
NPM Line-Reader Module
The line reader NPM module defines itself as “Asynchronous, buffered, line-by-line file/stream reader with support for user-defined line separators.” on its GitHub page. Since it is not the native module, you must install it with the following line in npm (Node Package Manager):
npm install line-reader
Using Line-Reader Module for Reading a File Line by Line in Node.js
In this example, we will see how to read the data of a file asynchronously line by line using the line reader.
const lineReader = require("line-reader");
let i = 1;
lineReader.eachLine("input.txt", function (line, last) {
console.log(`Line from file ${i}: ${line}`);
i++;
if (last) {
console.log("Last line printed.");
}
});
This Node.js code reads a file called “input.txt” line by line by using the “line-reader” module. A counter variable, initialized to 1, is defined in the code. Then, it iterates over each line in the file using the lineReader.eachLine function. Every time a line is recorded, the callback function given in the second argument of eachLine is called. It receives the text of the line as well as a boolean final parameter that indicates whether or not this is the last line in the file.
Using a template string, the callback outputs the line number (i) and the line content to the console. Next, the value of the counter i is increased. It also produces a message indicating that it has reached the final line if the current line is the last (last is true).
Output:
Run the above script by using the below command.
node index.js
This example of line-reader module is going to display every line from “input.txt” with the variable “i” to the console as it runs, and it will end the execution of the script by printing “Last line printed”.
Conclusion
Here, we have demonstrated two methods for reading a file line by line: one using an established open-source library (line-reader) and the other with a native Node.js module (readline). Select a method based on your requirements. It’s also a good idea to utilize these methods to prevent the Node server from crashing if the amount of data increases.
