Buffers in Nodejs: A 3 Minute Guide To Get You Started

Buffers In Nodejs Featured Image

This tutorial talks about Buffers in Nodejs. Remember we talked about streams? Well, that was a biggy, even got me writing a Part 2 for them! Go ahead and read them: Part 1 and Part 2.

And if you already understand what Streams are, you’re all set, mi amigo!

Getting back to the point, to understand what are buffers in Nodejs, you first need to understand what streams are.

Correlating Streams with Buffers in Nodejs

Assuming you understand what I am talking about at this point, buffers in Nodejs refer to temporary data that a stream holds onto before it is consumed.

The size of a buffer in a stream is defined by a property on the stream instance known as highWatermark. This property is a number that defines the size of a buffer in bytes.

As a default behavior, a buffer memory works on Buffer and String. We may also choose to make it work on JavaScript objects. In that case, we must set the objectMode property on the stream, to true.

Streams Buffers In Nodejs Diagram
Source: Programming Hunk

Now, when some data is pushed into a stream, it resides in a stream buffer until the stream gets consumed. When the buffer memory is full, and we try to forcefully push data into the stream, the data is rejected and the output returned is false for the push action.

Buffers in Nodejs – A Standalone Definition

The Buffer class is available in the global scope in Nodejs and hence, you do not ever have to import it using the require module.

Buffers in Nodejs are of less popularity or acquaintance so to say, to Node developers since they do not interact with system memory on a day-to-day basis.

Developers who work with Go, C or C++, or any other system programming language, are much familiar with the Buffers concept.

Buffers in Nodejs are a fixed size of memory that cannot be resized. As mentioned above, their size is defined by the highWatermark property on the stream instance.

We may think of Buffers in Nodejs as an array of integers, each defining the size of the memory in bytes of data. Buffers are located outside the JavaScript V8 engine.

Why Using Buffers in Nodejs is Useful?

Buffers in Node.js should not be confused with the concept of buffering. This is only the case when a stream processor receives data at a faster pace than it can digest!

Buffers in Nodejs are pure JavaScript. Although they work very well with Unicode-encoded strings, they do not get along well with binary data. However, it is fine for the browser, since most data are strings.

Originally, Nodejs used strings to deal with TCP streams and reading & writing to the file system. However, there’s a problem with strings. Both these tasks demand dealing with binary data streams.

Although you could use strings, however, that would only bring trouble your way. First, this approach slows down the process since you’ll have to use an API compatible with strings and NOT binary data.

Second, there’s also a tendency for the code to break in unexpected ways.

So, why use binary strings, when you can use buffers in Nodejs!

Creating a Buffer in Nodejs

Buffers in Nodejs can be created using Buffer.from(), Buffer.alloc() and Buffer.allocUnsafe() methods.

const buf = Buffer.from(‘Some silly text.’);

You may even use the following syntax to create buffers in Nodejs:

  • Buffer.from(array)
  • Buffer.from(arrayBuffer[, byteOffset[, length]])
  • Buffer.from(buffer)
  • Buffer.from(string[, encoding])

We may also allocate a size when initializing a buffer. Let’s create a 2KB buffer:

const buf = Buffer.alloc(2048)
//or
const buf = Buffer.allocUnsafe(2048)

Important Note

There are certain things you should know when working with buffers in Nodejs. Although both alloc and allocUnsafe define a size to a Buffer in bytes, there are differences between the two.

The alloc() method is initialized with zeroes and the allocUnsafe() method remains uninitialized. Meaning which, although allocUnsafe() might be comparatively faster than alloc(), the allocated piece of data may contain old and potentially sensitive data.

If there are older data present in memory, can be accessed or leaked when the Buffer is read. This is exactly why it is named as allocUnsafe, and this is what makes it unsafe!

It is recommended you take extra care when using this method.

Accessing Contents of a Buffer

Earlier in this article, I mentioned Buffers in Nodejs are more like an array of bytes. Hence, as we access data from a simple array, we can do it similarly here to access the contents of a buffer using indexing.

const buf = Buffer.from('Hey!')
console.log(buf[0]) //72
console.log(buf[1]) //101
console.log(buf[2]) //121

The numbers you see are UTF-8 bytes identifying the characters in that buffer (H → 72, e → 101, y → 121). You see this because UTF-8 is the default Unicode encoding. It is important to note that some characters may take up more of a byte of memory.

To print the entire contents of the buffer, use the toString() method:

console.log(buf.toString())

By default, buf.toString() also uses UTF-8 encoding.

When you initialize a buffer with a pre-specified number for size, you will not get an empty buffer, rather you would find some random data inside it.

Changing Contents of a Buffer

You can write an entire string of data to buffers in Nodejs by making use of the write() method.

const buf = Buffer.alloc(4)
buf.write('Dan!')

Just as we change data in an array by first accessing it, we can do the same here to change the contents of buffers in Nodejs:

const buf = Buffer.from('Dan!')
buf[2] = 79 // y in UTF-8
console.log(buf.toString()) //Day!

Conclusion

To understand what are buffers in Nodejs, you first need to understand what streams are. Read this post about buffers in Nodejs that explains what it is, how it works, and how you can use buffers in Nodejs projects.

Noteworthy References