/ nodejs

Promise Parallelism with NodeJS

Following on from yesterday's tutorial on Async Patterns in NodeJS there is something I wanted to come back to and that is the idea of single threads. As you already know about NodeJS:

"JS is single threaded. As such it can only do one thing at a time - if a function call takes time to complete then all other code is blocked until it finishes."

This former statement of doing one thing at a time is not strictly true when using promises. Although the latter statements about blocking code are correct due to the way promises work we can get the semblances of multiple things happening at once. After all your multithreaded yourself right?!? I mean I can drink a cuppa and write this tutorial so why can't node!

Lets take the final example from yesterday of reading three files and outputting the results. However this time lets extend it and say we want to read these three files but lets concatenate them and create a fourth file. The original code is below for your reference:

const promise = new Promise(function(resolve, reject) {  
  fs.readFile(process.cwd() + '/' + file, (err, data) => {
    if(data) {
      resolve(data);
    }

    if(err) {
      reject(err);
    }
  });
});

const promise2 = new Promise(function(resolve, reject) {  
  fs.readFile(process.cwd() + '/' + file2, (err, data2) => {
    if(data2) {
      resolve(data2);
    }

    if(err) {
      reject(err);
    }
  });
});

const promise3 = new Promise(function(resolve, reject) {  
  fs.readFile(process.cwd() + '/' + file3, (err, data3) => {
    if(data3) {
      resolve(data3);
    }

    if(err) {
      reject(err);
    }
  });
});

promise.then((data) => {  
  console.log(data.toString());
  return promise2;
})
.then((data2) => {
  console.log(data2.toString());
  return promise3;
})
.then((data3) => {
  console.log(data3.toString());
})
.catch((err) => {
  console.log(err);
});

We could carry on reading these files one after the other with the .then() statement but this is somewhat verbose given that we're writing asynchronously here. What we really should do is start reading all these files and only write to our fourth when all reads have finished. This can be achieved through the use of Promise.all() method. It accepts an array of promises and then instructs Node to do something only when all of those promises have been resolved. With that in mind lets refactor our code. We are going to refactor the final promise chain:

First lets write a promise for our new write operation:

// Write a new promise for our write operation
const writeFile = (data) => { // Note - Wrapped in a function so we can pass data to our promise
	return new Promise((resolve, reject) => {
		fs.writeFile(process.cwd() + '/' + final, data, (err) => {
			if(err) {
				reject(err); // Ooops something went wrong with the write
			} else {
				resolve(); // Done!
			}
		})
	});
}

This time rather than returning a new promise directly to our variable we have wrapped it in a function call. This allows us to pass data into our promises. The rest should be familiar to you. Pretty cool! Now lets write a new promise chain to incorporate this:

Promise.all([promise, promise2, promise3]) // Pass in array of promises
	.then((values) => { // Return each result in an array
		var data = "";
		for(var i=0; i<values.length; i++) {
			data = data + values[i].toString('utf8'); // Loop over each result, convert to a string and add to variable
		}
		return writeFile(data); // Return our promise with our data passed in
	})
	.then(() => {
		console.log('Written file ' + final); // Log out we've written the file
	})
	.catch((err) => {
		console.log(err); // Chuck out our error
	});

There are several things going on here. Lets break it down:

  1. We call Promise.all() and pass an array of our previous promises into it. This tells node run these three in parallel.
  2. We then chain in a then() block. This accepts a parameter of values which is an array of all of the results from our promises. In the then block we loop over each converting the buffers to strings and concatenating them in a variable. Finally we return our writeFile promise with the data passed in
  3. Lastly for some feedback we output something in the console.

Here is our final.txt file:

File 1
File 2
File 3

Wrapping up:

That's it - In this tutorial we have built upon our promise knowledge gained in Async Patterns in NodeJS and learned how to run several promises in parallel and do something with the results. Final thoughts on this topic don't forget node is single threaded meaning you can create blocks and crash your application. Promise.all() only gives the illusion on multi tasking on a single thread.

Final Code:

const fs = require('fs');
const file = 'file1.txt';
const file2 = 'file2.txt';
const file3 = 'file3.txt';
const final = 'final.txt';
const promise = new Promise((resolve, reject) => {
  fs.readFile(process.cwd() + '/' + file, (err, data) => {
    if(data) {
      resolve(data);
    }

    if(err) {
      reject(err);
    }
  });
});

const promise2 = new Promise((resolve, reject) => {
  fs.readFile(process.cwd() + '/' + file2, (err, data2) => {
    if(data2) {
      resolve(data2);
    }

    if(err) {
      reject(err);
    }
  });
});

const promise3 = new Promise((resolve, reject) => {
  fs.readFile(process.cwd() + '/' + file3, (err, data3) => {
    if(data3) {
      resolve(data3);
    }

    if(err) {
      reject(err);
    }
  });
});

// Write a new promise for our write operation
const writeFile = (data) => { // Note - Wrapped in a function so we can pass data to our promise
	return new Promise((resolve, reject) => {
		fs.writeFile(process.cwd() + '/' + final, data, (err) => {
			if(err) {
				reject(err); // Ooops something went wrong with the write
			} else {
				resolve(); // Done!
			}
		})
	});
}

Promise.all([promise, promise2, promise3]) // Pass in array of promises
	.then((values) => { // Return each result in an array
		var data = "";
		for(var i=0; i<values.length; i++) {
			data = data + values[i].toString('utf8'); // Loop over each result, convert to a string and add to variable
		}
		return writeFile(data); // Return our promise with our data passed in
	})
	.then(() => {
		console.log('Written file ' + final); // Log out we've written the file
	})
	.catch((err) => {
		console.log(err); // Chuck out our error
	});