Node.js streams in practice
What streams are for, the four kinds, backpressure, and the few APIs that cover most real work, from my talk at Uszanowanko Programowanko #35.
This post is the written version of “Node.js Stream API”, my talk at Uszanowanko Programowanko #35 on 13 February 2019: how streams work in practice, and how to start using them without drowning. It’s the day before Valentine’s Day, so the demo streams Romeo and Juliet, and so do the examples here.
Why streams
readFile loads a whole file into memory before you can touch it. That’s fine for a config file and a problem for a multi-gigabyte log: memory grows with the input, and nothing happens until the last byte has arrived. A stream hands you the data in chunks as it comes in (64 KiB at a time for files, by default), so memory stays flat and the work starts at once.
The four kinds
- Readable: a source, such as
fs.createReadStream, an incoming HTTP request orprocess.stdin. - Writable: a destination, such as
fs.createWriteStream, an HTTP response orprocess.stdout. - Duplex: both at once, independently, such as a TCP socket.
- Transform: a duplex stream whose output is computed from its input, such as
zlib.createGzip().
Start by reading with for await
Since Node 10, readable streams are async iterables, so the gentlest way in is a loop. readline turns the chunks into lines, and since Node 11 you can loop over it the same way. Both are still marked experimental, but they are already the easiest way to read a stream:
const fs = require('fs');
const readline = require('readline');
async function countLove() {
const lines = readline.createInterface({
input: fs.createReadStream('romeo-and-juliet.txt'),
crlfDelay: Infinity,
});
let count = 0;
for await (const line of lines) {
if (/\blove\b/i.test(line)) count++;
}
console.log(`${count} lines mention love`);
}
countLove().catch(console.error);
The file is never in memory as a whole, and the same code works for a file a thousand times larger.
Backpressure
When the destination is slower than the source, a slow disk or a slow client, data piles up somewhere. Writable streams signal it: write() returns false once the internal buffer passes its highWaterMark, and the stream emits 'drain' when there is room again. A producer that ignores the signal keeps buffering in memory, which defeats the point of streaming. You rarely need to handle this by hand, because pipeline does it for you.
Connect streams with pipeline
const fs = require('fs');
const stream = require('stream');
const util = require('util');
const zlib = require('zlib');
const pipeline = util.promisify(stream.pipeline);
pipeline(
fs.createReadStream('romeo-and-juliet.txt'),
zlib.createGzip(),
fs.createWriteStream('romeo-and-juliet.txt.gz'),
).catch(console.error);
pipeline is new in Node 10. It respects backpressure, passes an error from any stage to you, and destroys every stream in the chain when one fails, and util.promisify turns its callback into a promise. The older .pipe() handles backpressure as well, but if the source fails, the destination isn’t closed for you, which is how file handles and sockets leak.
Write a Transform when you need one
const fs = require('fs');
const stream = require('stream');
const util = require('util');
const pipeline = util.promisify(stream.pipeline);
const shout = new stream.Transform({
transform(chunk, encoding, callback) {
callback(null, chunk.toString().toUpperCase());
},
});
pipeline(
fs.createReadStream('romeo-and-juliet.txt', { encoding: 'utf8' }),
shout,
fs.createWriteStream('romeo-and-juliet-loud.txt'),
).catch(console.error);
The encoding option matters more than it looks. Chunk boundaries fall wherever they fall, so a character that takes more than one byte can be split across two chunks. A stream that decodes for you handles the split; calling toString() on raw chunks yourself doesn’t.
Starting without drowning
- Read with
for awaitbefore you reach for events. - Connect streams with
pipeline, not.pipe(). - Write your own
Transformonly when no existing one fits, and keep it small. - Use
objectModewhen the chunks are objects rather than bytes.