Friday 13 November 2020

Understanding streams in Node js

I am stuck on an issue where I need to create and download a zip of multiple files using NodeJs. Things I have tried and failed :

https://github.com/archiverjs/node-archiver/issues/364#issuecomment-573508251

https://github.com/kubernetes/kubernetes/issues/90483#issue-606722433

unexpected behavior using zip-stream NPM on Google k8s

In addition to this, now the files are encrypted as well so I have to decrypt them before adding them to zip on the fly.

Though I solved this also, my solution works perfectly while the server is running on a local machine but failed when on the Google Kubernetes Engine.

After some more research, I guess this might be because of a backpressure issue in the streams in NodeJs but as described in docs, backpressure is handled by the pipe method automatically. Is it possible that the receiving speed of the browser is not matching with the sending speed of my server/zipping if yes how to solve this problem?

All the samples related to the problem are in the links provided above.

In Addition to this readable stream is passed through decipher to decrypt it.

const decipher = crypto.createDecipheriv(algorithm, Buffer.from(key), Buffer.from(key.substring(0, 9)));
zipStreamer.entry(readableStream.pipe(decipher), {
    name: fileName
}, (error, result) => {
    if (!error) {
        resolve("done");
    } else {
        reject("error");
    }
});


from Understanding streams in Node js

No comments:

Post a Comment