Friday, 9 August 2019

Node v12.7 How to implement native brotli, gzip, deflate compression buffer

I am a front end developer upping my chops on the backend here.

I have a Node express server that hosts an app and serves a REST api on the same server/AWS EC2 instance.

I was using express-static-gzip npm package to enable brotli compression for the static app bundle and assets. This was great. Then however, I had to switch to SSR for Three.js objects since phones couldn't handle the massive dataset parsing. Unfortunately, it didn't apply compression to my REST data.

Currently, in the interim I have disabled express-static-gzip and enabled the compression npm package. This is enabling gzip only but for both the static bundle AND the REST API.

I specifically need Brotli with GZIP/Deflate fallback compression on both my static bundle AND my REST API. The largest GET request uncompressed can be 138MB. GZIP gets it down to 12.8MB. I want it under 10MB with Brotli.

My intention is to have express-static-gzip running compression on my bundle and manual node zlib compression on my REST API. If that isn't feasible then manual node zlib compression for everything!

I don't understand some things about Buffers and backend type of things... perhaps you can tell me what I'm doing wrong here:

app.get('/quakeData/:index', function(req, res){
    // Send Specific Selection or All
    const encoding = req.headers['accept-encoding'], 
          index    = req.params.index,
          jsonArr  = index != "all" ? [ quakes[index], threeData[index] ] : [ quakes, threeData ],
          jsonStr  = JSON.stringify(jsonArr),
          bData    = Buffer.from(jsonStr);

    if (encoding.includes('br')) {
        console.log("BROTLI RES");
        zlib.brotliCompress(bData, (err, result) => {
            console.log(result);
            !err ? res.send(result) : console.warn(err);
        });

    } else if (encoding.includes('gzip')) {
        console.log("GZIP RES");
        zlib.gzip(bData, (err, result) => {
            console.log(result);
            !err ? res.send(result) : console.warn(err);
        });

    } else if (encoding.includes('deflate')) {
        console.log("DEFLATE RES");
        zlib.deflate(bData, (err, result) => {
            console.log(result);
            !err ? res.send(result) : console.warn(err);
        })

    } else {
        console.warn("Unsupported Content Encoding Headers");
        res.setHeader('Content-Type', 'application/json');
        res.json(jsonArr);
    }

Also, I've realized that the compression module removes Content Length headers because my XHR Progress API code stopped working. I need to have Content Length headers no matter what solution is implemented. How do I go about that? Also, is there a way to setup a GET to receive content length ahead of time to estimate download times?

Thank you kindly!



from Node v12.7 How to implement native brotli, gzip, deflate compression buffer

No comments:

Post a Comment