Sunday 27 September 2020

how to download larger binary object from postgres using nodejs express?

I have node js rest api script to query from Postgres tables. It's working fine, but When I try to fetch a large objects using "lo_get" of Postgres, I get

JavaScript heap out of memory

Below is the toy example.

index.js

const express = require('express')
const bodyParser = require('body-parser')
const app = express()
const port = 8000
var controller= require('./src/controller');

app.use(bodyParser.json())
app.use(
  bodyParser.urlencoded({
    extended: true,
  })
)

app.use('/cont', controller);

app.listen(port, () => {
  console.log(`App running on port ${port}.`)
})

controller.js

var express = require('express');
var router = express.Router();

router.get('/fileID/:lo_oid/', function( req, res) {
    const lo_oid =  parseInt(req.params.lo_oid)
    model.exportLoSaint(lo_oid, function (err, file) {
        if (err) return res.send(err);
        response.status(200).file(file);
    });
});

module.exports = router;

Model.js

const Pool = require('pg').Pool
var exportLoSaintQuery= "SELECT lo_get($1)"

const exportLoSaint = (lo_oid, callback) => {
    pool.query( exportLoSaintQuery ,[lo_oid], (error, results) => {
                if (error) {
                    callback(error);
                }
                else {
                    callback(results.rows)
                }
            })
        }

on sending request .. http://ipaddress:8000/cont/fileID/<fileID>/ getting below error.


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1b2cbd65be1d]
Security context: 0x2bf3bbc08bd9 <JSObject>
    1: toJSON [0x259ded584f29] [buffer.js:~979] [pc=0x1b2cbd665145](this=0x306d2be17731 <Uint8Array map = 0x9dd44699a69>)
    2: arguments adaptor frame: 1->0
    3: InternalFrame [pc: 0x1b2cbd60ee75]
    4: EntryFrame [pc: 0x1b2cbd6092c1]
    5: builtin exit frame: stringify(this=0x0a6855c58f61 <Object map = 0x428c84042a9>,0x2160e26026f1 <undefined>,0x2160e...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0x56228da912e4 node::Abort() [node]
 2: 0x56228da91332  [node]
 3: 0x56228dc86d4a v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0x56228dc86fc5 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0x56228e0387d6  [node]
 6: 0x56228e04e018 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
 7: 0x56228e01a63b v8::internal::Factory::AllocateRawArray(int, v8::internal::PretenureFlag) [node]
 8: 0x56228e01aad8 v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::RootListIndex, int, v8::internal::Object*, v8::internal::PretenureFlag) [node]
 9: 0x56228dfe48fe  [node]
10: 0x56228dfe4ad3  [node]
11: 0x56228e143566 v8::internal::JSObject::AddDataElement(v8::internal::Handle<v8::internal::JSObject>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::ShouldThrow) [node]
12: 0x56228e17897e v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::LanguageMode, v8::internal::Object::StoreFromKeyed) [node]
13: 0x56228e2c1299 v8::internal::Runtime::SetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::LanguageMode) [node]
14: 0x56228e0a9b45 v8::internal::Runtime_KeyedStoreIC_Slow(int, v8::internal::Object**, v8::internal::Isolate*) [node]
15: 0x1b2cbd65be1d 

The size of binary object in postgres is only 128 mb. So, I increased the size of node js to 1gb, it did not help. Other solution I found was not using express to download binary from postgres.



from how to download larger binary object from postgres using nodejs express?

No comments:

Post a Comment