Wednesday, 28 August 2019

NodeJS MongoDB Mongoose export nested subdocuments and arrays to XLSX columns

I have query results from MongoDB as an array of documents with nested subdocuments and arrays of subdocuments.

[
  {
    RecordID: 9000,
    RecordType: 'Item',
    Location: {
      _id: 5d0699326e310a6fde926a08,
      LocationName: 'Example Location A'
    }
    Items: [
      {
        Title: 'Example Title A',
        Format: {
          _id: 5d0699326e310a6fde926a01,
          FormatName: 'Example Format A'
        }
      },
      {
        Title: 'Example Title B',
        Format: {
          _id: 5d0699326e310a6fde926a01,
          FormatName: 'Example Format B'
        }
      }
    ],
  },
  {
    RecordID: 9001,
    RecordType: 'Item',
    Location: {
      _id: 5d0699326e310a6fde926a08,
      LocationName: 'Example Location C'
    },
    Items: [
      {
        Title: 'Example Title C',
        Format: {
          _id: 5d0699326e310a6fde926a01,
          FormatName: 'Example Format C'
        }
      }
    ],
  }
]

Problem

I need to export the results to XLSX in column order. The XLSX library is working to export the top-level properties (such as RecordID and RecordType) only. I also need to export the nested objects and arrays of objects. Given a list of property names e.g. RecordID, RecordType, Location.LocationName, Items.Title, Items.Format.FormatName the properties must be exported to XLSX columns in the specified order.

Desired result (or something similar -- I just need a 'flat' structure to be able to convert to XLSX columns)

[
  {
    RecordID: 9000,
    RecordType: 'Item',
    Location.LocationName: 'Example Location A',
    Items.Title: 'Example Title A', 'Example Title B',
    Items.Format.FormatName: 'Example Format A', 'Example Format B',
  },
  {
    RecordID: 9001,
    RecordType: 'Item',
    Location.LocationName: 'Example Location C',
    Items.Title: 'Example Title C',
    Items.Format.FormatName: 'Example Format C',
  }
]

I am using the XLSX library to convert the query results to XLSX which works for top-level properties only.

  const worksheet: XLSX.WorkSheet = XLSX.utils.json_to_sheet(results.data);
  const workbook: XLSX.WorkBook = { Sheets: { 'data': worksheet }, SheetNames: ['data'] };
  const excelBuffer: any = XLSX.write(workbook, { bookType: 'xlsx', type: 'array' });

  const data: Blob = new Blob([excelBuffer], { type: EXCEL_TYPE });
  FileSaver.saveAs(data, new Date().getTime());

POSSIBLE OPTIONS

I am guessing I need to 'flatten' the structure either using aggregation in the query or by performing post-processing when the query is returned.

Option 1: Build the logic in the MongoDB query to flatten the results.

$replaceRoot might work since it is able to "promote an existing embedded document to the top level". Although I am not sure if this will solve the problem exactly, I do not want to modify the documents in place, I just need to flatten the results for exporting.

Here is the MongoDB query I am using to produce the results:

records.find({ '$and': [ { RecordID: { '$gt': 9000 } } ]},
  { skip: 0, limit: 10, projection: { RecordID: 1, RecordType: 1, 'Items.Title': 1, 'Items.Location': 1 }});

Option 2: Iterate and flatten the results on the Node server

This is likely not the most performant option, but might be the easiest if I can't find a way to do so within the MongoDB query.



from NodeJS MongoDB Mongoose export nested subdocuments and arrays to XLSX columns

No comments:

Post a Comment