Tuesday, 1 September 2020

Event listener for when a file has finished streaming to AWS S3 upload api?

I am creating a file backup between Google Drive and AWS S3. Where I create a Readable stream promise by downloading the file using the Google Get API and Pipping the data to AWS S3. As I have many files, each promise is added to a queue and only new promises enter when it resolves.

I'm struggling to only resolve the promise when the file has completed upload to AWS S3, rather than when the file has downloaded?

I thought using .on('finish', () => {resolve()}) should do this but it doesn't seem to be working.

Here is my code sample:

// download stream of NON gdocs files and pipe to destination
const getGFileContent = async (fileObj) => {  
  let fileExt = fileObj.path.join('/').concat('/',fileObj.name)

  return drive.files.get({fileId: fileObj.id, mimeType: fileObj.mimeType, alt: 'media'}, {responseType: 'stream'})
    .then(res => {
      return new Promise((resolve, reject) => {
        res.data
          .pipe(uploadS3(fileExt))
          .on('end', () => {console.log(`Done downloading file: ${fileExt}`)})
          .on('finish', () => {resolve(console.log(`File Backup Complete: ${fileExt}`))})
          .on('error', err => {reject(console.error(`Error downloading file: ${err}`))})
      })

// upload a file to AWS S3 by passing the file stream from getGFileContent into the 'body' parameter of the upload
const uploadS3 = (filePath) => {
  let pass = new stream.PassThrough()
  let params = {
    Bucket: awsBucketName, // bucket-name
    Key: filePath, // file will be saved as bucket-name/[uniquekey.csv]
    Body: pass  // file data passed through stream
  } 
  new aws.S3().upload(params).promise()
    .then(() => console.log(`Successfully uploaded to S3: ${filePath}`))
    .catch( err => console.log(`Error, unable to upload to S3: ${err}`))
  return pass
}


from Event listener for when a file has finished streaming to AWS S3 upload api?

No comments:

Post a Comment