File Download and Streaming Process with Next.js and CDN Integration

I am attempting to create a download system for large files on my website. I am using Next.js, and my large files are hosted on a CDN. What I need is to download several files from my CDN, create a zip file, and send this archive file to the client. I already have this working with the following structure:

/api/download.ts:

export const getS3Object = async (bucketParams: any) => {
  try {
    const response = await s3Client.send(new GetObjectCommand(bucketParams))
    return response
  } catch (err) {
    console.log('Error', err)
  }
}

async function downloadRoute(req: NextApiRequest, res: NextApiResponse) {
  if (req.method === 'POST') {
    res.setHeader('Content-Disposition', 'attachment; filename=pixel_capture_HDR.zip')
    res.setHeader('Content-Type', 'application/zip')

    const fileKeys = req.body.files
    const archive = archiver('zip', { zlib: { level: 9 } })

    archive.on('error', (err: Error) => {
      console.error('Error creating ZIP file:', err)
      res.status(500).json({ error: 'Error creating ZIP file' })
    })

    archive.pipe(res)

    try {
      for (const fileKey of (fileKeys as string[])) {
        const params = { Bucket: 'three-assets', Key: fileKey }
        const s3Response = await getS3Object(params) as Buffer
        archive.append(s3Response.Body, { name: fileKey.substring(fileKey.lastIndexOf('/') + 1) })
      }

      archive.finalize()
    } catch (error) {
      console.error('Error retrieving files from S3:', error)
      res.status(500).json({ error: 'Error retrieving files from S3' })
    }
  } else {
    res.status(405).json({ error: `Method '${req.method}' Not Allowed` })
  }
}

export const config = { api: { responseLimit: false } }

export default withIronSessionApiRoute(downloadRoute, sessionOptions)

my download function in pages/download.tsx:

  const handleDownload = async() => {
    setLoading(true)

    const files = [files_to_download]

    const response = await fetch('/api/download', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ files: files })
    })

    if (response.ok) {
      const blob = await response.blob()
      const url = URL.createObjectURL(blob)

      const link = document.createElement('a')
      link.href = url
      link.download = 'pixel_capture_HDR.zip'
      link.click()

      URL.revokeObjectURL(url)
    }

    setLoading(false)
  }

With the above structure, I display a loading UI on the client when they start fetching the files. Once everything is ready and zipped by the server, the zip file is downloaded almost instantly to the user’s download folder.

The problem with this structure is that if the user closes the browser tab, the download would stop and fail. What I hope to achieve is to stream/pipe the download and the zipping directly into the download queue of the browser, if that makes sense.

I attempted to use Readable Streams on the client side, which strangely seems to work because when I log the push function, I can see the values being logged and the “done” variable changing from false to true when everything is downloaded. However, the actual browser download of the zip file is still triggered when everything is ready, instead of “streaming” the download as soon as the user clicks the download button.

  if (response.ok) {
    const contentDispositionHeader = response.headers.get('Content-Disposition')
    
    const stream = response.body
    const reader = stream.getReader()

    const readableStream = new ReadableStream({
      start(controller) {
        async function push() {
          const { done, value } = await reader.read()
          if (done) {
            controller.close()
            return
          }
          controller.enqueue(value)
          push()
        }
        push()
      }
    })

    const blob = new Blob([readableStream], { type: 'application/octet-stream' })
    const url = URL.createObjectURL(blob)
  }

I realize it’s quite challenging to explain what I want, and I hope it’s understandable and achievable.

I would greatly appreciate any tips on how to achieve such a solution.

Thanks in advance for the help!

from File Download and Streaming Process with Next.js and CDN Integration