Sunday, 28 October 2018

How can I capture all network requests and full response data when loading a page in Chrome?

Using Puppeteer, I'd like to load a URL in Chrome and capture the following information:

  • request URL
  • request headers
  • request post data
  • response headers text (including duplicate headers like set-cookie)
  • transferred response size (i.e. compressed size)
  • full response body

Capturing the full response body is what causes the problems for me.

Things I've tried:

  • Getting response content with response.buffer - this does not work if there are redirects at any point, since buffers are wiped on navigation
  • intercepting requests and using getResponseBodyForInterception - this means I can no longer access the encodedLength, and I also had problems getting the correct request and response headers in some cases
  • Using a local proxy works, but this slowed down page load times significantly (and also changed some behavior for e.g. certificate errors)

Ideally the solution should only have a minor performance impact and have no functional differences from loading a page normally. I would also like to avoid forking Chrome.



from How can I capture all network requests and full response data when loading a page in Chrome?

No comments:

Post a Comment