Friday, 22 March 2019

How to stop NodeJS "Request" module changes request when using proxy

Sorry if this comes off as confusing.

I have written a script using the NodeJS request module that runs and performs a function on a website then returns with the data. This script works perfectly fine when I do not use a proxy by setting it to false. This is not a task that is NOT allowed to be done with Selenium/puppeteer

proxy: false

However, when I set a (working) proxy. It fails to perform the same task and is detected by the website firewall/antibot software.

proxy: http://xx.xxx.xx.xx:3128

Some things to note:

  • I have tried many (20+) different proxy providers (Residential and Datacenter) and they all have this issue
  • The issue does not occur if that proxy is set globally on my system
  • The issue does not occur if that proxy is set in a chrome extension
  • The SSL cipher suites do not match Chrome but they still don't match when not using a proxy so I assume that isn't the issue
  • It is very important to keep consistency in the header order

The question basically is. Does the request module change anything when using a proxy such as the header order?

The only difference is changing the proxy that causes this to fail. One request being made with, one request being made without.

        simple : false,
        forever: true,
        resolveWithFullResponse: true,
        gzip: true,
        headers: {
            'Host'             : 'www.sitename.com',
            'Connection'       : 'keep-alive',
            'Upgrade-Insecure-Requests': '1',
            'User-Agent'       : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36',
            'Accept'           : 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
            'Accept-encoding'  : 'gzip, deflate, br',
            'Accept-Language'  : 'en-GB,en-US;q=0.9,en;q=0.8',
        },
        method : 'GET',
        jar: globalJar,
        simple: false,
        followRedirect: false,
        followAllRedirects: false, ```




from How to stop NodeJS "Request" module changes request when using proxy

No comments:

Post a Comment