Sunday 29 November 2020

Python requests https: code 403 without but code 200 when using BurpSuite

I'm currently trying to scrap retailmenot.com this is how my code looks so far:

import requests
from collections import OrderedDict

s = requests.session()

s.headers = OrderedDict()
s.headers["Connection"] = "close"
s.headers["Upgrade-Insecure-Requests"] = "1"
s.headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
s.headers["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
s.headers["Sec-Fetch-Site"] = "none"
s.headers["Sec-Fetch-Mode"] = "navigate"
s.headers["Sec-Fetch-Dest"] = "document"
s.headers["Accept-Encoding"] = "gzip, deflate"
s.headers["Accept-Language"] = "en-GB,en-US;q=0.9,en;q=0.8"

s.get("https://www.retailmenot.com/sitemap/A")

When I use this code I instantly get redirected to a CloudFlare page. That said whenever I pass my traffic through burpsuite by replacing the last line of my code with this one:

s.get("https://www.retailmenot.com/sitemap/A", proxies = {"https":"https://127.0.0.1:8080"}, verify ="/Users/Downloads/cacert (1).pem")

I get straight to the website. I find this a bit strange and was wondering If anyone could possibly explain to me why this is happing and if there's a way to get similar results by using some different certificate (As in order to use the BurpSuite Certificate I need to keep the app open). Many thanks in advance!



from Python requests https: code 403 without but code 200 when using BurpSuite

No comments:

Post a Comment