I'm currently trying to scrap retailmenot.com this is how my code looks so far:
import requests
from collections import OrderedDict
s = requests.session()
s.headers = OrderedDict()
s.headers["Connection"] = "close"
s.headers["Upgrade-Insecure-Requests"] = "1"
s.headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
s.headers["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
s.headers["Sec-Fetch-Site"] = "none"
s.headers["Sec-Fetch-Mode"] = "navigate"
s.headers["Sec-Fetch-Dest"] = "document"
s.headers["Accept-Encoding"] = "gzip, deflate"
s.headers["Accept-Language"] = "en-GB,en-US;q=0.9,en;q=0.8"
s.get("https://www.retailmenot.com/sitemap/A")
When I use this code I instantly get redirected to a CloudFlare page. That said whenever I pass my traffic through burpsuite by replacing the last line of my code with this one:
s.get("https://www.retailmenot.com/sitemap/A", proxies = {"https":"https://127.0.0.1:8080"}, verify ="/Users/Downloads/cacert (1).pem")
I get straight to the website. I find this a bit strange and was wondering If anyone could possibly explain to me why this is happing and if there's a way to get similar results by using some different certificate (As in order to use the BurpSuite Certificate I need to keep the app open). Many thanks in advance!
from Python requests https: code 403 without but code 200 when using BurpSuite
No comments:
Post a Comment