Saturday, 15 January 2022

Python cloudscraper requests slow, with 403 responses

I am using Cloduscraper Python library in order to obtain a JSON response from an url. The probem is that I have to retry the same request 2-3 times before I get the correct output. The first responses have a 403 HTTP status code.

Here is my code:

import json
from time import sleep
import cloudscraper

url = "https://www.endpoint.com/api/"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0",
    "Accept": "*/*",
    "Content-Type": "application/json"
}
json_response = 0
while json_response == 0:
    try:
        scraper = cloudscraper.create_scraper()
        r = scraper.get(url, headers=headers)
        json_response = json.loads(r.text)
    except:
        print(r.status_code)
        sleep(2)
return json_response

What can I do in order to optimize my code and prevent the 403 responses?



from Python cloudscraper requests slow, with 403 responses

No comments:

Post a Comment