Thursday, 26 January 2023

Request headers to scrape pypi.org

I am trying to screenscrape PyPI packages using the requests library and beautiful soup - but am met with an indefinite hang. I am able to retrieve html from a number of sites with:

session = requests.Session()
session.trust_env = False
response = session.get("http://google.com")
print(response.status_code)

i.e. without providing headers. I read from Python request.get fails to get an answer for a url I can open on my browser that the indefinite hang is likely caused by incorrect headers. So, using the developer tools, I tried to grab my request headers from the Networking tab (using Edge) with "Doc" filter to select the pypi.org response/request. I simply copy pasted these into my header variable that is passed to the get method:

headers = {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
'cookie': 'session_id=<long string>',
'dnt': '1',
'sec-ch-ua': '"Not?A_Brand";v="8", "Chromium";v="108", "Microsoft Edge";v="108"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
'sec-fetch-dest': 'document',
'sec-fetch-mode': 'navigate',
'sec-fetch-site': 'none',
'sec-fetch-user': '?1',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.54'}

(and changing get method to response = session.get("http://pypi.org", headers=headers))

But I get the same hang. So, I think something is wrong with my headers but I'm not sure what. I'm aware that the requests Session() "handles" cookies so I tried removing the cookie key/value pair in my request header dictionary but achieved the same result.

How can I determine the problem with my headers and/or why do my current headers not work (assuming this is even the problem)?



from Request headers to scrape pypi.org

No comments:

Post a Comment