Wednesday, 31 July 2019

Can't scrape the links of different companies from a website using requests

I'm trying to get the links of different companies from a webpage but the script I've tried with throws the error below. In chrome dev tools I could see that I can get the ids of different companies using post http requests. However, if I can get the ids then I will be able to make use of this link 'https://angel.co/startups/{}' adding id's in string format to make a full-fledged company link.

Webpage link

I've tried with:

import requests

link = 'https://angel.co/company_filters/search_data'
base = 'https://angel.co/startups/{}'

payload={'sort':'signal','page':'2'}

r = requests.post(link,data=payload,headers={
    'x-requested-with':'XMLHttpRequest'
    'User-Agent":"Mozilla/5.0'
    })
print(r.json())

The above script throws the following error:

raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

How can I get the links of different companies from the aforementioned site using requests?



from Can't scrape the links of different companies from a website using requests

No comments:

Post a Comment