Tuesday, 1 February 2022

Can't fetch tabular content from a webpage using requests

I would like to scrape tabular content from the landing page of this website. There are 100 rows in it's first page. When I observe network activity in dev tools, I could notice that some get requests is being issued to this url https://io6.dexscreener.io/u/ws3/screener3/ with appropriate parameters which ends up producing json content.

However, when I try to mimic that requests through my following efforts:

import requests

url = 'https://io6.dexscreener.io/u/ws3/screener3/'
params = {
    'EIO': '4',
    'transport': 'polling',
    't': 'NwYSrFK',
    'sid': 'ztAOHWOb-1ulTq-0AQwi',
}

headers = {
    'accept': '*/*',
    'referer': 'https://dexscreener.com/',
    'user-agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36'
}
with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(url,params=params)
    print(res.content)

I get this response:

`{"code":3,"message":"Bad request"}`

How can I get response having tabular content from that webpage?



from Can't fetch tabular content from a webpage using requests

No comments:

Post a Comment