Sunday, 22 January 2023

Unable to scrape different company names from a static webpage using the requests module

I've created a script to collect the different company names from this website using the requests module, but when I execute the script, it ends up getting nothing. I looked for the company names in the page source and found that the names are available there, so they seem to be static.

import requests
from bs4 import BeautifulSoup

link = 'https://clutch.co/agencies/digital-marketing'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link)
    soup = BeautifulSoup(res.text,"lxml")
    for item in soup.select("h3.company_info > a"):
        print(item.text)


from Unable to scrape different company names from a static webpage using the requests module

No comments:

Post a Comment