I've created a script to collect the different company names from this website using the requests module, but when I execute the script, it ends up getting nothing. I looked for the company names in the page source and found that the names are available there, so they seem to be static.
import requests
from bs4 import BeautifulSoup
link = 'https://clutch.co/agencies/digital-marketing'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}
with requests.Session() as s:
s.headers.update(headers)
res = s.get(link)
soup = BeautifulSoup(res.text,"lxml")
for item in soup.select("h3.company_info > a"):
print(item.text)
from Unable to scrape different company names from a static webpage using the requests module
No comments:
Post a Comment