I'm trying to grab job titles from the search result of a webpage, indeed.com, using the requests module. Here is the link to that webpage where I wish to fetch the job titles.
The following is how I've already tried:
import requests
from bs4 import BeautifulSoup
link = "https://www.indeed.com/jobs"
params={
'q': 'motorcycle mechanic',
'l': 'New York, NY'
}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}
def get_job_titles(url):
res = requests.get(url,params=params,headers=headers)
soup = BeautifulSoup(res.text,"lxml")
link_list = []
for item in soup.select("#mosaic-jobResults td.resultContent h2 > a > span[id^='jobTitle']"):
link_list.append(item.get("href"))
return link_list
if __name__ == '__main__':
for title in get_job_titles(link):
print(title)
When I run the script, I always get status 403. How can I get the job titles from that webpage using the requests module?
from Failed to fetch the job titles from indeed using the requests module
No comments:
Post a Comment