Saturday, 1 April 2023

Failed to fetch the job titles from indeed using the requests module

I'm trying to grab job titles from the search result of a webpage, indeed.com, using the requests module. Here is the link to that webpage where I wish to fetch the job titles.

The following is how I've already tried:

import requests
from bs4 import BeautifulSoup

link = "https://www.indeed.com/jobs"
params={
    'q': 'motorcycle mechanic',
    'l': 'New York, NY'
}
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}
def get_job_titles(url):
    res = requests.get(url,params=params,headers=headers)
    soup = BeautifulSoup(res.text,"lxml")
    link_list = []
    for item in soup.select("#mosaic-jobResults td.resultContent h2 > a > span[id^='jobTitle']"):
        link_list.append(item.get("href"))
    return link_list

if __name__ == '__main__':
    for title in get_job_titles(link):
        print(title)

When I run the script, I always get status 403. How can I get the job titles from that webpage using the requests module?



from Failed to fetch the job titles from indeed using the requests module

No comments:

Post a Comment