I've written a script to parse the linked name populated upon filling in two inputboxes in a webpage (first name and last name) taken from a csv file. That csv file contains thousands of names and I'm trying to scrape the linked name using those names.
The problem is the spider always scrapes the linked name of the last name.
How can I scrape the individual linked name associated with each search?
This is few of the first and last names chronologically for your consideration [taken from the csv file]:
ANTONIO AMADOR ACOSTA
JOHN ROBERT ADAIR
ROBERT CURTIS ADAMEK
CY RITCHIE ADAMS
I've tried like this:
import csv
import scrapy
from scrapy.crawler import CrawlerProcess
class AmsrvsSpider(scrapy.Spider):
name = "amsrvsSpiderscript"
lead_url = "https://amsrvs.registry.faa.gov/airmeninquiry/Main.aspx"
def start_requests(self):
with open("document.csv","r") as f:
reader = csv.DictReader(f)
itemlist = [item for item in reader]
for item in itemlist:
yield scrapy.Request(self.lead_url,meta={"fname":item['FIRST NAME'],"lname":item['LAST NAME']},dont_filter=True, callback=self.parse)
def parse(self,response):
fname = response.meta.get("fname")
lname = response.meta.get("lname")
payload = {item.css('::attr(name)').get(default=''):item.css('::attr(value)').get(default='') for item in response.css("input[name]")}
payload['ctl00$content$ctl01$txtbxFirstName'] = fname
payload['ctl00$content$ctl01$txtbxLastName'] = lname
payload.pop('ctl00$content$ctl01$btnClear')
yield scrapy.FormRequest(self.lead_url,formdata=payload,dont_filter=True,callback=self.parse_content)
def parse_content(self,response):
name = response.css("a[id$='lnkbtnAirmenName']::text").get()
print(name)
if __name__ == "__main__":
c = CrawlerProcess({
'USER_AGENT': 'Mozilla/5.0',
'DOWNLOAD_TIMEOUT' : 5,
'LOG_LEVEL':'ERROR'
})
c.crawl(AmsrvsSpider)
c.start()
This is how the result looks like in that site:
Current output:
CY RITCHIE ADAMS
CY RITCHIE ADAMS
CY RITCHIE ADAMS
CY RITCHIE ADAMS
Expected output:
ANTONIO AMADOR ACOSTA
JOHN ROBERT ADAIR
ROBERT CURTIS ADAMEK
CY RITCHIE ADAMS
from Can't scrape different results derived from different searches instead of a single result multiple times

No comments:
Post a Comment