Saturday, 18 January 2020

Can't scrape different results derived from different searches instead of a single result multiple times

I've written a script to parse the linked name populated upon filling in two inputboxes in a webpage (first name and last name) taken from a csv file. That csv file contains thousands of names and I'm trying to scrape the linked name using those names.

The problem is the spider always scrapes the linked name of the last name.

How can I scrape the individual linked name associated with each search?

This is few of the first and last names chronologically for your consideration [taken from the csv file]:

ANTONIO AMADOR      ACOSTA 
JOHN ROBERT         ADAIR 
ROBERT CURTIS       ADAMEK 
CY RITCHIE          ADAMS 

I've tried like this:

import csv
import scrapy
from scrapy.crawler import CrawlerProcess

class AmsrvsSpider(scrapy.Spider):
    name = "amsrvsSpiderscript"
    lead_url = "https://amsrvs.registry.faa.gov/airmeninquiry/Main.aspx"

    def start_requests(self):
        with open("document.csv","r") as f:
            reader = csv.DictReader(f)
            itemlist = [item for item in reader]

        for item in itemlist:
            yield scrapy.Request(self.lead_url,meta={"fname":item['FIRST NAME'],"lname":item['LAST NAME']},dont_filter=True, callback=self.parse)

    def parse(self,response):
        fname = response.meta.get("fname")
        lname = response.meta.get("lname")
        payload = {item.css('::attr(name)').get(default=''):item.css('::attr(value)').get(default='') for item in response.css("input[name]")}
        payload['ctl00$content$ctl01$txtbxFirstName'] = fname
        payload['ctl00$content$ctl01$txtbxLastName'] = lname
        payload.pop('ctl00$content$ctl01$btnClear')
        yield scrapy.FormRequest(self.lead_url,formdata=payload,dont_filter=True,callback=self.parse_content)

    def parse_content(self,response):
        name = response.css("a[id$='lnkbtnAirmenName']::text").get()
        print(name)


if __name__ == "__main__":
    c = CrawlerProcess({
        'USER_AGENT': 'Mozilla/5.0',
        'DOWNLOAD_TIMEOUT' : 5,
        'LOG_LEVEL':'ERROR'
    })
    c.crawl(AmsrvsSpider)
    c.start()

This is how the result looks like in that site:

enter image description here

Current output:

CY RITCHIE  ADAMS 
CY RITCHIE  ADAMS 
CY RITCHIE  ADAMS 
CY RITCHIE  ADAMS 

Expected output:

ANTONIO AMADOR  ACOSTA 
JOHN ROBERT     ADAIR 
ROBERT CURTIS   ADAMEK 
CY RITCHIE      ADAMS 


from Can't scrape different results derived from different searches instead of a single result multiple times

No comments:

Post a Comment