Thursday, 27 June 2019

Click Button in Scrapy-Splash

I am writing a scrapy-splash program and I need to click on the display button on the webpage, as seen in the image below, in order to display the data, for 10th edition, so I can scrape it. I have the code I tried below but it does not work. The information I need is only accessible if I click the display button. UPDATE: Still struggling with this and I have to believe there is a way to do this. I do not want to scrape the JSON because that could be a red flag to site owners.

import scrapy
from ..items import NameItem

class LoginSpider(scrapy.Spider):
    name = "LoginSpider"
    start_urls = ["http://www.starcitygames.com/buylist/"]

    def parse(self, response):

        return scrapy.FormRequest.from_response(
        response,
        formcss='#existing_users form',
        formdata={'ex_usr_email': 'email123@example.com', 'ex_usr_pass': 'password123'},
        callback=self.after_login
        )


    def after_login(self, response):
        item = NameItem()
        display_button= response.xpath('//a[contains(., "- Display>>")]/@href').get()
        response.follow(display_button, self.parse)
        item["Name"] = response.css("div.bl-result-title::text").get()
        return item

Snapshot of Webpage HTML Code



from Click Button in Scrapy-Splash

No comments:

Post a Comment