Monday, 2 July 2018

Using scrapy-splash clicking a button

I am trying to use Scrapy-splash to click a button on a page that I'm being redirected to.

I have tested manually clicking on the page, and I am redirected to the correct page after I have clicked the button that gives my consent. I have written a small script to click the button when I am redirected to the page, but this is not working.

I have included a snippet of my spider below - am I missing something in my code?:

script="""
function main(splash)
    splash:go(splash.args.dest_url)
    splash:wait(1)
    splash:runjs('document.querySelector("input.btn.btn-primary.agree").click()')
    splash:wait(1)
    return {
        html = splash:html(),
    }
end
"""

class MySpider(scrapy.Spider):

    def do_consent_required_check(self, response):
        url = response.url

        if 'guce.oath.com/collectConsent' in url:
            yield SplashRequest(
                                    url, 
                                    callback = self.foobar, 
                                    endpoint = 'execute',
                                    args = {'lua_source': script, 'dest_url': response.meta.get('dest_url'), 'timeout': 3600},
                                    meta = response.meta 
                                )

        else:
            self.foobar(response)


    def foobar(self, response):
        pass

How do I fix this so that I can correctly give consent, so I'm directed to the page I want?



from Using scrapy-splash clicking a button

No comments:

Post a Comment