I am trying to use Scrapy-splash to click a button on a page that I'm being redirected to.
I have tested manually clicking on the page, and I am redirected to the correct page after I have clicked the button that gives my consent. I have written a small script to click the button when I am redirected to the page, but this is not working.
I have included a snippet of my spider below - am I missing something in my code?:
script="""
function main(splash)
splash:go(splash.args.dest_url)
splash:wait(1)
splash:runjs('document.querySelector("input.btn.btn-primary.agree").click()')
splash:wait(1)
return {
html = splash:html(),
}
end
"""
class MySpider(scrapy.Spider):
def do_consent_required_check(self, response):
url = response.url
if 'guce.oath.com/collectConsent' in url:
yield SplashRequest(
url,
callback = self.foobar,
endpoint = 'execute',
args = {'lua_source': script, 'dest_url': response.meta.get('dest_url'), 'timeout': 3600},
meta = response.meta
)
else:
self.foobar(response)
def foobar(self, response):
pass
How do I fix this so that I can correctly give consent, so I'm directed to the page I want?
from Using scrapy-splash clicking a button
No comments:
Post a Comment