Thursday, 21 June 2018

Can't reach the bottom of a webpage

I've written a script in python with selenium to handle the infinite scrolling webpage. The problem I'm facing is that It scrolls few times then quits the browser. It never reaches the bottom. I tried with Explicit Wait as well but that gives even fewer scrolling. How can I reach the bottom when there will be no more scrolling to do.

This is my try:

import time
from selenium import webdriver
from urllib.parse import urljoin

url = "https://www.instagram.com/explore/tags/travelphotoawards/"

driver = webdriver.Chrome()
driver.get(url)

last_len = len(driver.find_elements_by_css_selector(".v1Nh3 a"))
new_len = last_len

while True:
    last_len = new_len
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    time.sleep(5)

    items = driver.find_elements_by_css_selector(".v1Nh3 a")
    new_len = len(items)
    if last_len == new_len:break

for item in items:
    print(urljoin(url,item.get_attribute("href")))
driver.quit()

Edit:

If I try like below, I can do the scrolling as many times as I want but that is not a good idea to cope with:

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

url = "https://www.instagram.com/explore/tags/travelphotoawards/"

driver = webdriver.Chrome()
driver.get(url)

for scroll in range(1,10):  #I can do the scrolling as many times as I want but it is fully hardcoded
    item = driver.find_element_by_tag_name("body")
    item.send_keys(Keys.END)
    elems = driver.find_elements_by_css_selector(".v1Nh3 a")
    time.sleep(3)

driver.quit()

I hope there is any way to do the scrolling automatically until it reaches the bottom.



from Can't reach the bottom of a webpage

No comments:

Post a Comment