Friday 5 March 2021

Selenium Not Finding Element Present in HTML Even After Waiting for DOM to update

I am trying to scrape information on a website where the information is not immediately present. When you click a certain button, the page begins to load new content on the bottom of the page, and after it's done loading, red text shows up as "Assists (At Least)". I am able to find the first button "Go to Prop builder", which doesn't immediately show up on the page, but after the script clicks the button, it times out when trying to find the "Assists (At Least)" text, in spite of the script sleeping and being present on the screen.

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By 
import time
from bs4 import BeautifulSoup

driver = webdriver.Chrome()
driver.get('https://www.bovada.lv/sports/basketball/nba')

# this part succeeds
element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located(
        (By.XPATH, "//span[text()='Go to Prop builder']")
    )
)
element.click()
time.sleep(5)

# this part fails 
element2 = WebDriverWait(driver, 6).until(
    EC.visibility_of_element_located(
        (By.XPATH, "//*[text()='Assists (At Least)']")
    )
)
time.sleep(2)

innerHTML = driver.execute_script('return document.body.innerHTML')
driver.quit()
    
soup = BeautifulSoup(innerHTML, 'html.parser')


from Selenium Not Finding Element Present in HTML Even After Waiting for DOM to update

No comments:

Post a Comment