Monday, 14 June 2021

scraping page links using selenium always return limited number of links

I want to scrape all matches links from this page 'https://ift.tt/3veVGzG' but can get only limiter numberof matches: I tried this code :

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless") 
driver = webdriver.Chrome(executable_path=r"C:/chromedriver.exe", options=options)

url = 'https://m.aiscore.com/basketball/20210610'
driver.get(url)

driver.maximize_window()
driver.implicitly_wait(60) 

driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")    

soup = BeautifulSoup(driver.page_source, 'html.parser')

links = [i['href'] for i in soup.select('.w100.flex a')]
links_length = len(links) #always return 16
driver.quit()

When I run the code, I get always 16 matches links only, but the page has 35 matches. I need to get allthe matches links in the page.



from scraping page links using selenium always return limited number of links

No comments:

Post a Comment