I want to scrape all matches links from this page 'https://ift.tt/3veVGzG' but can get only limiter numberof matches: I tried this code :
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
driver = webdriver.Chrome(executable_path=r"C:/chromedriver.exe", options=options)
url = 'https://m.aiscore.com/basketball/20210610'
driver.get(url)
driver.maximize_window()
driver.implicitly_wait(60)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
soup = BeautifulSoup(driver.page_source, 'html.parser')
links = [i['href'] for i in soup.select('.w100.flex a')]
links_length = len(links) #always return 16
driver.quit()
When I run the code, I get always 16 matches links only, but the page has 35 matches. I need to get allthe matches links in the page.
from scraping page links using selenium always return limited number of links
No comments:
Post a Comment