Python:
Regarding the bounty: please, if possible, in addition to helping me solve the current problem, indicate me an improved and faster option for the method I currently use. (I'm still learning, so my methods are pretty archaic)
Code Proposal Summary:
Collecting the links to all the games of the day present on the page (https://int.soccerway.com/matches/2021/07/31/), giving me the freedom to change the date to whatever I want, such as 2021/08/01 and so on. So that in the future I can loop and collect the list from several different days at the same time, in one code call.
Even though it's a very slow model, without using Headless, this model clicks all the buttons, expands the data and imports all 465 listed match links:
for btn in driver.find_elements_by_xpath("//tr[contains(@class,'group-head clickable')]"):
btn.click()
Full Code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
options = Options()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-logging"])
driver = webdriver.Chrome(r"C:\Users\Computador\Desktop\Python\chromedriver.exe", options=options)
url = "https://int.soccerway.com/matches/2021/07/28/"
driver.get(url)
driver.find_element_by_xpath("//div[@class='language-picker-trigger']").click()
driver.find_element_by_xpath("//a[@href='https://int.soccerway.com']").click()
time.sleep(10)
for btn in driver.find_elements_by_xpath("//tr[contains(@class,'group-head clickable')]"):
btn.click()
time.sleep(10)
jogos = driver.find_elements_by_xpath("//td[contains(@class,'score-time')]//a")
for jogo in jogos:
resultado = jogo.get_attribute("href")
print(resultado)
driver.quit()
But when I add options.add_argument("headless") so that the browser is not opened on my screen, the model returns the following error:
Message: element click intercepted
To get around this problem, I analyzed options and found this one on WebDriverWait (https://stackoverflow.com/a/62904494/11462274) and tried to use it like this:
for btn in WebDriverWait(driver, 1).until(EC.element_to_be_clickable((By.XPATH, "//tr[contains(@class,'group-head clickable')]"))):
btn.click()
Full Code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
options.add_argument("headless")
options.add_experimental_option("excludeSwitches", ["enable-logging"])
driver = webdriver.Chrome(r"C:\Users\Computador\Desktop\Python\chromedriver.exe", options=options)
url = "https://int.soccerway.com/matches/2021/07/28/"
driver.get(url)
driver.find_element_by_xpath("//div[@class='language-picker-trigger']").click()
driver.find_element_by_xpath("//a[@href='https://int.soccerway.com']").click()
time.sleep(10)
for btn in WebDriverWait(driver, 1).until(EC.element_to_be_clickable((By.XPATH, "//tr[contains(@class,'group-head clickable')]"))):
btn.click()
time.sleep(10)
jogos = driver.find_elements_by_xpath("//td[contains(@class,'score-time')]//a")
for jogo in jogos:
resultado = jogo.get_attribute("href")
print(resultado)
driver.quit()
But because it's not iterable, it returns in error:
'NoneType' object is not iterable
Why do I need this option?
1 - I'm going to automate it in an online terminal, so there won't be any browser to open on the screen and I need to make it fast so I don't spend too much of my time limits on the terminal.
2 - I need to find an option that I can use any date instead of 2021/07/28 in:
url = "https://int.soccerway.com/matches/2021/07/28/"
Where in the future I'll add the parameter:
today = date.today().strftime("%Y/%m/%d")
In this answer (https://stackoverflow.com/a/68535595/11462274), a guy indicated a very fast and interesting option (He named the option at the end of the answer as: Quicker Version) without the need for a WebDriver, but I was only able to make it work on the first page of the site, when I try to use other dates of the year, he keeps returning only the links to the games of the current day.
Expected Result (there are 465 links but I didn't put the entire result because there is a character limit):
https://int.soccerway.com/matches/2021/07/28/europe/uefa-champions-league/fc-sheriff-tiraspol/alashkert-fc/3517568/
https://int.soccerway.com/matches/2021/07/28/europe/uefa-champions-league/fk-neftchi/olympiakos-cfp/3517569/
https://int.soccerway.com/matches/2021/07/28/europe/uefa-champions-league/scs-cfr-1907-cluj-sa/newcastle-fc/3517571/
https://int.soccerway.com/matches/2021/07/28/europe/uefa-champions-league/fc-midtjylland/celtic-fc/3517576/
https://int.soccerway.com/matches/2021/07/28/europe/uefa-champions-league/fk-razgrad-2000/mura/3517574/
https://int.soccerway.com/matches/2021/07/28/europe/uefa-champions-league/galatasaray-sk/psv-nv/3517577/
https://int.soccerway.com/matches/2021/07/28/europe/uefa-champions-league/bsc-young-boys-bern/k-slovan-bratislava/3517566/
https://int.soccerway.com/matches/2021/07/28/europe/uefa-champions-league/fk-crvena-zvezda-beograd/fc-kairat-almaty/3517570/
https://int.soccerway.com/matches/2021/07/28/europe/uefa-champions-league/ac-sparta-praha/sk-rapid-wien/3517575/
https://int.soccerway.com/matches/2021/07/28/world/olympics/saudi-arabia-u23/brazil--under-23/3497390/
https://int.soccerway.com/matches/2021/07/28/world/olympics/germany-u23/cote-divoire-u23/3497391/
https://int.soccerway.com/matches/2021/07/28/world/olympics/romania-u23/new-zealand-under-23/3497361/
https://int.soccerway.com/matches/2021/07/28/world/olympics/korea-republic-u23/honduras-u23/3497362/
https://int.soccerway.com/matches/2021/07/28/world/olympics/australia-under-23/egypt-under-23/3497383/
https://int.soccerway.com/matches/2021/07/28/world/olympics/spain-under-23/argentina-under-23/3497384/
https://int.soccerway.com/matches/2021/07/28/world/olympics/france-u23/japan-u23/3497331/
https://int.soccerway.com/matches/2021/07/28/world/olympics/south-africa-u23/mexico-u23/3497332/
https://int.soccerway.com/matches/2021/07/28/africa/cecafa-senior-challenge-cup/uganda-under-23/eritrea-under-23/3567664/
https://int.soccerway.com/matches/2021/07/28/africa/cecafa-senior-challenge-cup/ethiopia-u23/congo-dr-under-23/3567663/
https://int.soccerway.com/matches/2021/07/28/argentina/primera-division/boca-juniors/club-atletico-san-lorenzo-de-almagro/3528753/
https://int.soccerway.com/matches/2021/07/28/argentina/primera-division/ca-union-de-santa-fe/ca-banfield/3528752/
https://int.soccerway.com/matches/2021/07/28/argentina/primera-division/godoy-cruz-antonio-tomba/club-atletico-tucuman/3528762/
https://int.soccerway.com/matches/2021/07/28/argentina/primera-division/club-atletico-sarmiento/ca-platense/3528758/
https://int.soccerway.com/matches/2021/07/28/argentina/primera-division/club-atletico-velez-sarsfield/csd-defensa-y-justicia/3528761/
https://int.soccerway.com/matches/2021/07/28/argentina/primera-division/ca-independiente/club-atletico-patronato/3528757/
https://int.soccerway.com/matches/2021/07/28/australia/queensland/queensland-lions-sc/capalaba/3460465/
https://int.soccerway.com/matches/2021/07/28/australia/queensland/sunshine-coast-wanderers/brisbane-roar-ii/3460466/
https://int.soccerway.com/matches/2021/07/28/australia/queensland-pl2-youth/north-star-u23/samford-rangers-u23/3498140/
https://int.soccerway.com/matches/2021/07/28/australia/western-australia-npl-women/australia-murdoch-university-melville-fc/perth/3469912/
https://int.soccerway.com/matches/2021/07/28/australia/western-australia-npl-women/freemantle-city/northern-redbacks/3469950/
https://int.soccerway.com/matches/2021/07/28/australia/capital-territory-npl-youth-league/monaro-panthers-u23/cooma-tigers-u23/3433866/
https://int.soccerway.com/matches/2021/07/28/australia/capital-territory-npl-2-youth/canberra-we-u23/weston-molonglo-u23/3433696/
https://int.soccerway.com/matches/2021/07/28/australia/capital-territory-npl-women/gungahlin-united/australia-canberra-fc/3433608/
https://int.soccerway.com/matches/2021/07/28/australia/northern-nsw/weston-workers/broadmeadow-magic/3432958/
https://int.soccerway.com/matches/2021/07/28/australia/northern-nsw/valentine-phoenix/adamstown-rosebuds/3432957/
https://int.soccerway.com/matches/2021/07/28/australia/northern-nsw/edgeworth-eagles/lake-macquarie/3432954/
https://int.soccerway.com/matches/2021/07/28/australia/northern-nsw-reserve-league/edgeworth-eagles-res/lake-macquarie-res/3434270/
https://int.soccerway.com/matches/2021/07/28/australia/brisbane-womens-cup/australia-grange-thistle-sc/australia-albany-creek-excelsior-fc/3514706/https://int.soccerway.com/matches/2021/07/28/austria/regionalliga/fc-red-bull-salzburg-amateure/sv-wals-grunau/3559518/
https://int.soccerway.com/matches/2021/07/28/austria/regionalliga/sk-bischofshofen/sportverein-austria-salzburg/3559520/
https://int.soccerway.com/matches/2021/07/28/austria/regionalliga/admira-dornbirn/sc-austria-lustenau-ii/3560565/
https://int.soccerway.com/matches/2021/07/28/austria/regionalliga/fc-rot-weiss-rankweil/sc-bregenz/3560566/
https://int.soccerway.com/matches/2021/07/28/austria/regionalliga/dornbirner-sv/fc-lauterach/3560567/
https://int.soccerway.com/matches/2021/07/28/austria/landesliga/fc-hochst/bezau/3577691/
https://int.soccerway.com/matches/2021/07/28/austria/landesliga/fc-nenzing/fc-andelsbuch/3577692/
https://int.soccerway.com/matches/2021/07/28/austria/landesliga/fc-lustenau-07/ludesch/3577693/
https://int.soccerway.com/matches/2021/07/28/austria/landesliga/gofis/sc-rheindorf-altach-ii/3577694/
https://int.soccerway.com/matches/2021/07/28/austria/landesliga/fc-alberschwende/schruns/3577695/
https://int.soccerway.com/matches/2021/07/28/austria/landesliga/fc-hard/sc-fussach/3577696/
https://int.soccerway.com/matches/2021/07/28/austria/landesliga/horbranz/sk-cht-austria-meiningen/3577697/
https://int.soccerway.com/matches/2021/07/28/austria/ofb-stiegl-cup/sv-kuchl/fc-blau-weiss-linz/3527604/
https://int.soccerway.com/matches/2021/07/28/belarus/2-division/ostrovets-fc/slonim-city/3501140/
https://int.soccerway.com/matches/2021/07/28/belarus/2-division/belaya-rus/hcs-olympia/3501141/
https://int.soccerway.com/matches/2021/07/28/belarus/2-division/svislach/schuchin/3501137/
https://int.soccerway.com/matches/2021/07/28/belarus/2-division/neman-mosty/fk-tsementnik-krasnoselsky/3501138/
https://int.soccerway.com/matches/2021/07/28/belarus/2-division/smena-vawkavysk/chayka-zelva/3501139/
https://int.soccerway.com/matches/2021/07/28/belarus/2-division/uzda/kolos-cherven/3502257/
https://int.soccerway.com/matches/2021/07/28/belarus/2-division/fc-osipovichi/spartak-shklov/3502182/
https://int.soccerway.com/matches/2021/07/28/belarus/2-division/krasnopole/fk-dnepr-mogilev-ii/3502176/
https://int.soccerway.com/matches/2021/07/28/belarus/premier-league-women/gomel/dinamo-bgu/3477008/
https://int.soccerway.com/matches/2021/07/28/brazil/serie-b/botafogo-de-futebol-e-regatas/centro-sportivo-alagoano/3482911/
https://int.soccerway.com/matches/2021/07/28/brazil/copa-do-brasil/criciuma-esporte-clube/fluminense-football-club/3521228/
https://int.soccerway.com/matches/2021/07/28/brazil/copa-do-brasil/esporte-clube-vitoria/gremio-foot-ball-porto-alegrense/3521231/
https://int.soccerway.com/matches/2021/07/28/brazil/copa-do-brasil/clube-atletico-paranaense/atletico-clube-goianiense/3521234/
https://int.soccerway.com/matches/2021/07/28/brazil/carioca-a2/audax-rio-de-janeiro-ec/associacao-desportiva-cabofriense/3508338/
https://int.soccerway.com/matches/2021/07/28/brazil/carioca-a2/cfrj--marica/duque-de-caxias-futebol-clube/3508339/
https://int.soccerway.com/matches/2021/07/28/brazil/carioca-a2/angra-dos-reis-esporte-clube/goncalense/3508340/
https://int.soccerway.com/matches/2021/07/28/brazil/carioca-a2/america-football-club-rio-de-janeiro/artsul-futebol-clube/3508341/
https://int.soccerway.com/matches/2021/07/28/brazil/carioca-a2/macae-esporte-futebol-clube/americano-futebol-clube/3508342/
https://int.soccerway.com/matches/2021/07/28/brazil/carioca-a2/sampaio-correa-fe/friburguense-atletico-clube/3508343/
https://int.soccerway.com/matches/2021/07/28/brazil/catarinense-2/sociedade-esportiva-recreativa-clube-guarani/atletico-catarinense/3527050/
https://int.soccerway.com/matches/2021/07/28/brazil/catarinense-2/nacao/acre-cidade-azul/3527051/
https://int.soccerway.com/matches/2021/07/28/brazil/cearense-2-div/floresta/cariri/3582360/
https://int.soccerway.com/matches/2021/07/28/brazil/cearense-2-div/uniao-ce/itapipoca-esporte-clube/3582361/
https://int.soccerway.com/matches/2021/07/28/brazil/cbf-brasileiro-u20/fortaleza-u19/atletico-go-u19/3520266/
https://int.soccerway.com/matches/2021/07/28/brazil/cbf-brasileiro-u20/cr-flamengo-u20/cruzeiro-ac-u20/3520267/
https://int.soccerway.com/matches/2021/07/28/brazil/cbf-brasileiro-u20/botafogo-fc-u20/sport-club-do-recife-u20/3520269/
https://int.soccerway.com/matches/2021/07/28/brazil/cbf-brasileiro-u20/ca-mineiro-u20/sociedade-esportiva-palmeiras-u20/3520270/
https://int.soccerway.com/matches/2021/07/28/brazil/cbf-brasileiro-u20/santos-futebol-clube-sao-paulo-u20/sao-paulo-futebol-clube-u20/3520273/
https://int.soccerway.com/matches/2021/07/28/brazil/cbf-brasileiro-u20/ec-bahia-u20/cr-vasco-da-gama-u20/3520271/
https://int.soccerway.com/matches/2021/07/28/brunei-darussalam/premier-league/jerudong-fc/kuala-belait/3521922/
https://int.soccerway.com/matches/2021/07/28/chile/primera-division/audax-club-sportivo-italiano/corporacion-deportiva-everton/3478522/
https://int.soccerway.com/matches/2021/07/28/chile/primera-division/club-de-desportes-cobresal/deportes-melipilla/3478523/
https://int.soccerway.com/matches/2021/07/28/chile/primera-division/union-espanola/club-deportivo-huachipato/3478526/
https://int.soccerway.com/matches/2021/07/28/china-pr/csl/hebei-zhongji/dalian-aerbin-fc/3492367/
https://int.soccerway.com/matches/2021/07/28/china-pr/csl/beijing-guoan-football-club/shanghai-east-asia/3492368/
https://int.soccerway.com/matches/2021/07/28/china-pr/csl/tianjin-teda/changchun-yatai/3492366/
https://int.soccerway.com/matches/2021/07/28/china-pr/csl/shanghai-shenhua/hubei-luyin-fc/3492365/
https://int.soccerway.com/matches/2021/07/28/china-pr/china-league-one/nanjing-city/guizhou-zhicheng-toro-fc/3545195/
https://int.soccerway.com/matches/2021/07/28/china-pr/china-league-one/shenyang-city/hubei-huakaier/3545193/
https://int.soccerway.com/matches/2021/07/28/china-pr/china-league-one/wuhan-three-towns/sichuan-fc/3545196/
https://int.soccerway.com/matches/2021/07/28/colombia/primera-a/la-equidad/deportivo-pasto/3554519/
https://int.soccerway.com/matches/2021/07/28/colombia/primera-b/real-cartagena/valledupar-fc/3553296/
https://int.soccerway.com/matches/2021/07/28/costa-rica/primera-division/municipal-perez-zeledon/adr-jicaral/3529055/
https://int.soccerway.com/matches/2021/07/28/costa-rica/primera-division/sporting-san-jose/ad-guanacasteca/3529056/
https://int.soccerway.com/matches/2021/07/28/costa-rica/primera-division/deportivo-saprissa/santos-de-guapiles-fc/3529053/
https://int.soccerway.com/matches/2021/07/28/costa-rica/primera-division/grecia-fc/club-sport-cartagines/3529054/
https://int.soccerway.com/matches/2021/07/28/ecuador/primera-b/alianza-cotopaxi/cumbaya/3468963/
https://int.soccerway.com/matches/2021/07/28/ecuador/primera-b/gualaceo/club-deportivo-america/3468962/
https://int.soccerway.com/matches/2021/07/28/ecuador/primera-b/puerto-quito/ldu-de-portoviejo/3468965/
https://int.soccerway.com/matches/2021/07/28/estonia/esiliiga/fc-elva/fc-flora-tallinn-ii/3499410/
Note 1: There are multiple types of score-time, such as score-time status and score-time score, that's why I used contains in "//td[contains(@class,'score-time')]//a"
from Without using WebDriverWait return: element click intercepted, with WebDriverWait returns 'NoneType' object is not iterable
No comments:
Post a Comment