I read that Selenium Chrome can run faster if you use implicit waits, headless, ID and CSS selectors etc. Before implementing those changes, I want to know whether cookies or caching could be slowing me down.
Does Selenium store cookies and cache like a normal browser or does it reload all assets everytime it navigates to a new page on a website?
If yes, then this would slow down the process of scraping websites with millions of identical profile pages, where the scripts and images are similar for each profile.
Is yes, is there a way to avoid this problem? Interested in using cookies and cache during a session and then destroying after the browser is closed.
Edit, more details:
sel_options = {'proxy': {'https': pString}}
prefs = {'download.default_directory' : dFolder}
options.add_experimental_option('prefs', prefs)
blocker = os.path.join( os.getcwd(), "extension_iijehicfndmapfeoplkdpinnaicikehn")
options.add_argument(f"--load-extension={blocker}")
wS = "--window-size="+s1+","+s2
options.add_argument(wS)
if headless == "yes": options.add_argument("--headless");
driver = uc.Chrome(seleniumwire_options=sel_options, options=options, use_subprocess=True, version_main=109)
stealth(driver, languages=["en-US", "en"], vendor="Google Inc.", platform="Win32", webgl_vendor="Intel Inc.", renderer="Intel Iris OpenGL Engine", fix_hairline=True)
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": agent})
navigate("https://linkedin.com")
I don't think my proxy or extension is the culprit, because I have a similar automation app running with no speed issue.
from Selenium Python Chrome: Extremely slow. Are cookies the problem?
No comments:
Post a Comment