I'm trying to extract the address of different properties using some links I already have in a text file. I've created this script using asyncio library. The script is doing fine until it encounters this type of page thrown by that site. I also checked implementing proxies but no luck. Although it certainly is not a captcha page, I end up getting that page after few requests while using asyncio. FYI, when I go for requests module, I don't encounter that page.
How can I get rid of that error page?
Here are few of the urls that I'm using in the text file.
I've tried with:
import asyncio
import aiohttp
import random
import requests
from bs4 import BeautifulSoup
async def get_text(session,url):
async with session.get(url,ssl=False) as resp:
assert resp.status == 200
print("----------",str(resp.url))
if "Error" in str(resp.url):raise
return await resp.read()
async def get_info(sem,session,link):
async with sem:
r = await get_text(session,link)
soup = BeautifulSoup(r,"html.parser")
try:
address = soup.select_one("h1#mainaddresstitle").get_text(strip=True)
except AttributeError: address = ""
print(address)
async def main():
sem = asyncio.Semaphore(5)
with open("link_list.txt","r") as f:
link_list = [url.strip() for url in f.readlines()]
async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=10)) as session:
await asyncio.gather(
*(get_info(sem,session,item) for item in link_list)
)
if __name__ == '__main__':
asyncio.run(main())
PS when the script crosses the rate limit, it is supposed to encounter some page like /Property/UsageValidation
but not /Property/Error/?id=14e53e71-11b1-4f5e-a88c-f8a4721de99e
from Can't get rid of some error page which shows up while using asyncio
No comments:
Post a Comment