Monday 5 October 2020

Can't get rid of some error page which shows up while using asyncio

I'm trying to extract the address of different properties using some links I already have in a text file. I've created this script using asyncio library. The script is doing fine until it encounters this type of page thrown by that site. I also checked implementing proxies but no luck. Although it certainly is not a captcha page, I end up getting that page after few requests while using asyncio. FYI, when I go for requests module, I don't encounter that page.

How can I get rid of that error page?

Here are few of the urls that I'm using in the text file.

I've tried with:

import asyncio
import aiohttp
import random
import requests
from bs4 import BeautifulSoup

async def get_text(session,url):
    async with session.get(url,ssl=False) as resp:
        assert resp.status == 200
        print("----------",str(resp.url))
        if "Error" in str(resp.url):raise
        return await resp.read()

async def get_info(sem,session,link):
    async with sem:
        r = await get_text(session,link)          
        soup = BeautifulSoup(r,"html.parser")
        try:
            address = soup.select_one("h1#mainaddresstitle").get_text(strip=True)
        except AttributeError: address = ""
        print(address)

async def main():
    sem = asyncio.Semaphore(5)
    with open("link_list.txt","r") as f:
        link_list = [url.strip() for url in f.readlines()]
        
    async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=10)) as session:
        await asyncio.gather(
            *(get_info(sem,session,item) for item in link_list)
        )

if __name__ == '__main__':
    asyncio.run(main())

PS when the script crosses the rate limit, it is supposed to encounter some page like /Property/UsageValidation but not /Property/Error/?id=14e53e71-11b1-4f5e-a88c-f8a4721de99e



from Can't get rid of some error page which shows up while using asyncio

No comments:

Post a Comment