Thursday, 26 September 2019

How to Convert Sync To Async in Python

My code works fine to fetch data between n to q range but it is too slow when I want to work with a large range(1-100000). My code takes 3-4 seconds to each post request. so I want to speed it up following is my code:

import requests
from tqdm import tqdm

url = "https://www.example.com/info/item"
headers = {
    'Content-Type': "application/x-www-form-urlencoded",
    'Access-Control-Allow-Origin': "*",
    'Accept-Encoding': "gzip, deflate",
    'Accept-Language': "en-US",
    }

n = 1
q = 50
sum = 0
for i in tqdm(range(n,q)):
    payload = "item_id={}".format(i+1)
    response = requests.request("POST", url, data=payload, headers=headers)
    print(response.text)
    sum = sum + i

The above code takes >150sec to execute all 50 requests.

That's why? I tried to make it Async and Now the total time it takes to send 50 requests is <2sec :

import asyncio
import requests
import aiohttp
import time

async def make_numbers(numbers, _numbers):
    for i in range(numbers, _numbers):
        yield i

async def fetch():
    url = "https://www.example.com/info/item"
    async with aiohttp.ClientSession() as session:
        post_tasks = []
        # prepare the coroutines that poat
        async for x in make_numbers(1, 10):
            post_tasks.append(do_post(session, url, x))
        # now execute them all at once
        await asyncio.gather(*post_tasks)

async def do_post(session, url, x):
    async with  session.post(url, data ={
    'Content-Type': "application/x-www-form-urlencoded",
    'Access-Control-Allow-Origin': "*",
    'Accept-Encoding': "gzip, deflate",
    'Accept-Language': "en-US",
          }) as response:
          data = await response.text()
          print("-> Created account number %d" % x)
          print (data)

s = time.perf_counter()
loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(fetch())
finally:
    loop.close()

elapsed = time.perf_counter() - s
print(f"{__file__} executed in {elapsed:0.2f} seconds.")

But! I got stuck... the output shows invalid "item_id". the request sends successfully but I think the parameters are not sent along with the request.

Where is it going wrong? I need solution(answer) to fix this problem.



from How to Convert Sync To Async in Python

No comments:

Post a Comment