My code works fine to fetch data between n to q range but it is too slow when I want to work with a large range(1-100000). My code takes 3-4 seconds to each post request. so I want to speed it up following is my code:
import requests
from tqdm import tqdm
url = "https://www.example.com/info/item"
headers = {
'Content-Type': "application/x-www-form-urlencoded",
'Access-Control-Allow-Origin': "*",
'Accept-Encoding': "gzip, deflate",
'Accept-Language': "en-US",
}
n = 1
q = 50
sum = 0
for i in tqdm(range(n,q)):
payload = "item_id={}".format(i+1)
response = requests.request("POST", url, data=payload, headers=headers)
print(response.text)
sum = sum + i
The above code takes >150sec to execute all 50 requests.
That's why? I tried to make it Async and Now the total time it takes to send 50 requests is <2sec :
import asyncio
import requests
import aiohttp
import time
async def make_numbers(numbers, _numbers):
for i in range(numbers, _numbers):
yield i
async def fetch():
url = "https://www.example.com/info/item"
async with aiohttp.ClientSession() as session:
post_tasks = []
# prepare the coroutines that poat
async for x in make_numbers(1, 10):
post_tasks.append(do_post(session, url, x))
# now execute them all at once
await asyncio.gather(*post_tasks)
async def do_post(session, url, x):
async with session.post(url, data ={
'Content-Type': "application/x-www-form-urlencoded",
'Access-Control-Allow-Origin': "*",
'Accept-Encoding': "gzip, deflate",
'Accept-Language': "en-US",
}) as response:
data = await response.text()
print("-> Created account number %d" % x)
print (data)
s = time.perf_counter()
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(fetch())
finally:
loop.close()
elapsed = time.perf_counter() - s
print(f"{__file__} executed in {elapsed:0.2f} seconds.")
But! I got stuck... the output shows invalid "item_id". the request sends successfully but I think the parameters are not sent along with the request.
Where is it going wrong? I need solution(answer) to fix this problem.
from How to Convert Sync To Async in Python
No comments:
Post a Comment