I've an async code that looks like this:
-
There's a third-party function that performs some operations on the string and returns a modified string, for the purpose of this question, it's something like
non_async_func
. -
I've an
async def async_func_single
function that wraps around thenon_async_func
that performs a single operation. -
Then another
async def async_func_batch
function that nested-wraps aroundasync_func_single
to perform the function for a batch of data.
The code kind of works but I would like to know more about why/how, my questions are
-
Is it necessary to create the
async_func_single
and haveasync_func_batch
wrap around it? -
Can I directly just feed in a batch of data in
async_func_batch
to callnon_async_func
? -
I have a per_chunk function that feeds in the data in batches, is there any asyncio operations/functions that can avoid the use of pre-batching the data I want to send to
async_func_batch
?
import nest_asyncio
nest_asyncio.apply()
import asyncio
from itertools import zip_longest
from loremipsum import get_sentences
def per_chunk(iterable, n=1, fillvalue=None):
args = [iter(iterable)] * n
return zip_longest(*args, fillvalue=fillvalue)
def non_async_func(text):
return text[::-1]
async def async_func_single(text):
# Perform some string operation.
return non_async_func(text)
async def async_func_batch(batch):
tasks = [async_func_single(text) for text in batch]
return await asyncio.gather(*tasks)
# Create some random inputs
thousand_texts = get_sentences(1000)
# Loop through 20 sentence at a time.
for batch in per_chunk(thousand_texts, n=20):
loop = asyncio.get_event_loop()
results = loop.run_until_complete(async_func_batch(batch))
for i, o in zip(thousand_texts, results):
print(i, o)
from Simplify nested asyncio operations for string modification function
No comments:
Post a Comment