Saturday, 18 December 2021

Simplify nested asyncio operations for string modification function

I've an async code that looks like this:

  • There's a third-party function that performs some operations on the string and returns a modified string, for the purpose of this question, it's something like non_async_func.

  • I've an async def async_func_single function that wraps around the non_async_func that performs a single operation.

  • Then another async def async_func_batch function that nested-wraps around async_func_single to perform the function for a batch of data.

The code kind of works but I would like to know more about why/how, my questions are

  • Is it necessary to create the async_func_single and have async_func_batch wrap around it?

  • Can I directly just feed in a batch of data in async_func_batch to call non_async_func?

  • I have a per_chunk function that feeds in the data in batches, is there any asyncio operations/functions that can avoid the use of pre-batching the data I want to send to async_func_batch?

import nest_asyncio
nest_asyncio.apply()

import asyncio
from itertools import zip_longest

from loremipsum import get_sentences

def per_chunk(iterable, n=1, fillvalue=None):
  args = [iter(iterable)] * n
  return zip_longest(*args, fillvalue=fillvalue)

def non_async_func(text):
  return text[::-1]

async def async_func_single(text):
  # Perform some string operation.
  return non_async_func(text)

async def async_func_batch(batch):
  tasks = [async_func_single(text) for text in batch]
  return await asyncio.gather(*tasks)

# Create some random inputs
thousand_texts = get_sentences(1000)

# Loop through 20 sentence at a time.
for batch in per_chunk(thousand_texts, n=20):  
  loop = asyncio.get_event_loop()
  results = loop.run_until_complete(async_func_batch(batch))
  for i, o in zip(thousand_texts, results):
    print(i, o)


from Simplify nested asyncio operations for string modification function

No comments:

Post a Comment