Monday, 28 October 2019

Python3 Flask asyncio subprocess in route hangs

I'm using Flask 1.0.2 with Python 3.6 on Ubuntu 18.04. My app should use asyncio and "create_subprocess_exec" to lauch a background script, read stdout from it, and then return status when the script is done.

I am basically trying to implement an answer from this post: Non-blocking read on a subprocess.PIPE in python

The script is successfully launched, and I get all of my expected output from it, but the problem is that it never returns ( meaning the "Killing subprocess now" line is never reached ). When I check the process list ( ps ) from the Linux terminal, the background script has exited.

What am I doing wrong and how can I successfully break out of the "async for line in process.stdout" loop?

At the top of my file after my imports I create my event loop:

# Create a loop to run all the tasks in.
global eventLoop ; asyncio.set_event_loop(None)
eventLoop = asyncio.new_event_loop()
asyncio.get_child_watcher().attach_loop(eventLoop)

I define my async coroutine above my route:

async def readAsyncFunctionAndKill(cmd):

    # Use global event loop
    global eventLoop

    print("[%s] Starting async Training Script ..." % (os.path.basename(__file__)))
    process = await asyncio.create_subprocess_exec(cmd,stdout=PIPE,loop=eventLoop)
    print("[%s] Starting to read stdout ..." % (os.path.basename(__file__)))
    async for line in process.stdout:
        line = line.decode(locale.getpreferredencoding(False))
        print("%s"%line, flush=True)
    print("[%s] Killing subprocess now ..." % (os.path.basename(__file__)))
    process.kill()
    print("[%s] Training process return code was: %s" % (os.path.basename(__file__), process.returncode))
    return await process.wait()  # wait for the child process to exit

And my ( abbreviated ) route is here:

@app.route("/train_model", methods=["GET"])
def train_new_model():

    # Use global event loop
    global eventLoop   

    with closing(eventLoop):        
        eventLoop.run_until_complete(readAsyncFunctionAndKill("s.py"))

    return jsonify("done"), 200


from Python3 Flask asyncio subprocess in route hangs

1 comment: