Sunday 14 May 2023

Managing a long-running API listener using Python / Flask

My python script (e.g. downloader.py) listens to a streaming REST API endpoint which continuously sends it data. After making an initial GET request, it uses a function such as requests iter_content() to read the stream and then writes the downloaded data into a Mongo DB. It handles authentication and it has some logic for re-authentication if there are auth issues and pause + retry if there are certain types of errors.

The script (downloader.py) works well but I would like to be able to "manage" it through a web interface. Specifically, I want to be able to start the script, see if the script is running, stop the script and re-start it. Making the front end for the web interface is trivial using some front end JS or anything that makes API calls but how to encapsulate the downloader.py script into something that exposes a REST API which allows me to control the script? e.g. a manage.py

manage.py could be a Flask app to handle REST API calls from the web interface and then there are two options:

  1. Merge them...

I could import code from downloader.py into the manage.py Flask App and use Redis/Celery to manage asynchronous long-running tasks

or

  1. Keep them separate...

I could have downloader.py periodically poll the Flask API manage.py with status updates or to check for requests for termination. manage.py could launch downloader.py as a process if requested to using the API.

Some guidance as to which pattern might work better would be great. Thanks for your help.



from Managing a long-running API listener using Python / Flask

No comments:

Post a Comment