Sunday, 25 August 2019

Python 3 Flask and NGINX Streaming File Upload through Gunicorn

I am using Flask ( 1.0.2 ) with Python 3.7 on Linux ( 4.9.13 ) with Gunicorn ( 19.9.0 ) proxied by NGINX ( 1.15.7 ).

I can successfully upload a large ( 1.2GB ) file to my Flask server with the following code. However, the entire file is buffered in RAM before being written to disk with the file.save() function shown below. I've tried Googling and found various posts allegedly streaming the file to disk instead of buffering in RAM, but I have been unable to get their methods to work.

How can I get the file to stream directly to disk instead of buffering in RAM first and then to disk? Thanks.

Here is how I launch gunicorn:

gunicorn --workers=4 --threads=8 --bind localhost:8000 StartFlaskServer:app

Here is my code for the Flask Endpoint:

@app.route("/firmware_update", methods=["GET", "POST"])
def upload_video():

    if request.method == "POST":

        # check if the post request has the file part
        if 'file' not in request.files:
            flash('No file part')

            return make_response(jsonify({"message": "No File Part Specified!"}), 500)

        file = request.files['file']

        # if user does not select file, browser also
        # submit an empty part without filename
        if file.filename == '':
            flash('No selected file')

            return make_response(jsonify({"message": "No Selected File!"}), 500)

        if not allowed_file(file.filename):

            suffix = file.filename.rsplit('.', 1)[1].lower()
            return make_response(jsonify({"message": "Filetypes of %s not accepted ( Must be of type: %s )!"%(suffix,ALLOWED_EXTENSIONS)}), 500)

        fileFullPath = os.path.join(app.config['UPLOAD_FOLDER'], secure_filename(request.files['file'].filename))

        # Why does this not work? Cannot convert file to stream type with no buffering in NGINX?
        # with open(fileFullPath, "wb") as f:
        #    chunk_size = 4096
        #    while True:
        #        chunk = request.stream.read(chunk_size)
        #        print("Flask Writing Chunk: %s"%(len(chunk)))
        #        if len(chunk) == 0:
        #            break
        #        print("Wrote this much: %s"%(f.write(chunk)))

        file.save(fileFullPath)

        return make_response(jsonify({"message": "File uploaded"}), 200)

    return render_template("upload_firmware_bundle.html")

Here is my NGINX endpoint configuration:

   # Proxy upload
   location /firmware_update {

       # Proxy config
       proxy_pass http://localhost:8000;
       proxy_set_header Host $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

       # Do not buffer body
       client_max_body_size 0;
       proxy_http_version 1.1;
       proxy_buffering off;
       proxy_request_buffering off;

   }



from Python 3 Flask and NGINX Streaming File Upload through Gunicorn

No comments:

Post a Comment