I am using Flask ( 1.0.2 ) with Python 3.7 on Linux ( 4.9.13 ) with Gunicorn ( 19.9.0 ) proxied by NGINX ( 1.15.7 ).
I can successfully upload a large ( 1.2GB ) file to my Flask server with the following code. However, the entire file is buffered in RAM before being written to disk with the file.save() function shown below. I've tried Googling and found various posts allegedly streaming the file to disk instead of buffering in RAM, but I have been unable to get their methods to work.
How can I get the file to stream directly to disk instead of buffering in RAM first and then to disk? Thanks.
Here is how I launch gunicorn:
gunicorn --workers=4 --threads=8 --bind localhost:8000 StartFlaskServer:app
Here is my code for the Flask Endpoint:
@app.route("/firmware_update", methods=["GET", "POST"])
def upload_video():
if request.method == "POST":
# check if the post request has the file part
if 'file' not in request.files:
flash('No file part')
return make_response(jsonify({"message": "No File Part Specified!"}), 500)
file = request.files['file']
# if user does not select file, browser also
# submit an empty part without filename
if file.filename == '':
flash('No selected file')
return make_response(jsonify({"message": "No Selected File!"}), 500)
if not allowed_file(file.filename):
suffix = file.filename.rsplit('.', 1)[1].lower()
return make_response(jsonify({"message": "Filetypes of %s not accepted ( Must be of type: %s )!"%(suffix,ALLOWED_EXTENSIONS)}), 500)
fileFullPath = os.path.join(app.config['UPLOAD_FOLDER'], secure_filename(request.files['file'].filename))
# Why does this not work? Cannot convert file to stream type with no buffering in NGINX?
# with open(fileFullPath, "wb") as f:
# chunk_size = 4096
# while True:
# chunk = request.stream.read(chunk_size)
# print("Flask Writing Chunk: %s"%(len(chunk)))
# if len(chunk) == 0:
# break
# print("Wrote this much: %s"%(f.write(chunk)))
file.save(fileFullPath)
return make_response(jsonify({"message": "File uploaded"}), 200)
return render_template("upload_firmware_bundle.html")
Here is my NGINX endpoint configuration:
# Proxy upload
location /firmware_update {
# Proxy config
proxy_pass http://localhost:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Do not buffer body
client_max_body_size 0;
proxy_http_version 1.1;
proxy_buffering off;
proxy_request_buffering off;
}
from Python 3 Flask and NGINX Streaming File Upload through Gunicorn
No comments:
Post a Comment