NGINX, handling file uploads securely without touching your application.

· by author · Read in about 3 min · (458 words)

Keep the processing of files out of your application while still authenticating the upload!

In this post I will explain how to perform file uploads entirely at the NGINX level only passing off the file handler once the file has been written to disk. It used to be that you needed to install a third party plugin to allow NGINX to handle file uploads. It had it’s pros and cons but fell out of development and current versions of NGINX no longer support it.

I’ve read countless tutorials that show how to have python write a byte stream to disk. For example the following is an often recommended way of handling uploads in Tornado:

import tornado
import tornado.ioloop
import tornado.web
import os, uuid

__UPLOADS__ = "uploads/"

class Upload(tornado.web.RequestHandler):
    def post(self):
        fileinfo = self.request.files['filearg'][0]
        fname = fileinfo['filename']
        extn = os.path.splitext(fname)[1]
        cname = str(uuid.uuid4()) + extn
        fh = open(__UPLOADS__ + cname, 'w')
        fh.write(fileinfo['body'])
        self.finish(cname + ' is uploaded!! Check %s folder' %__UPLOADS__)

And Django does file uploads like this:

from django.http import HttpResponseRedirect
from django.shortcuts import render
from .forms import UploadFileForm

# Imaginary function to handle an uploaded file.
from somewhere import handle_uploaded_file

def upload_file(request):
    if request.method == 'POST':
        form = UploadFileForm(request.POST, request.FILES)
        if form.is_valid():
            handle_uploaded_file(request.FILES['file'])
            return HttpResponseRedirect('/success/url/')
    else:
        form = UploadFileForm()
    return render(request, 'upload.html', {'form': form})

The Django docs provide the following as a common way you might handle an uploaded file:

def handle_uploaded_file(f):
    with open('some/file/name.txt', 'wb+') as destination:
        for chunk in f.chunks():
            destination.write(chunk)

But all of these methods result in locking threads in your application. Instead we can have NGINX write the file to disk and signal the application when it’s done being written. We can even have NGINX authenticate the request before writing the file to disk to prevent the endpoint from being abused.

server {
...
    location ~* /upload/ {
        client_body_temp_path      /tmp/; #Path where you want NGINX to put the file
        client_body_in_file_only   on;
        client_body_buffer_size    128K;

        proxy_pass_request_headers on;
        proxy_set_header           X-FILE $request_body_file;
        proxy_pass_request_body    off;
        proxy_set_header Content-Length "";
        proxy_redirect             off;
        proxy_pass http://resource-upload_frontends;
    }
...
}

But WAIT! I said you could do this SECURELY! and so far all you’ve done is open the flood gates to allow anyone who makes and http request to upload file data! Using the NGINX auth_request feature we can pass the request to an authentication location.

NOTE This module is not built by default, it should be enabled with the –with-http_auth_request_module configuration parameter.

location = /upload {
  auth_request               /upload/authenticate;
  ...
}

location = /upload/authenticate {
  internal;
  proxy_set_body             off;
  proxy_pass                 http://backend;
}

P.S. client_body_in_file_only incompatible with multi-part data upload, so you can use it via XMLHttpRequest2 (without multi-part) and binary data upload only

curl --data-binary '@file' http://localhost/upload

This method is preferred to use with native mobile applications that handle big file upload all the time.