restful python upload file to server

Question

I have a server running flask and some api open for client to upload file.

The client is having problem when uploading big file(>100MB).

At first, I tried urllib2 using the following code at client:

def uploadFile(url, params, user, token):
    try:
        from poster.encode import multipart_encode
        from poster.streaminghttp import register_openers
        import urllib2
        file_path = params["filepath"]
        register_openers()
        tempparams = {'file': open(file_path, "rb")}
        f = open(file_path,'rb')
        tempparams = {'file': open(file_path, "rb")}
        finalparams = dict(params, **tempparams)
        datagen, headers = multipart_encode(finalparams)
        request = urllib2.Request(url, datagen, headers)
        request.add_header("Authorization", "Basic %s" % base64string)
        result = urllib2.urlopen(request)
        resultData =  result.read()
        resultJson = json.loads(resultData)
        return resultJson
    except Exception as e:
        return {"error": e, "error_code": 503}

At server:

def upload_file_old():
    fileobj =request.files["file"]

    try:
        fileobj.save(webserver_filepath)
    except Exception as e:
        logger.info(e)
    .....
    return some_result, 201

But the above code will lead to the following error when uploading big file at client with server recording no log at all.

urlopen error [errno 10054]

Then I tried using requests toolbelt to send file since many post at stack overflow suggest using request other than using urllib.

At client:

def uploadFile(url, params, user, token):
    try:
        import requests
        from requests_toolbelt import MultipartEncoder
        file_path = params["filepath"]
        m = MultipartEncoder(fields={
        'file':(params["name"],open(file_path,'rb'),"application/octet-stream" )
        }) 
        r = requests.post(url, data=m, headers={'Content-Type': m.content_type}, auth=(user, token))
        return r.json()
    except Exception as e:
        return {"error": e, "error_code": 503}

At server:

def upload_file_old():
    fileobj =request.files["file"]

    try:
        fileobj.save(webserver_filepath)
    except Exception as e:
        logger.info(e)
    .....
    return some_result, 201

Again, sending small file works fine while big file failed for the above code, the error output is as foloowing:

 ('Connection aborted.', error(104, 'Connection reset by peer'))

What would be the correct way to upload big file in a restful http request using python?


Show source
| rest   | python   | python-requests   | urllib2   | urllib   2016-10-24 15:10 0 Answers

Answers ( 0 )

◀ Go back