Error with very large file uploading

bug
1

Hi! :slight_smile:

We are testing Discourse to use as a forum application for our community, and so far it looks quite great.

In addition to normal forum functionality, we have some specific needs.

From time to time we need to upload files to the forum, these are about 10MB-200MB in size (audio recordings).

Unfortunately we are now experiencing some difficulties with the file uploading. If the file is about 50MB or less it seems to upload fine, but something like 80MB and larger gives "Sorry, there was an error uploading that file. Please try again.” error to the screen.

Our Discourse version is v2.0.0.beta2 +115

We have added the parameter upload_size: 150m into app.yml (and ran ”./launcher rebuild app”), and changed the ”max attachment size kb” from /admin settings to ”150000”.

i can’t see anything error related in production.log and there is nothing in production_errors.log, but nginx/error.log shows this:

2018/02/11 16:53:35 [error] 70#70: *8891 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xx.xx.xx.xx, server: _, request: “POST /uploads.json?client_id=b4952f07d4d949e4a87c176473fbb5d6&authenticity_token=e%2F%2FRKI3YDCVCixp76YaIqpLTib6zdrO0vZBAYlHPVcZVBJUx%2BEy2juQ5xMrTeUPZoyX0uJzTPW7pCey47YRvwQ%3D%3D HTTP/1.1”, upstream: “http://127.0.0.1:3000/uploads.json?client_id=b4952f07d4d949e4a87c176473fbb5d6&authenticity_token=e%2F%2FRKI3YDCVCixp76YaIqpLTib6zdrO0vZBAYlHPVcZVBJUx%2BEy2juQ5xMrTeUPZoyX0uJzTPW7pCey47YRvwQ%3D%3D”, host: “server”, referrer: “http://server/c/test-category”

Around the same time as the upload fails, the file unicorn.stderr gives this error:

E, [2018-02-11T16:53:35.331265 #73] ERROR -- : worker=0 PID:6403 timeout (31s > 30s), killing

Could there be a some kind of timeout/limit issue somewhere?

i’ve been trying to find a solution to this, but it seems i can’t get anywhere.

Our test server is a VPS with 1GB of memory (and a 3GB swap file).

We would not want to give up on Discourse just yet, so any help is truly appreciated! :grinning:

2

There is a topic that describes how to modify the nginx container to show larger uploads. I did it once for a group that wanted to discuss videos.

3

Thank you for the reply :slight_smile:
Do you mean there is something else that needs to be modified too? In addition to changing the upload limit on admin panel and changing client_max_body_size in nginx.conf (nginx.conf was modified by adding upload_size parameter to app.yml)?

I have been trying to look for this topic you mentioned, but all i can find are references to these modifications above, which we have already done. Perhaps i’m missing something here?

Thanks in advance for all the help!

2 Likes
4

Oops. I missed that you’d changed the upload size in app.yml. That’s what I was talking about. Sorry.

5

You are forgiven :grin:

So, we are still stuck with this. Is there anything else we could check in order to figure this out? Some additional log files? Perhaps a way to increase debug level and see what’s going on in detail?

6

As a side note, the server is running quite low on memory and the output of free -m often looks something like this:

                   total        used        free      shared  buff/cache   available
     Mem:            992         735          86          22         170          83
     Swap:          3068         496        2572

Could this perhaps be a performance issue due to lack of memory?

1 Like
7

Yes. Does free -h show 1GB? ./discourse-setup won’t install if it sees less than 1GB of RAM. I’m not sure how and whether free rounds, though. Is this a standard install?

Where are you hosted? But increasing RAM is definitely a good idea. If you’ve got a 150GB upload, that’s a pretty substantial part of your available RAM.

I’d bump to at least 2GB of RAM to see if that solves your problem.

1 Like
8

Yes, this is just a test server (VPS at Transip) to see if Discourse suits our needs. So it has minimum requirements (1GB RAM). We have not yet decided our final hosting provider.

We followed the installation instructions described here. So it should be pretty standard.

To clarify: the upload is 150MB, not 150GB.

From the original post:

So it’s megabytes, not gigabytes :slight_smile:

1 Like
9

Yeah. I mean MB, but 150MB is a sizable part of 1GB.

Is the error you’re getting “Sorry, there was an error uploading that file. Please try again.?”

I confirm that I just did a new install and upped the RAM to 4GB (and rebuilt after adjusting the memory settings). Upoading a 40MB file works fine, but an 83MB file gets that error.

Seems like this might be a bug.

1 Like
10

Yes, exactly that.
Along with the log file findings shown in the first post :slight_smile:

1 Like
11

If we can repro a bug here is there anything we can do to address it @zogstrip?

12

Hi again :slight_smile:
Are there any news regarding this issue?

14

Sorry hadn’t had a chance to look at it yet. Added to my list.

2 Likes
15

Okay - thank you!
Let us know how it goes :slight_smile:

16

I’m not sure there’s much to do here :confused:

If uploading a file monopolizes a unicorn worker for more than 30 seconds, then you should use a proper file hosting service instead? There’s a good reason we have a “short” timeout :wink: (hint DoS)

3 Likes