Upload_size in app.yml not working


(Vikas Kedia) #1

Steps followed:

  1. In app.yml I have set:
    params:
    upload_size: 1000m

  1. I did docker exec to enter the discourse docker and confirmed that /etc/nginx/conf.d/discourse.conf has:

client_max_body_size 1000m ;

  1. I am trying to upload a file of size 165 MB. I get the following error message:

image

  1. This error comes after the browser finishes uploading 100% of the file as show below the post textarea.

  2. I am on the discourse commit id from yesterday.


(Jay Pfaffman) #2

Sounds like this: Error with very large file uploading


(Frederik) #3

It might be stupid as I know nothing about nginx config: but the m should not be in capital?

1000M

instead of

1000m


(Jay Pfaffman) #4

Nope. I just tried that and uploaded a 460MB file. It kept going to 100% and then gave the same error.


(Frederik) #5

It was worth a try :smiley:


(Jeff Atwood) #6

I believe basic web uploading through NGINX does require some multiple of free server memory times the actual file size though. It is not sophisticated FTP or anything like that.


(Jay Pfaffman) #7

I am pretty sure that in the past I’ve uploaded files larger than 100MB, and I don’t think that the droplet had more than 1GB. But it was a long time ago. Maybe it was only 50MB.


(Vikas Kedia) #8

Two more data points in addition to the 5 above:

  1. Server has 96gb ram. So it can not be a issue with memory not being free as suggested by @codinghorror

  2. Discourse runs behind a nginx proxy and I am able to upload a file to another software running behind the same nginx proxy.

If a dev wants to take a look at this error I can give access to my discourse instance, to help in troubleshooting.


(Sam Saffron) #9

Step one, how big does a file need to be to fail, try 1mb 10mb 100mb 50mb and so on to find the spot that stops working.

moved to #support cause this is a very non-traditional setup


(Jay Pfaffman) #10

I got the same error with a vanilla install. I’ve not done a test of what size first fails, though.


(Vikas Kedia) #11

Some more data points to the above 7 to help debug:

  1. Tried with a tradition vanilla install and face the same error.

@sam maybe a good idea to move it back to bug :slight_smile:


(Régis Hanol) #12

It could be a timeout issue. @vikaskedia do you know how long it takes to get the error? Is it approximately always the same time?


(Vikas Kedia) #13

The file finishes uploaded it says 100% and then immediately after gives the error. The screenshot of the error is posted in point 3 of the first post.

I will create a video and post it soon.


(Jeff Wong) #14

@zogstrip was asking about literal time in seconds/minutes, not relatively “when” in the process of uploading. If it is consistently failing after the same time in seconds regardless of filesize, that could be a sign that it is hitting a timeout.


(Jay Pfaffman) #15

It seems that if fails when the file “finishes” uploading, not at some fixed time in the upload process.


(Sam Saffron) #16

You got to answer my question though

There is a threshold where this stops working…

Works for 92.3mb fails for 92.4mb is the kind of post I am looking for


(Jay Pfaffman) #17

It looks like it’s failing during the time between when the web interface thinks the file is uploaded and when it gets processed and put in to the composer.

It’s somewhere around 75MB

  • 74773504 bytes (75 MB, 71 MiB) failed
  • 74772480 bytes succeeded

I’m sure that the above file failed, but then in uploads I see this:

14962217  73024 -rw-r--r--   1 pfaffman www-data 74776576 Feb 28 21:03 ./2X/4/488897cf90584109000499ce74b76f69ce7b9dd4.txt
 14962243  73024 -rw-r--r--   1 pfaffman www-data 74773504 Feb 28 21:22 ./2X/b/b25d8c8191a0e9686d106a039f522760b63329ff.txt
 14962216  70024 -rw-r--r--   1 pfaffman www-data 71704576 Feb 28 20:54 ./2X/b/b6d1f8c831a9bfd671c4839efbcb7e84d1a4e1fe.txt

(Sam Saffron) #18

Thing is, to support enormous uploads we need a completely different scheme. Resumable uploads like we have for backups.

This is a pretty large feature request for a few edge case users. Recommend you open that feature request if you wish.

Very little value adding a box for 999mb if shutting the laptop during upload means you have to start from scratch.


(Jay Pfaffman) #19

Sounds right. I had one potential user who wanted to use Discourse to discuss videos that needed to be on a non-public server. I think that for that case I’d done a proof of concept with videos of several minutes, but the details are hazy now.

They didn’t respond to my email anyway. :expressionless:


(Jeff Atwood) #20

“I regularly need to upload 100 - 500 megabyte files” should be a HUGE red flag :triangular_flag_on_post:

Discourse is many things but it is not a file storage repository, or a video sharing service…