API upload via url inconsistent results

Dev
1

Hi All,

Looking for some insight into this one.
I am trying to get uploads to work via API and I keep getting the error

{"errors"=>["Sorry, you must provide a file to upload."]}

This happens when I try to upload a generated public url from canvas i.e.

https://instructure-uploads-apse2.s3.ap-southeast-2.amazonaws.com/account_24310000000000001/attachments/777/IT%20Helpdesk%20Support%20Duty%20Statement%205%20Nov%202015.pdf?response-content-disposition=inline%3B%20filename%3D%22IT%20Helpdesk%20Support%20Duty%20Statement%205%20Nov%202015.pdf%22%3B%20filename%2A%3DUTF-8%27%27IT%2520Helpdesk%2520Support%2520Duty%2520Statement%25205%2520Nov%25202015.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIZAVS3WMRR54DT4Q%2F20180328%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-Date=20180328T032334Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=d4407ec41af0ffbf0bff523dba1322a18b726939242383876ed92ab1798977ee

If I use another url i.e.

http://unec.edu.az/application/uploads/2014/12/pdf-sample.pdf

it seems to work.

I can definitely browse to the url via browser and display the pdf file.
What am I doing wrong here?

Thanks
Michael

2

It looks like that url isn’t browsable and has some permission locking on it. Your browser can access it because you are logged in to aws, but your Discourse instance cannot access it because it has not been authorized.

1 Like
3

The URL has an expiry set on it, that is probably why you can’t see it now. It is generated on the fly when using the Canvas API

It is difficult to show you but here is another URL that has just been generated. I can access it from incognito browser too.

https://instructure-uploads-apse2.s3.ap-southeast-2.amazonaws.com/account_24310000000000001/attachments/777/IT%20Helpdesk%20Support%20Duty%20Statement%205%20Nov%202015.pdf?response-content-disposition=inline%3B%20filename%3D%22IT%20Helpdesk%20Support%20Duty%20Statement%205%20Nov%202015.pdf%22%3B%20filename%2A%3DUTF-8%27%27IT%2520Helpdesk%2520Support%2520Duty%2520Statement%25205%2520Nov%25202015.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIZAVS3WMRR54DT4Q%2F20180328%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-Date=20180328T201440Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=a476c1046e6b356c3bee618247957aea3c1dce3f0f442391130515a030fb8016

I guess I am just trying to find out why these kinds of files are not able to be uploaded via Discourse API and others are.

4

Something is still up with that url. Chrome is just being fancy and loading it right away for you (handling the headers or something).

image
image.png1054Ă—629 49.2 KB

Once you are able to get it to work with curl -O <url> it should work in Discourse.

2 Likes
5

When using curl -O <url> It kept giving errors: Warning: Failed to create the file
But I can get curl -0 <url> -o filename.pdf to work.
Can discourse use other curl command lines? Or can I somehow get it to perform the same download as curl -0 <url> -o filename

Thanks

6

I’m unable to get curl -0 <url> -o filename.pdf to work. Can you verify that the filename.pdf is the actual pdf document and not just some xml error.

Also -0 just changes it to “Use HTTP 1.0 (H)” where as -O saves it as a file instead of trying to render the pdf in the console.

The Discourse app doesn’t actually use curl to download the file, but it is just a good way to test anonymous downloads of files and when the discourse app uses ruby to download the file it does do something similar to curl.

In order to determine what the issue is here can you provide some more info on how your s3 bucket file permissions are setup? How do you generate all the extra url parameters to allow “anonymous” access to a private file?

Another option might be to use curl or whatever your using to access the discourse api to download the file or use the aws-sdk to download the file as an authenticated user and then upload the actual file to the discourse api instead of just a url.

Also once you get this all working you might want to turn on this setting in Discourse “prevent anons from downloading files”.

3 Likes
7

Thanks for the tip.
I ended up downloading the file to local storage and then uploading to Discourse.
A bit hacky but works well.

Thanks

1 Like