A new era for file uploads in Discourse

You may have noticed over the past several months many commits related to uploads in Discourse core. This has been part of an overall effort within core to replace usages of jQuery file upload from our codebase, as part of an even broader overall effort to replace usages of jQuery itself from our codebase. jQuery file uploader is a very old project, and has been part of Discourse core since almost the beginning. I think I’ve used it for my whole career in other projects as well. But it’s time to retire Ol’ Reliable:

We have replaced jQuery file upload (and will soon also replace another library, resumable.js) with a library called Uppy. This is a much more modern upload library, that is easily extendable with plugins and is able to handle all the different workflows we throw at it. Importantly, this allows us to do direct multipart uploads to S3 from the Discourse client, rather than having to send large files through our API.

The composer now uses Uppy for all uploads, and many other places throughout the app use it (avatar uploads, profile background uploads, etc.). The last few holdouts will be gone over the next few weeks. This should be a largely invisible change for most users, but plugin and theme component authors will need to make some changes.

Plugin API

Preprocessors

All upload pre-processors now need to be written as an Uppy plugin. These plugins are quite straightforward to write, and use a simple promise-based workflow. An upload pre-processor can modify a file or add metadata to it before Uppy uploads it to S3 or to the /uploads.json endpoint. We already have several pre-processors in core that you can use as reference when writing your own:

Upload pre-processors for the composer are registered via api.addComposerUploadPreProcessor using the plugin API:

Upload Handlers

Upload handlers are not written as Uppy plugins; they still work as they always have with a minor change. Now when a file matches an extension registered to an upload handler, all of the files that match will be sent through at once. Previously only one file at a time would be sent through to the upload handler, and an array is sent through instead now:

Files handled by an upload handler will not be further processed in the Uppy upload pipeline. Pre-processors are run before upload handlers are invoked.

S3 Multipart Direct Uploads

Earlier I mentioned that our use of Uppy also allows us to do direct multipart uploads to S3 from the UI. To enable this feature, you need to set the enable_direct_s3_uploads site setting to true.

If you are hosted with us, the relevant S3 permissions are already applied to your bucket. However if you are self hosting, there are several permissions and CORS rules that must be set up on your bucket for this to work.

For the CORS rules, you need only to run the s3:ensure_cors_rules rake task with rake s3:ensure_cors_rules. It will add the following rules to your bucket as long as you have S3:GetBucketCors and S3:PutBucketCors permissions enabled for whatever access and secret key you have set up for your S3 credentials on your Discourse instance.

{
  "AllowedHeaders": [
    "Authorization",
    "Content-Disposition",
    "Content-Type"
  ],
  "AllowedMethods": [
    "GET",
    "HEAD",
    "PUT"
  ],
  "AllowedOrigins": [
    "*"
  ],
  "ExposeHeaders": [
    "ETag"
  ],
  "MaxAgeSeconds": 3000
}

For the permissions, you will need to have the following permissions enabled for whatever access and secret key you have set up for your S3 credentials on your Discourse instance.

{
    "Sid": "YourSid",
    "Effect": "Allow",
    "Action": [
        "s3:PutObjectVersionAcl",
        "s3:PutObjectAcl",
        "s3:PutObject",
        "s3:GetObjectAcl",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:CreateMultipartUpload",
        "s3:CompleteMultipartUpload",
        "s3:AbortMultipartUpload"
    ],
    "Resource": [
        "YOUR_RESOURCE"
    ]
}

This has been a process several long months in the making and we are not done yet! I will post in this topic when we are about to remove jQuery file uploader and resumable.js completely from Discourse core. Let me know if you have any questions about what I’ve posted here!

47 Likes

5 posts were split to a new topic: Uppy not working on Firefox 68

itsgoneitsdone

With these two commits, jQuery file uploader and resumable.js are no longer part of Discourse core:

I’ve done my best to remove all references to this and our old UploadMixin in all of the plugins that we know about, but there may have been some I missed or am unaware of. Never fear, the migration process is easy. 99% of use cases can just use our new UppyUploadMixin as a drop-in replacement with very minimal changes needed. For an example take a look here:

For the other 1%, you can create an instance of Uppy and hook into the events directly. For an example take a look here:

I’ve also covered the plugin changes in the OP of this topic. We still have a few weeks until our next release, so if anyone has any issues please report them here. It’s been a wild ride! :roller_coaster:

14 Likes

On another note, the API documentation has now been updated with the new Upload endpoints. Go to Discourse API Docs to see.

(cc @mattdm, you may be interested in this)

6 Likes

After enabling direct S3 upload, we are getting reports from users in China being unable to upload images — it gets stuck at 0% and times out.

The first thought could be that S3 may be blocked in China, but we know for a fact it’s not the case — at least not fully: our users from China can see the images stored in S3 just fine (bucket in eu-central-1 region in our case). But, somehow, only the upload does not seem to work.

This is hard to debug while not being behind the GFW, but some of our users in China mentioned that one perhaps relevant difference seems to be that images are being loaded using the dual-stack endpoint, but the upload uses the regular (IPv4 only) endpoint (bucket-name.s3.dualstack.eu-central-1.amazonaws.com vs bucket-name.s3.eu-central-1.amazonaws.com). From some tests, we see that indeed that seems to be the case but I’m not sure if that’s intended or required for upload purpose.

More telling perhaps is that some reported that by adding an IP resolved from the dualstack hostname to their hosts file (for the non-dualstack name hostname) fully overcome the problem and they were able to upload with just that change.

Not sure if the Discourse team has someone located in China that can help debug this better?

4 Likes