You may have noticed over the past several months many commits related to uploads in Discourse core. This has been part of an overall effort within core to replace usages of jQuery file upload from our codebase, as part of an even broader overall effort to replace usages of jQuery itself from our codebase. jQuery file uploader is a very old project, and has been part of Discourse core since almost the beginning. I think I’ve used it for my whole career in other projects as well. But it’s time to retire Ol’ Reliable:
We have replaced jQuery file upload (and will soon also replace another library, resumable.js) with a library called Uppy. This is a much more modern upload library, that is easily extendable with plugins and is able to handle all the different workflows we throw at it. Importantly, this allows us to do direct multipart uploads to S3 from the Discourse client, rather than having to send large files through our API.
The composer now uses Uppy for all uploads, and many other places throughout the app use it (avatar uploads, profile background uploads, etc.). The last few holdouts will be gone over the next few weeks. This should be a largely invisible change for most users, but plugin and theme component authors will need to make some changes.
Plugin API
Preprocessors
All upload pre-processors now need to be written as an Uppy plugin. These plugins are quite straightforward to write, and use a simple promise-based workflow. An upload pre-processor can modify a file or add metadata to it before Uppy uploads it to S3 or to the /uploads.json endpoint. We already have several pre-processors in core that you can use as reference when writing your own:
- discourse/uppy-checksum-plugin.js at 6662101208089def86ed18a81ac90d1c52670569 · discourse/discourse · GitHub
- discourse/uppy-media-optimization-plugin.js at 6662101208089def86ed18a81ac90d1c52670569 · discourse/discourse · GitHub
Upload pre-processors for the composer are registered via api.addComposerUploadPreProcessor using the plugin API:
Upload Handlers
Upload handlers are not written as Uppy plugins; they still work as they always have with a minor change. Now when a file matches an extension registered to an upload handler, all of the files that match will be sent through at once. Previously only one file at a time would be sent through to the upload handler, and an array is sent through instead now:
Files handled by an upload handler will not be further processed in the Uppy upload pipeline. Pre-processors are run before upload handlers are invoked.
S3 Multipart Direct Uploads
Earlier I mentioned that our use of Uppy also allows us to do direct multipart uploads to S3 from the UI. To enable this feature, you need to set the enable_direct_s3_uploads site setting to true.
If you are hosted with us, the relevant S3 permissions are already applied to your bucket. However if you are self hosting, there are several permissions and CORS rules that must be set up on your bucket for this to work.
For the CORS rules, you need only to run the s3:ensure_cors_rules rake task with rake s3:ensure_cors_rules. It will add the following rules to your bucket as long as you have S3:GetBucketCors and S3:PutBucketCors permissions enabled for whatever access and secret key you have set up for your S3 credentials on your Discourse instance.
{
"AllowedHeaders": [
"Authorization",
"Content-Disposition",
"Content-Type"
],
"AllowedMethods": [
"GET",
"HEAD",
"PUT"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": [
"ETag"
],
"MaxAgeSeconds": 3000
}
For the permissions, you will need to have the following permissions enabled for whatever access and secret key you have set up for your S3 credentials on your Discourse instance.
{
"Sid": "YourSid",
"Effect": "Allow",
"Action": [
"s3:PutObjectVersionAcl",
"s3:PutObjectAcl",
"s3:PutObject",
"s3:GetObjectAcl",
"s3:GetObject",
"s3:DeleteObject",
"s3:CreateMultipartUpload",
"s3:CompleteMultipartUpload",
"s3:AbortMultipartUpload"
],
"Resource": [
"YOUR_RESOURCE"
]
}
This has been a process several long months in the making and we are not done yet! I will post in this topic when we are about to remove jQuery file uploader and resumable.js completely from Discourse core. Let me know if you have any questions about what I’ve posted here!
