Inquiry About Discourse’s Support for Amazon S3 Storage

[wenqin]

I’d like to ask about a technical issue regarding Discourse’s support for Amazon S3 storage.

I came across a forum post from as early as 2013 that already discussed Discourse’s S3 support. I also found a few threads related to CDN offloading, but there doesn’t seem to be much detailed information on this topic.

I’ve built a Discourse forum myself and have been very persistent about enabling video uploads. I’ve come up with many ideas, but since I’m not a programmer, it’s been difficult to put them into practice—I always need help from others to make progress bit by bit.

Discourse itself supports small video uploads, and I’ve already enabled that feature. However, the architecture of Discourse is not really suited for handling video uploads efficiently. After a lot of research, I discovered that it’s actually possible to use Amazon S3 storage together with CDN offloading, or alternatively, S3 storage with a proxy cache setup.

Indeed, I’ve already asked a friend to help me configure S3 storage and proxy caching. (For clarification, I didn’t consider using a CDN because of cost concerns—Amazon S3’s CDN service can be quite expensive.)

Now I’m facing a problem: during testing, even though S3 storage has been properly configured, and theoretically all uploads from the forum should go directly to S3, video files still cannot be uploaded to S3 storage.

Could anyone please explain what might be misconfigured or what could be causing this issue?

thanks

[RGJ (Richard - Communiteq)]

@wenqin

video files still cannot be uploaded to S3 storage.

What happens instead? Do you get an error message?

You might also want to check the settings “max attachment size kb” and “authorized extensions”, both in Site Settings - Files.

Please note that even with CDN and S3, this setup is still not very suitable for videos, since there is no transcoding to offer the best fit for the clients bandwidth and resolution.

[wenqin]

Thank you very much for your reply. Here’s the situation:
Right now, images from my forum can be successfully uploaded to S3, but videos cannot. It’s really confusing to me.

Although I understand that S3 might still not be ideal for handling videos, the main content of the forum is text and images. The video feature is just to meet the needs of some users, and the videos themselves won’t be very large. Thank you very much for the reminder.

[RGJ (Richard - Communiteq)]

You forgot to answer my question

@wenqin

but videos cannot. It’s really confusing to me.

@RGJ

What happens instead? Do you get an error message?

[pfaffman (Jay Pfaffman)]

@wenqin

I didn’t consider using a CDN because of cost concerns—Amazon S3’s CDN service can be quite expensive.

Amazon’s CDN is cheaper than egress from S3 buckets. You definitely want a CDN in front of your S3 bucket.

I suspect that the error you haven’t told us about has to do with videos being too big. There are settings for changing that, but it might be better to enable Enable direct S3 uploads .

[wenqin]

When I try to upload a video to my forum, it now directly shows the message:

“Sorry, that file is too big (maximum size is 50 MB). Why not upload your large file to a cloud sharing service, then paste the link?”

However, my video itself is only 10 MB. Currently, I’m using S3 storage with CloudFront CDN for delivery.

I find troubleshooting very difficult. Could you provide the correct procedure assuming no prior operations were done? I will ask my friend to follow the correct steps and try again.

[wenqin]

Hello, thank you for your reply.

I have already asked my friend to switch from the S+ storage with proxy caching method to the S3 storage + CloudFront CDN delivery method. However, it seems to have failed, and I don’t know the reason yet. So I was wondering if you could provide the exact execution commands so that my friend can try again.

In addition, I am really persistent about having video functionality on my forum for the convenience of my users. So I came up with a new idea and would like to know if it is feasible—I really hope to get your opinion.

First, I should explain that I have absolutely no knowledge of coding or programming. My university major was Japanese, so apart from learning the Japanese language, I truly have no understanding of code. If some of my ideas seem naïve at first glance, please forgive me.

Thanks to the server I rented for my forum shutting down, I had to invest effort in rebuilding it, which allowed me to learn the key points of S3 storage and CloudFront CDN. Then, I applied this knowledge and wondered if there are other storage and CDN service providers. I consulted ChatGPT and was glad to get the answers I wanted.

Actually, I had always wanted to build a video website, but I quickly gave up because of the hard drive costs. Now that I know about the new solution, I’ve rekindled the idea of building a video website to support the Discourse forum videos.

Based on GPT’s recommendation, I learned that I can use either PeerTube or MediaCMS for the video website system—they are both excellent.

I currently have two points of doubt:

First point: Both video website systems can support Backblaze B2 storage with Cloudflare CDN delivery. Since videos are not stored on my own hard drive, the video website’s playback already calls the Backblaze B2 API. What I want is for the forum to be connected to the video website, not by posting links, but by using the forum’s upload button to directly call my MediaCMS (for example) video website as the source. This would mean two API calls:

1. The forum calls the MediaCMS video website.
2. The MediaCMS video website calls Backblaze B2 storage.

I would like to know if such repeated API calls are technically feasible.

Second point: Whether the forum system and video website system can share the user account system, so that users can use the same account. This affects how the API calls would work.

• Option 1: As the administrator of the video website, I register an admin account linked to the forum. When the forum uploads a video via the API, it is technically the administrator uploading it to the MediaCMS website.
• Option 2: With a shared user account system, the video upload is performed by the user’s own forum account, and via the API it is synchronized to the MediaCMS website. Then the user can see the uploaded video under their own account on the video website.

I have consulted GPT about these approaches, and GPT generally gave positive feedback. However, unfortunately, in repeated use, I found that GPT can provide guidance and help with high-level decisions, but for detailed answers, it can sometimes be inaccurate. Since I have no coding knowledge myself, I’m not sure if these ideas are really feasible.

If possible, I would greatly appreciate it if you could point out which parts of these ideas are feasible and which might not be.

[pfaffman (Jay Pfaffman)]

@wenqin

I have already asked my friend to switch from the S+ storage with proxy caching method to the S3 storage + CloudFront CDN delivery method. However, it seems to have failed, and I don’t know the reason yet.

They should probably be participating in this topic, then. Did they follow Configure an S3 compatible object storage provider for uploads and put the setttings in the ymll file and add the part that pushes assets to S3?

@wenqin

“Sorry, that file is too big (maximum size is 50 MB). Why not upload your large file to a cloud sharing service, then paste the link?”

@wenqin

When I try to upload a video to my forum, it now directly shows the message:

“Sorry, that file is too big (maximum size is 50 MB). Why not upload your large file to a cloud sharing service, then paste the link?”

However, my video itself is only 10 MB. Currently, I’m using S3 storage with CloudFront CDN for delivery.

Maybe the browser somehow isn’t communicating the size appropriately.

@pfaffman

it might be better to enable Enable direct S3 uploads .

Did you try this?

[wenqin]

Hello. Actually, here’s the situation:
Videos smaller than 10MB can be uploaded to my own server, but the video files themselves cannot be uploaded to S3.
When uploading a video, the system automatically captures the first frame as the thumbnail, and the thumbnail image can be uploaded to S3.

However, for videos larger than 10MB, nothing can be uploaded at all.

[pfaffman (Jay Pfaffman)]

If your friend had followed Configure an S3 compatible object storage provider for uploads you’d not see enable_s3_uploads. It’s unclear whether setting those in the database rather than ENV is part of the problem.

That’s not the same as Enable direct S3 uploads . That’s “Enable direct S3 uploads”

You also need to change Max image size KB and Max attachment size KB

But I changed my max upload to 100MB and tried to uplaod a 37MB file and got the “file is too big” error, so you may be right that there is a bug. I’ve not updated the site where I tested in a while.

[Moin]

@pfaffman

But I changed my max upload to 100MB and tried to uplaod a 37MB file and got the “file is too big” error, so you may be right that there is a bug.

Did you change it in app.yml too?

Change the maximum attachment/upload size

This guide explains how to change the maximum attachment upload size for self-hosted Discourse instances. Required user level: System Admin The default attachment upload size limit is 10MB If you are self-hosting Discourse and need to allow uploads larger than 10 MB, follow the steps below. Changing the upload size Connect to your server: Use SSH to access your server’s command line interface. Edit the configuration file: Navigate to the configurati…

[wenqin]

Thank you very much for all your help.
He said he had followed the instructions in Configure an S3 compatible object storage provider for uploads, but it didn’t work.
However, just now, after he did the same as you suggested and changed the maximum upload file size to 100MB, the video was successfully uploaded to the S3 storage.
Thank you so much!

[pfaffman (Jay Pfaffman)]

@wenqin

He said he had followed the instructions in Configure an S3 compatible object storage provider for uploads, but it didn’t work.

Then he did it wrong and should have asked for help. He probably omitted the part that uploads assets to S3.

@wenqin

However, just now, after he did the same as you suggested and changed the maximum upload file size to 100MB, the video was successfully uploaded to the S3 storage.
Thank you so much!

Nice! There’s a topic somewhere about how to increase that 100MB limit as well.