Best Practices for Disk Space Management in Community w/ lots of Images

Product Support Self-hosting

AI-generated summary

A discussion on best practices for managing disk space in a community with many images is underway. keith1 wants to avoid exponential disk space growth as the community grows and is seeking advice on user uploads.

itsbhanusharma suggests serving static resources via AWS to offload data from the server. meglio points out that local copies of images would still reside on the server, but itsbhanusharma clarifies that rebuilding the container and offloading images to an S3 bucket can resolve this.

Regarding backups, itsbhanusharma notes that they will contain references to images stored on AWS, and zogstrip explains that images stored on S3 won’t be backed up by Discourse. However, vinothkannans assures that restoring a backup will maintain S3 configurations.

Other topics discussed include:

  • Storing only optimized images, as raised by Dmitry_Krasnoperov
  • Client-side image optimization, also mentioned by Dmitry_Krasnoperov
  • The availability of a script to downsize uploads, shared by jomaxro
  • The possibility of making image optimization available directly from the admin panel, proposed by sam
1

I’m running a very visual oriented community (www.realtimevfx.com) and have watched my disk space creep up over the past year (obviously). I’m about to do some cleanup (per this discussion “https://meta.discourse.org/t/low-on-disk-space-cleaning-up-old-docker-containers/15792/83”), but I wanted to start a general conversation to get people’s thoughts on the best way to approach this - If I can avoid it, I’d rather not have exponential disk space growth as the community grows.

Are there best practices / standards that I should implement regarding user uploads? Should I disable uploads entirely and have people only host through imgur, etc?

2

Serve static resources via AWS is the best You can do as it will offload a significant amount of less important data from the server.

3

Won’t the local copies of all uploaded images still reside on your server?

4

Rebuild Your container once the Images are offloaded to s3 bucket and then all those can be safely removed from your local server. system is supposed to automatically clean up the content once it is properly uploaded to the AWS cloud.

5

I see, didn’t know about it.
What about making a backup of a Discourse instance, it won’t contain those images, will it?

6

it will contain necessary references to those images and when You restore such a backup, it will link them to Your instance’s AWS s3 configuration and look for resources in the relative directories afaik but maybe someone from the discourse team can provide better details about how backups are handled in case of files.

@HAWK @sam can You provide some extra details about how backups handle the media files uploaded by users?

7

When images are stored on S3, they won’t be backed up by Discourse anymore. This would be pointless since most of the time, you’re using S3 because you don’t have enough space locally.

If you want to backup your images, I’m sure there are plenty of tools available to do that by talking to S3 directly :wink:

8

@zogstrip one Question though!
what if that backup is restored when the site is moved to a new host? will that break all the images or will it be as simple as configuring with the same AWS details again to restore all the media?

9

Backup will contain all your site settings. So new restored instance also will have the same S3 configuration.

10

I have several question about images storage/processing

  1. I have two folders
    /var/discourse/shared/standalone/uploads/default/original — 9GB in my case
    ../3X
    ../2X
    ../1X
    /var/discourse/shared/standalone/uploads/default/optimized — 4GB in my case
    ../3X
    ../2X
    ../1X

Is there way to keep only optimized images?

  1. Does folder
    /var/discourse/shared/standalone/postgres_data/base
    store the same images as folders above?

  2. Is there any Discourse settings for storing images in lower quality?

  3. Is there any client-side image optimization to make it lower size?

11

Original images are shown when their optimised counterparts within topics/replies are clicked.

13

이 @sam 관련해서 다른 의견 있으신가요? 이미지에 대한 최적화 rake 태스크가 있다는 걸 알고 있는데, @jomaxro님, 셀프 호스터들도 이걸 사용할 수 있나요?

14

레이크 태스크는 아니지만, 해당 스크립트는 셀프 호스터가 사용할 수 있도록 제공됩니다. https://github.com/discourse/discourse/blob/master/script/downsize_uploads.rb

사용하려면 다음을 실행하세요:

./launcher enter app
cd /var/www/discourse
RAILS_ENV=production bundle exec ruby script/downsize_uploads.rb
15

이것은 S3 백업에 대한 전체적인 설명이 더 이상 아닙니다.

16

관리자 패널에서 바로 사용할 수 있도록 하고 싶습니다. 다음 릴리스에 포함할까요, 아니면 그다음 릴리스에 포함할까요?