My backup is getting bigger and bigger, and the bulk of it is uploaded images.
Most of the images posted on the forum are of things like screengrabs or other images where the highest quality is not critical. Really large images are rarely needed too (users are often lazy and don’t always resize before uploading).
Anyone else think we need some sort of option when it comes to the size of uploaded images? Perhaps we can set dimensions in the ACP and images are resized and at a quality level we specify?
Or maybe some way to backup images and forum data separately?
You can backup the database without files when you do a manual backup (not sure about auto)
If you have a lot of images, I’d recommend that you setup Discourse to store images on Amazon S3. You’ll likely better performance along with fault tolerance (if you want it) in the mix.
I think we’d need an automated way to back up just data - and somewhere accessible on the server so we could rsync the uploads directory off server for a complete back up.
I still think an option to allow reduction of quality size of images would be handy - there’s rarely a need for a forum to host large high quality images, especially of things like screengrabs.
I think that’s a fair analysis. Never hurts to keep things efficient and nimble. Maybe there there is something that could be put in between S3 or insert storage area here (trying to think of ways to do this without writing plugin or other code for Discourse).
Wordpress has some options that could perhaps be refactored in some way. I’m sure theres some command line tool you could throw in cronjobs tolook at images and resize/recompess them.
You’re mostly thinking of the storage space right, not the actual size of the write to the server when the post is made.
Yep, storage space. Smaller image sizes will greatly reduce backup sizes, which makes it easier to make and restore backups (especially off server backups which are usually done daily).
I have 2TB on my server so space itself isn’t an issue - it’s the backing up of things and moving to a new server and then if you need to, restoring it all. I wonder how challenging it might be to restore a forum with a DB that’s bigger than 100GB.
So yeah I definitely think smaller is better and letting admins decide the quality of images they want seems like the best option
(I would just do it so that it applies to all uploads - don’t have to worry about specifics like S3 then.)
You could certainly write a script to loop through all JPGs on the server and re-save them at a higher compression level, say 80% compression. That’d probably reduce size by half and is completely safe.
If Discourse uses ImageMagick setting a compression rate via the ACP shouldn’t really interfere with anything else. There may even be a converter, and perhaps that could be utilised with a prompt asking the user if they really want to keep the format to PNG or whether they are happy for it to be converted to a jpg - though I appreciate this might need more work on implementing.
I can see this as very useful feature… Not only preset compression level at ACP but also to limit maximum resolution. I agree with AstonJ that it is not needed to keep maximum resolution in archive… Maybe to recompress files after some time and only files with kilobyte size exceeding some level.
Actually some PNGs make sense - for screengrabbing for example… (it wouldn’t do any good to convert screengrabbed PNGs to JPG).
But I think we should focus on JPEGs where we can be very efficient and save space.
Just for interest sake, how long has your Discourse site been live and what size is your current backup file. I’ve just started up a new site with Discourse which is going to involve lots of image uploads so just wondering where you stand after X months. @codinghorror solution was indeed what I was thinking to do is every month run a cron and compress all images older than X months using littleutils or any other image compression software. Littleutils handles PNG’s quite well and is used as the backend to a number of Wordpress image compression plugins.
My Discourse instance (a local community forum) has been up nine months and is now 2GB.
There’s some Data Explorer SQL that shows upload data volume by user. In my case, a handful of users have been responsible for the majority of uploads.
I’d really like to see a Discourse feature that recompressed and resized large image uploads of a certain age (say, 1 month)
Thanks for the reply. I guess 2Gb over 9 months is not too bad, I’ve allocated 1Tb to my site so I should be good for some time to come I’ll do some testing however with some bash using littleutils using opt-jpg opt-png and opt-gif, shouldn’t be too difficult to have a cron do this once a month. Would be better though as a core feature.
This looks like a promising method, going to test with this on a local dev server.
We’ve been online a year now and the backup is 1GB. However we are a programming forum with relatively few uploads. I imagine one of my other forums would have a significantly bigger DB with all the uploads the members do on those (they are not Discourse forums and images are compressed and saved in the file system).
We’ve been on discourse since the end of Nov. 2016. Our uploads grow by about 0.5MB/month. Our old platform had an image upload facility, but it was a manually vetted process, so there was a delay until the photos were available (and the photos weren’t inline in the forum posts, you had to post a link to the photo album). It’s an antique car forum, so including a photo showing some detail is a boon, and the users love it - I would guess that 10-15% of the posts include a photo.
Reducing the max upload size (3072) isn’t an option, as the pics need enough detail that they show… detail.
Limiting the user’s ability to upload (either by group membership, or by MB/month) wouldn’t really work either - if some new user posts a question of the form “Is this right?” the best way to do that is often to include a picture of the carb, or brake pipe, or whatever, and the easiest answer is that someone with the same car answers by uploading a picture in response. Often a post will start with a detail question and quickly garner 5 “here is how mine is” responses with photos
We’re not using S3 for uploads yet, but it’s only a matter of time. As the users master uploading photos, and start really using it our backup window will get longer. At the moment our backup is 5GB: 1.5GB of uploads, and 1.8 Million posts.
A typical example: Someone will be asking about a construction or assembly detail which isn’t shown in the shop manuals, see this example (a picture is worth a thousand words - and a great many hours in the shop): New old Door hinge issues - XK - Jag-lovers Forums
