Expire orphaned attachments to avoid illegal hosting


(Sander Datema) #1

Every time you drag an image or file to the editor it will be uploaded. After that you could cancel the post, but keep the links to the attachment.

Some users on my forum came with the idea to host illegal files without me knowing cause I can’t check all the files on Amazon S3.

I think this is a realistic possibility. The fact that by default you can’t upload very large files won’t stop you if you e.g. want to share eBooks.

Possible solutions:

  • if you’re using Amazon S3, you could prevent hotlinking I think
  • have a Sidekiq task check for orphans and remove them, say, every day or week.

(Michael - DiscourseHosting.com) #2

That said, the image:clean_orphans rake task is broken and / because the images:reindex rake task has been removed from the code base about a month ago.

Before that task was removed, it wasn’t working either, we had problems getting Upload.uploaded_regex.match to work.

Apologies for not reporting that earlier. It just crossed my mind again when I read Sander’s post.


(Jeff Atwood) #3

Ah yes, the intent is that old unused (unreferenced by any post…) images are cleaned up periodically. If that is broken @zogstrip we should fix it.


(Régis Hanol) #4

I’ve been working on this lately:

https://github.com/discourse/discourse/commit/4536b5fe04182baadcb76371d429949bb0a3cfa0

It doesn’t delete any file by default. I’m being super precautious here as it’s removing files on the server and I really don’t want to make a mistake here. If you want to enable this feature, change the value of the clean_up_uploads site setting.

The images:clean_orphans rake task has been replaced with a background job which uses the reverse index to clean up uploads that aren’t referenced in any post or used as an avatar.

The job will run hourly and will only remove uploads that are older than uploads_grace_period_in_hours (which defaults to 1).


(Sam Saffron) #5

This has been complete for quite a while … closing.


(Sam Saffron) #6