Upload Deletion via API

Contribute Feature
,
3

The use case is for time critical removal of uploads as part of human in the loop automation workflows using activepieces, data explorer and the API.

The goal is to enable regular moderators to be able to have access to a simple way to confidently fully remove uploads immediately without SSH access and comprehensively cover all known cases for breaking upload references as well as automatically CDN purge the specific urls (and for the case of proxied avatar, the URL prefix based on the username).

I was relieved when testing, to see that avatar, profile background and card background references are automatically handled when an upload is destroyed.

https://github.com/discourse/discourse/commit/e1975e293f2625259e925b4a3c93d88d5acfcaa8

https://github.com/discourse/discourse/commit/38e7b1a0492dd4282c3cd3b1ddb2b3343661d31f

The two main scenarios would be ‘scorched earth’ and ‘surgical removal’. This is the work in progress for scorched earth:

Converting to an upload hash list:

  • avatar urls (extract username using regex) → get avatar hash via data explorer query (the hash is not in the url)

  • topic/post urls → collect all upload hashes used in that post using data explorer

  • direct original/optimised upload urls (incl. profile background and card background) → extract hashes using regex

Then query each hash to find all occurrences (one data explorer query to cover all cases, single hash at a time):

  • username/id list, for users who used it for an avatar

  • username/id list, for users who used it for a profile background

  • username/id list, for users who used it for a card background

  • list of all posts (raw) that use this upload

Actions:

  • suspend all users who used the upload for an avatar, profile background or card background

  • suspend all users who have posts referencing the target upload, but exclude if their reference is nested inside a quote

  • delete all topics

  • delete all posts

  • destroy upload (no API endpoint)

  • purge all CDN urls (optimised/unoptimised)

  • purge the standard prefix for proxied avatar urls for each associated username (to cover all sizes)

The surgical removal scenario would essentially be the same, but not suspend associated users and need some changes regarding collecting all the upload hashes from post/topic urls.

Likely still delete the posts/topics themselves to avoid broken references, but stripping the upload markdown for that specific upload (without touching other uploads) would be superior if possible. Like this automation if it didn’t strip all markdown upload references.

Ideally I’d like to be able to block hashes also, so after going through the above, someone can’t just simply create a new account and re-upload. :melting_face: :coffin:

I don’t think it’s possible currently for a regular mod, like by using watched words. So perhaps doing a periodic scan like the above for a list of hashes would be a way to handle that.

1 Like