Uploaded files mysteriously going missing

On our forum at https://discourse.vvvv.org/ we have uploaded files and images disappearing after some time. The images turn up broken (example ) and the file links go into nowhere (example ). I looks like somehow the physical copy of the file get’s deleted after a certain time, but I can’t see a pattern right now. We’re running a standard docker installation. Is there something I might be missing here? Is there some kind of cleanup that discourse does which might be removing those files? I have seen related posts here, but none really seem to give the exact answer or explanation of what might be going on.

Thanks all for your help in advance!

1 Like

Are you running into this?

@cpradio no we have never rebaked the posts so far…

Well that also happens automatically in a job (IIRC) – for example, there is a process where it comes back and downloads images to the system that were linked to external sources. So you might not be doing it automatically, but a sidekiq job will be.

There is also a sidekiq job named Jobs::CleanUpUploads

Ok, the setting was enabled for us. I disabled it and we’ll see what happens. Thanks for pointing me there!

Bonus question: I copied back the file from the tombstone directory into the default directory. The deleted files are back in default, but they don’t seem to be reachable via their public link. For example this is a dead link on our page:

https://discourse.vvvv.org/uploads/default/original/1X/19c3c4928c977ba74ccadc77f904ab1c1a1fa0d5.zip

But the file

/var/discourse/shared/standalone/uploads/default/original/1X/19c3c4928c977ba74ccadc77f904ab1c1a1fa0d5.zip

exists in the filesystem. I also did a ./launcher restart app, but the link is still not reachable.

What if you “rebuild the html” (also known as rebake) the post that contains the link to this attachment?

How do I do that for an individual post? Searching for this on meta tells me this option should be in the wrench menu, but there does not seem to be an entry for this (and neither do I find something in the admin section of the page).

You have to click the button under said post, then click the wrench button, then click the Rebuild HTML button :wink:

Weirdly I do not have that button, but I just rebuilt all the posts now, oh well :slight_smile: . The Problem with this file is still persisting though…

Have you got .zip upload enabled in Settings (authorized extensions)?

What happens if you upload any .zip file through manual upload to post? Is it placed in directory correctly?

Try it and check for difference compared to your other zip file… (file system permissions?)

We managed to fix this with big help by @zogstrip. To me it’s not quite clear yet why it happened, but I have disabled the the cleaning up of files for now. It may have to do with a massive import of posts we did when starting of with discourse, as there was some hackery involved from my side :slight_smile:

2 Likes

Unfortunately, I‘ve noticed the same issue. There are several topics with missing uploads and broken images. I‘ve rebake the HTML of these specific sites, but can’t fix it. :roll_eyes: Most of them are pretty old (2014-2016)

I‘m not gonna disable the auto clean option, for good reasons.

Pretty old is likely the old upload scheme.

./launcher enter app
rails c
> SiteSetting.migrate_to_new_scheme = true
.... wait a day
rake posts:rebake

May fix it.

3 Likes
[1] pry(main)> rake posts:rebake
NameError: undefined local variable or method `rebake' for main:Object
from (pry):1:in `__pry__'
[2] pry(main)>

?

Not in rails c… just ./launcher enter app then type that.

2 Likes

My fault! Thanks for the hint. Unfortunately your workaround didn’t solve my issue. There are a lot of files missing and images still broken.