Old image uploads become broken images

For the past week or so (after the most recent upgrade?), I’ve noticed that some users’ uploaded images are broken. It is clear that they are pasting an image, as the code in their post looks like that, but the files are missing.

For example, this post (raw version has three broken images in it. I cannot imagine what the user could have done to break them. And stranger still, sometimes a post will have some images that work fine, but others that are missing. I’ve done find in the uploads directory and the files aren’t there anywhere.

I can’t see any rhyme or reason to it. Has anyone else noticed a similar problem lately?

1 Like

Lest anyone thinks that this query went unanswered, many thanks to @zogstrip for solving this. It seems that Discourse got confused about whether those uploaded images were being used and moved them for subsequent deletion.

Though you will probably never need it, the fix went something like this:

# copy the "deleted" images from their to-be-deleted staging area
cd /var/discourse
./launcher enter app
rake uploads:recover_from_tombstone

# rebake to see that it doesn't happen again
rake posts:rebake
8 Likes

It looks like the first rake uploads:recover_from_tombstone command now handles the rebaking done by the rake posts:rebake command, after this commit:

https://github.com/discourse/discourse/commit/333d6f9f10a4a2b62393188a8d4595f44b764242

So anyone finding this thread now, you can probably skip the second part (unless I am mistaken! I am new to this).

Since this is the only place I found information about this fix, and the two commands take a very long time to run (over 12 hours here), I wanted to update the thread to help save time for anyone else.

Have some odd problems on my new forum (imported from phpBB), where attachments keep being deleted. Initially, it may have been due to me setting the grace period too low and things being deleted as their corresponding posts were still in the import queue, but a few things still seem to be getting deleted, which I’ve not got to the bottom of yet.

2 Likes

Happened to us with 1.8.0.beta6.

While these steps helped:

rake uploads:recover_from_tombstone
rake posts:rebake

I am still not sure what was the cause.

We have seen this happen a few times, immediately after an update.
Since we found this out, we have been disabling Sidekiq during the upgrade and this issue never happened again.

2 Likes

This thread has been very helpful.

A couple of issues I have encountered with broken upload links:

  1. My board has a non-traditional file upload enabled (mp3) and Discourse was going through deleting all uploads older than 48 hours. I have resolved this by unchecking clean up uploads in admin.

  1. If you host uploads on Amazon S3 rake uploads:recover_from_tombstone will not work. You get this error instead: This task only works for internal storages

Is there a way to keep the remove orphan unreferenced uploads option enabled and have it recognise and not delete non-regular files (such as mp3)?

Uploads that are linked in posts shouldn’t be removed. That setting only helps with uploads that are not linked in any posts.

I have a similar problem (I think) after an update to stable v1.8.7, but the command rake uploads:recover_from_tombstone says:

  1 / 9 ( 11.1%)
Restoring /var/www/discourse/public/uploads/tombstone/default/original/1X/27bf77d1d3f09d13b1f89cf97552a8ae19aa4d98.png...rake aborted!
TypeError: no implicit conversion of Symbol into Integer
/var/www/discourse/lib/upload_creator.rb:190:in `[]'
/var/www/discourse/lib/upload_creator.rb:190:in `should_crop?'
/var/www/discourse/lib/upload_creator.rb:47:in `block in create_for'
/var/www/discourse/lib/distributed_mutex.rb:21:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:5:in `synchronize'
/var/www/discourse/lib/upload_creator.rb:34:in `create_for'
/var/www/discourse/lib/tasks/uploads.rake:441:in `block (4 levels) in recover_from_tombstone'
/var/www/discourse/lib/tasks/uploads.rake:440:in `open'
/var/www/discourse/lib/tasks/uploads.rake:440:in `block (3 levels) in recover_from_tombstone'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/nokogiri-1.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/nokogiri-1.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/nokogiri-1.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
/var/www/discourse/lib/tasks/uploads.rake:428:in `block (2 levels) in recover_from_tombstone'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/delegation.rb:46:in `each'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/delegation.rb:46:in `each'
/var/www/discourse/lib/tasks/uploads.rake:424:in `block in recover_from_tombstone'
/var/www/discourse/lib/tasks/uploads.rake:420:in `each'
/var/www/discourse/lib/tasks/uploads.rake:420:in `each_with_index'
/var/www/discourse/lib/tasks/uploads.rake:420:in `recover_from_tombstone'
/var/www/discourse/lib/tasks/uploads.rake:401:in `block (2 levels) in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rails_multisite-1.0.6/lib/rails_multisite/connection_management.rb:126:in `block in each_connection'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rails_multisite-1.0.6/lib/rails_multisite/connection_management.rb:124:in `each'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/rails_multisite-1.0.6/lib/rails_multisite/connection_management.rb:124:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:401:in `block in <top (required)>'
/usr/local/bin/bundle:22:in `load'
/usr/local/bin/bundle:22:in `<main>'
Tasks: TOP => uploads:recover_from_tombstone
(See full trace by running task with --trace)

Is this a bug or a problem with my installation?

I rebuild the app and restarted the server (as it seems to have a problem with sidekiq) and then I did a hack (?) but it this seems to work for all the broken images:

./launcher enter app
cp public/uploads/tombstone/default/original/1X/* public/uploads/default/original/1X/

Here’s what you want to do:

cd /var/discourse
./launcher enter app
bundle exec rake uploads:recover_from_tombstone
3 Likes

Ah, thanks. Should I still do this or is this doing just the cp like I did?

You should do it. My incomplete understanding is that it updates the database so that this won’t happen again.

1 Like

Hmmh, I do something wrong. It either says:

URGENT: FATAL:  Peer authentication failed for user "discourse"
 Failed to initialize site default
rake aborted!
PG::ConnectionBad: FATAL:  Peer authentication failed for user "discourse"
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `initialize'
...

or when I do sudo -u discourse it says:

rake aborted!
Errno::EACCES: Permission denied
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/bootsnap-0.3.0/lib/bootsnap/compile_cache/iseq.rb:34:in `fetch'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/bootsnap-0.3.0/lib/bootsnap/compile_cache/iseq.rb:34:in `load_iseq'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/bootsnap-0.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/bootsnap-0.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
...

Did you do ./launcher enter app?

Yes, and then I end up in the folder /var/www/discourse

Hmm. Try just rake uploads:recover_from_tombstone (without bundle exec), but I just did the bundle exec and it worked.

I did this in my original comment

1 Like

Sorry. And this is a standard Docker install?

Yes. I’ve started using it with a very early version (I think I started with the first docker shipped version) but the upgrades always went without problems, except recently (but was ‘fixable’)

I’m guessing here, but… maybe try running /var/discourse/launcher rebuild app first, to make sure you have the latest base image? :thinking:

2 Likes