Fixing lost images

So over the various version updates, random images (and other downloads) across my site have lost. For example:

Both the macro download file and the image have disappeared and are now broken links.

I have run a script over my site to look for all links, and it appears there are two formats:

The latter seems to be a newer format, and as far as I can tell the old ones lost are of the old format.

I have lots of backups, and so I retain the files, so I have backups of the 7e111a68ca4297f1.png image for example.

I can restore that file simply placing it in the /var/discourse/shared/standalone/uploads/default folder, and the file is restored and the forum topic repaired.

But will it just be deleted again? Will it be migrated to the new format in due course? Can I induce the server to migrate the file?

Thanks.

3 Likes

@tgxworld can advise you on this

1 Like

@peternlewis You can run the following which will automatically try to recover lost images

./launcher enter app
rake posts:missing_uploads
2 Likes

The image files no longer exist on the disk (at least locate cannot find them). If I put them back and then run that would that help?

Yup if you put them back and run the command, it’ll automatically pick those files up.

3 Likes

OK, I will try that and let you know. Thanks.

Sorry about this, it is our bug based on 2015-era changes to Discourse around image handling. Mistakes were made.

Unfortunately without a time machine it is difficult to properly fix

4 Likes

These things happen. That’s what backups are for. I have pretty good long term backups of the forum so I should be able to restore most of them.

6 Likes

Hmm:

root@forum-app:/var/www/discourse# rake posts:missing_uploads
rake aborted!
Don't know how to build task 'posts:missing_uploads' (See the list of available tasks with `rake --tasks`)
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
(See full trace by running task with --trace)

Did I miss a step somewhere?

Upgrade to latest first :slight_smile:

I’m afraid I am only on the release version (Latest 2.2.4) so I am as upgraded as I can be.

What version was this added in?

This image work is all recent, you are going to have to take the plunge and move to tests-passed.

If you folks are not willing to call it a release grade, I am not willing to install it.

So I guess I’ll have to wait until the release grade is released, and then try to process them.

It is a release grade, this is what most our customers are running.

Anyway, if you want to wait you can and then follow it up when we do next major point release. Just be sure to keep all your backups.

Notes refinements to the recovery process will never be backported to stable.

2 Likes

So do you expect will this be in 2.2.5, 2.3 or 3.0?

Thanks.

This will be in 2.3.0

3 Likes

Unfortunately this produced an error now in 2.3:

Looking for missing uploads on: default
Fixing missing uploads: 
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................rake aborted!
Errno::EISDIR: Is a directory @ io_fread - /var/www/discourse/public/uploads/default/original/3X/2/1/
/var/www/discourse/lib/tasks/posts.rake:438:in `read'
/var/www/discourse/lib/tasks/posts.rake:438:in `block in missing_uploads'
/var/www/discourse/app/models/post.rb:954:in `block (4 levels) in find_missing_uploads'
/var/www/discourse/app/models/post.rb:919:in `block in each_upload_url'
/var/www/discourse/app/models/post.rb:899:in `each'
/var/www/discourse/app/models/post.rb:899:in `each_upload_url'
/var/www/discourse/app/models/post.rb:942:in `block (3 levels) in find_missing_uploads'
/var/www/discourse/app/models/post.rb:941:in `each'
/var/www/discourse/app/models/post.rb:941:in `block (2 levels) in find_missing_uploads'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:222:in `loop'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:222:in `in_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/var/www/discourse/app/models/post.rb:937:in `block in find_missing_uploads'
/var/www/discourse/lib/distributed_mutex.rb:42:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:12:in `synchronize'
/var/www/discourse/app/models/post.rb:928:in `find_missing_uploads'
/var/www/discourse/lib/tasks/posts.rake:416:in `missing_uploads'
/var/www/discourse/lib/tasks/posts.rake:512:in `block (2 levels) in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.0.7/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.0.7/lib/rails_multisite/connection_management.rb:73:in `each_connection'
/var/www/discourse/lib/tasks/posts.rake:511:in `block in <top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => posts:missing_uploads
(See full trace by running task with --trace)

Any suggestions on how to resolve this?

The /var/www/discourse/public/uploads/default/original/3X/2/1/ path is indeed a directory and contains a number of files.

total 3188…
drwxr-xr-x  2 discourse www-data    4096 Jun 26 05:49 .
drwxr-xr-x 18 discourse www-data    4096 Dec 11  2017 ..
-rw-r--r--  1 discourse www-data    2411 Jan  4  2018 21071b37ddd34…8577415a35.kmactions
-rw-r--r--  1 discourse www-data   99179 May  3 22:25 21095a92b7d3a…8bb8fd2f5b.png
-rw-r--r--  1 discourse www-data   22783 Sep 27  2018 2114c84a89433…abd120e3e1.kmmacros
-rw-r--r--  1 discourse www-data   46340 May 17  2018 21164f5ec4756…90a0bbe07b.png
-rw-r--r--  1 discourse www-data   47651 May  8 16:46 211e85592e1a0…601255f5f3.png
-rw-r--r--  1 discourse www-data   67333 Jul 11  2018 2124ab5989748…e3e0d33e16.png
-rw-r--r--  1 discourse www-data     432 Dec 22  2017 2126c7f6da05a…c9a710d23d.kmactions
-rw-r--r--  1 discourse www-data  133395 Oct 26  2018 2129e51f0c84e…b1fc6127c8.png
-rw-r--r--  1 discourse www-data  129649 Oct 30  2017 212ae1b2c938b…3853968d85.png
-rw-r--r--  1 discourse www-data  213930 Jan 20  2018 21319003d36a9…f58405289c.png
-rw-r--r--  1 discourse www-data  103916 Oct 20  2018 213f32c7a3255…580eb8a189.png
-rw-r--r--  1 discourse www-data   24387 Dec 23  2017 21604d2266496…dd7334553b.kmmacros
-rw-r--r--  1 discourse www-data   27339 Dec 23  2017 21629ccc332d9…00ac5aabdc.kmmacros
-rw-r--r--  1 discourse www-data   52099 Dec 13  2018 216c06cf7a7ca…110c1694d5.jpeg
-rw-r--r--  1 discourse www-data    5975 Mar  3  2018 2171321edfe3c…1749a7bfc1.kmmacros
-rw-r--r--  1 discourse www-data   65151 Apr 17 21:29 217137bc6fe62…a12ec4e470.jpeg
-rw-r--r--  1 discourse www-data  363796 Aug  4  2018 217b5d67819af…6bb19c9ef3.png
-rw-r--r--  1 discourse www-data   36719 Jun 26  2018 2187ee6d05c9a…714884942e.png
-rw-r--r--  1 discourse discourse  44526 Jun 26 05:49 218834ab50dda…17bd7fa0d0.png
-rw-r--r--  1 discourse www-data    2252 Jan 20  2018 2198f4d6d83c6…be3fe5af41.kmactions
-rw-r--r--  1 discourse www-data   24262 Jan  4 09:43 21a5a59ea14d1…b52df95133.png
-rw-r--r--  1 discourse www-data   38080 Jul  2  2018 21be08e3b21b3…952a5020e2.png
-rw-r--r--  1 discourse www-data   57504 Dec 15  2017 21bee8898daa9…91ba94914c.png
-rw-r--r--  1 discourse www-data  158573 Apr  8 19:45 21c1a5428a787…566c290bbe.png
-rw-r--r--  1 discourse www-data  213153 Jul 27  2018 21c62d9d28152…3ed5a56700.jpg
-rw-r--r--  1 discourse www-data    5416 Jul  9  2018 21cedc56429e5…1ce3867e0e.png
-rw-r--r--  1 discourse www-data  133401 Oct 27  2017 21da4763f4a08…7ceb7f511e.png
-rw-r--r--  1 discourse www-data   52865 Apr  7  2018 21efa676099fc…bd916337aa.png
-rw-r--r--  1 discourse www-data   44587 May 30  2018 21f4718e1d732…c75d39ec19.jpg
-rw-r--r--  1 discourse www-data   27539 Jan 24 14:14 21fc92d58550a…9b4ff1a960.png
-rw-r--r--  1 discourse www-data  957839 Oct 25  2018 21fd7efbacd50…3b2b0b3fb2.jpeg

(I replaced some of the unique IDs characters with … for irrelevant security, ignore that).