So over the various version updates, random images (and other downloads) across my site have lost. For example:
In Yosemite, the Finder finally allows Batch Rename of files by control-clicking on a set of files. However, the options are fairly limited and their is no preview at all. The end of the TidBITS article and the comments list various third party...
Reading time: 1 mins 🕑
Likes: 10 ❤
Both the macro download file and the image have disappeared and are now broken links.
I have run a script over my site to look for all links, and it appears there are two formats:
The latter seems to be a newer format, and as far as I can tell the old ones lost are of the old format.
I have lots of backups, and so I retain the files, so I have backups of the 7e111a68ca4297f1.png image for example.
I can restore that file simply placing it in the /var/discourse/shared/standalone/uploads/default folder, and the file is restored and the forum topic repaired.
But will it just be deleted again? Will it be migrated to the new format in due course? Can I induce the server to migrate the file?
Thanks.
3 Likes
@tgxworld can advise you on this
1 Like
tgxworld
May 2, 2019, 1:06am
4
@peternlewis You can run the following which will automatically try to recover lost images
./launcher enter app
rake posts:missing_uploads
2 Likes
The image files no longer exist on the disk (at least locate cannot find them). If I put them back and then run that would that help?
tgxworld
May 2, 2019, 1:08am
6
Yup if you put them back and run the command, it’ll automatically pick those files up.
3 Likes
OK, I will try that and let you know. Thanks.
Sorry about this, it is our bug based on 2015-era changes to Discourse around image handling. Mistakes were made.
Unfortunately without a time machine it is difficult to properly fix
4 Likes
These things happen. That’s what backups are for. I have pretty good long term backups of the forum so I should be able to restore most of them.
6 Likes
Hmm:
root@forum-app:/var/www/discourse# rake posts:missing_uploads
rake aborted!
Don't know how to build task 'posts:missing_uploads' (See the list of available tasks with `rake --tasks`)
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
(See full trace by running task with --trace)
Did I miss a step somewhere?
I’m afraid I am only on the release version (Latest 2.2.4) so I am as upgraded as I can be.
What version was this added in?
sam
May 2, 2019, 6:31am
13
This image work is all recent, you are going to have to take the plunge and move to tests-passed.
If you folks are not willing to call it a release grade, I am not willing to install it.
So I guess I’ll have to wait until the release grade is released, and then try to process them.
sam
May 2, 2019, 7:03am
15
It is a release grade, this is what most our customers are running.
Anyway, if you want to wait you can and then follow it up when we do next major point release. Just be sure to keep all your backups.
Notes refinements to the recovery process will never be backported to stable.
2 Likes
So do you expect will this be in 2.2.5, 2.3 or 3.0?
Thanks.
Unfortunately this produced an error now in 2.3:
Looking for missing uploads on: default
Fixing missing uploads:
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................rake aborted!
Errno::EISDIR: Is a directory @ io_fread - /var/www/discourse/public/uploads/default/original/3X/2/1/
/var/www/discourse/lib/tasks/posts.rake:438:in `read'
/var/www/discourse/lib/tasks/posts.rake:438:in `block in missing_uploads'
/var/www/discourse/app/models/post.rb:954:in `block (4 levels) in find_missing_uploads'
/var/www/discourse/app/models/post.rb:919:in `block in each_upload_url'
/var/www/discourse/app/models/post.rb:899:in `each'
/var/www/discourse/app/models/post.rb:899:in `each_upload_url'
/var/www/discourse/app/models/post.rb:942:in `block (3 levels) in find_missing_uploads'
/var/www/discourse/app/models/post.rb:941:in `each'
/var/www/discourse/app/models/post.rb:941:in `block (2 levels) in find_missing_uploads'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:222:in `loop'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:222:in `in_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/var/www/discourse/app/models/post.rb:937:in `block in find_missing_uploads'
/var/www/discourse/lib/distributed_mutex.rb:42:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:12:in `synchronize'
/var/www/discourse/app/models/post.rb:928:in `find_missing_uploads'
/var/www/discourse/lib/tasks/posts.rake:416:in `missing_uploads'
/var/www/discourse/lib/tasks/posts.rake:512:in `block (2 levels) in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.0.7/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.0.7/lib/rails_multisite/connection_management.rb:73:in `each_connection'
/var/www/discourse/lib/tasks/posts.rake:511:in `block in <top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => posts:missing_uploads
(See full trace by running task with --trace)
Any suggestions on how to resolve this?
The /var/www/discourse/public/uploads/default/original/3X/2/1/ path is indeed a directory and contains a number of files.
total 3188…
drwxr-xr-x 2 discourse www-data 4096 Jun 26 05:49 .
drwxr-xr-x 18 discourse www-data 4096 Dec 11 2017 ..
-rw-r--r-- 1 discourse www-data 2411 Jan 4 2018 21071b37ddd34…8577415a35.kmactions
-rw-r--r-- 1 discourse www-data 99179 May 3 22:25 21095a92b7d3a…8bb8fd2f5b.png
-rw-r--r-- 1 discourse www-data 22783 Sep 27 2018 2114c84a89433…abd120e3e1.kmmacros
-rw-r--r-- 1 discourse www-data 46340 May 17 2018 21164f5ec4756…90a0bbe07b.png
-rw-r--r-- 1 discourse www-data 47651 May 8 16:46 211e85592e1a0…601255f5f3.png
-rw-r--r-- 1 discourse www-data 67333 Jul 11 2018 2124ab5989748…e3e0d33e16.png
-rw-r--r-- 1 discourse www-data 432 Dec 22 2017 2126c7f6da05a…c9a710d23d.kmactions
-rw-r--r-- 1 discourse www-data 133395 Oct 26 2018 2129e51f0c84e…b1fc6127c8.png
-rw-r--r-- 1 discourse www-data 129649 Oct 30 2017 212ae1b2c938b…3853968d85.png
-rw-r--r-- 1 discourse www-data 213930 Jan 20 2018 21319003d36a9…f58405289c.png
-rw-r--r-- 1 discourse www-data 103916 Oct 20 2018 213f32c7a3255…580eb8a189.png
-rw-r--r-- 1 discourse www-data 24387 Dec 23 2017 21604d2266496…dd7334553b.kmmacros
-rw-r--r-- 1 discourse www-data 27339 Dec 23 2017 21629ccc332d9…00ac5aabdc.kmmacros
-rw-r--r-- 1 discourse www-data 52099 Dec 13 2018 216c06cf7a7ca…110c1694d5.jpeg
-rw-r--r-- 1 discourse www-data 5975 Mar 3 2018 2171321edfe3c…1749a7bfc1.kmmacros
-rw-r--r-- 1 discourse www-data 65151 Apr 17 21:29 217137bc6fe62…a12ec4e470.jpeg
-rw-r--r-- 1 discourse www-data 363796 Aug 4 2018 217b5d67819af…6bb19c9ef3.png
-rw-r--r-- 1 discourse www-data 36719 Jun 26 2018 2187ee6d05c9a…714884942e.png
-rw-r--r-- 1 discourse discourse 44526 Jun 26 05:49 218834ab50dda…17bd7fa0d0.png
-rw-r--r-- 1 discourse www-data 2252 Jan 20 2018 2198f4d6d83c6…be3fe5af41.kmactions
-rw-r--r-- 1 discourse www-data 24262 Jan 4 09:43 21a5a59ea14d1…b52df95133.png
-rw-r--r-- 1 discourse www-data 38080 Jul 2 2018 21be08e3b21b3…952a5020e2.png
-rw-r--r-- 1 discourse www-data 57504 Dec 15 2017 21bee8898daa9…91ba94914c.png
-rw-r--r-- 1 discourse www-data 158573 Apr 8 19:45 21c1a5428a787…566c290bbe.png
-rw-r--r-- 1 discourse www-data 213153 Jul 27 2018 21c62d9d28152…3ed5a56700.jpg
-rw-r--r-- 1 discourse www-data 5416 Jul 9 2018 21cedc56429e5…1ce3867e0e.png
-rw-r--r-- 1 discourse www-data 133401 Oct 27 2017 21da4763f4a08…7ceb7f511e.png
-rw-r--r-- 1 discourse www-data 52865 Apr 7 2018 21efa676099fc…bd916337aa.png
-rw-r--r-- 1 discourse www-data 44587 May 30 2018 21f4718e1d732…c75d39ec19.jpg
-rw-r--r-- 1 discourse www-data 27539 Jan 24 14:14 21fc92d58550a…9b4ff1a960.png
-rw-r--r-- 1 discourse www-data 957839 Oct 25 2018 21fd7efbacd50…3b2b0b3fb2.jpeg
(I replaced some of the unique IDs characters with … for irrelevant security, ignore that).
