Images lost in backup/restore, change of domain, can't upload new

[anon23840344]

My group was running our Discourse on a temporary domain for two months until we agreed on our actual domain name. Yesterday I attempted to migrate all content from the old domain to the new one. All text content, user accounts, and even inter-thread links transferred correctly. But now:

• All past image embeds appear to be lost.
• No new images can be uploaded.

This was my process:

• Generated a new Discourse app on a new DigitalOcean droplet.
• Connected the new domain name to that new droplet.
• Confirmed both Discourse apps and all plugs were up to date with latest available software versions.
• Put the old Discourse into read-only mode, to prevent additions of any new content.
• Ran a backup of the old Discourse.
• Uploaded the backup to the new Discourse.
• Updated the Discourse email address from the old domain’s email to the new domain’s email.
• Ran a test of notifications, and they worked for the new Discourse.
• Clicked through all Discourse settings to update any mentions of old domain to new domain.
• Changed the old subdomain to redirect to its proper domain, and temporarily added a note and link to the new Discourse there.

As written above, most content appeared to transfer flawlessly. But now, a day later, we’ve noticed old image embeds are lost, and no new images can be uploaded. Only their “alt” content appears. Screenshot of an example follows.

In googling, it appears there are several lengthy threads about this, but I didn’t see one that includes a change in domain name and an inability to re-upload.

I’ve attempted just now to resolve this by:

• SSH into the machine.
• CD in the Discourse directory and enter the app.
• Run rake posts:missing_uploads. This reports:

Looking for missing uploads on: default

0 post uploads are missing.

• Run rake uploads:missing. This reports a long list:

/var/www/discourse/public/uploads/default/original/1X/bbc547e72f080561282be277749165709cbb0983.ico
/var/www/discourse/public/uploads/default/original/1X/0a421ccd1a08047895e2355f44c332f8b069107d.jpeg
/var/www/discourse/public/uploads/default/original/1X/034e0353b7558a26252c82982de53002fda0a33f.jpeg

[…]

/var/www/discourse/public/uploads/default/original/1X/f7a6164ffa55af4ee2706d2386227183ef6c2d61.png
96 of 281 uploads are missing
/var/www/discourse/public/uploads/default/optimized/1X/997bc5536763d84a8d035ff7becd98277a158680_2_45x45.png

[…]

/var/www/discourse/public/uploads/default/optimized/1X/8944afba36549c9050ef074b391625ef93d4d0e3_2_1035x582.jpeg
/var/www/discourse/public/uploads/default/optimized/1X/8944afba36549c9050ef074b391625ef93d4d0e3_2_10x10.png
247 of 761 optimized_images are missing

• Run rake uploads:recover_from_tombstone. This outputs nothing.

I admit I don’t know what those Rake commands are doing.

I can also see within the containers/app.yml file that DISCOURSE_HOSTNAME is the correct (new) subdomain and domain.

Running ./launcher rebuild app appears to change nothing.

Can anyone help, please? Thank you.

[omarfilip]

Have you tried a rebake?

cd /var/discourse
./launcher enter app
rake posts:rebake

Rebake all posts?

If the URL to your site changed, yes, it’s a good idea to rebake.

[anon23840344]

@omarfilip

Have you tried a rebake?

Thanks for the suggestion. I have input your commands just now, and I see no difference. Old images are still not seen. New images still fail to upload. The web browser console throws a “GET 404” error upon completion of the upload attempt, by the way.

[anon23840344]

Update:

I don’t know which command exactly improved the situation, but it’s improved yet not resolved.

• I can create and upload new files of type TXT, DOCX, XLSX, JPG, and PNG.
• Most (but not all) old uploads of all types are now displaying and downloading properly.

The rogue uploads were displaying/downloading fine before without complaint. And they refuse to show after re-upload in new test posts. As an example:

This image: https://www.frontiersin.org/files/Articles/470644/fphys-10-00944-HTML-r1/image_m/fphys-10-00944-t001.jpg

Displays properly (in my Discourse) when entered a URL on its own line, as below here:

But it disappears from the post after I finish post and refresh the page, leaving what appears to be only a single blank line. Inspecting the source reveals an image element, with a SRC attr set to a path on my domain structured as https://discourse.my_domain_name.org/uploads/default/original/1X/61ae2bdeff3dfe334ad6803409560b667d7dc246.jpeg. And that path opened in a new tab loads an NGINX 404 page.

When I download it from its original source to my laptop, and I upload to my Discourse (as I’ll do here in this Discourse below), it fails on my Discourse.

On my Discourse, it looks like this:

And if I control-click that failed image and click “Open Image in New Tab”, it loads an NGINX 404 page.

I see in this composition dialog right now that Discourse changes the image extension from JPG to JPEG, but I don’t think that would matter.

If I open the downloaded image in Affinity Photo app, export it with no compression as a new JPG, and re-upload it to my Discourse, it displays fine on my Discourse.

This would suggest to me the problem is still with my Discourse, not with the uploaded files.

[Alec]

I came across this problem during recent transfer.

The easiest non tech way is resolving this, was logging on to you original site/host with FileZilla, navigate to

VAR/Discourse/Uploads

Under here you will find subfolders with images. I simply downloaded them from original host and uploaded them to the new one.

An quicker alternative way is to use this method of backup. It’s manual process but worked well also.

Manually create and restore Discourse backups

As discussed below, a warning for newbies: this is an advanced method! there is an automated, web UI way to restore from backups at /admin/backups which is so much easier. with the method below, no automatic backup is kept of the /var/discourse/containers/app.yml file which contains essential SMTP credentials and SSL customizations. Keep a copy of this offsite whenever you change it. If you are using digital ocean, keep frequent snapshots of your droplets - preferably before each time you run .…

[anon23840344]

@Alec

The easiest non tech way is resolving this, was logging on to you original site/host with FileZilla, navigate to

VAR/Discourse/Uploads

Thanks for replying. I’ve never used FileZilla app. I use Transmit app for FTP. But I don’t know how to connect to my Discourse app on my DigitalOcean droplet using FTP.

I can SSH into my droplet, and I can navigate to /var/discourse/, but there’s no uploads directory there.

Even if I were manually to move image files from the old location to the new location, how would that address the failure of re-uploading the same images?

[Alec]

My very basic understand of Discourse (and I am very much a novice) is that uploaded images are stored on your site, and an optimised file size version of the image are also stored. Initially, the original image will be displayed to users, and at some point, the forum switches over to an optimized image (to me that appeared to be after 24 hours).

I experimented a little bit with the settings as I was concerned about running out of storage.

On your admin dashboard, check them out (Under Files section). There are various settings to control image size, and dimensions. This might be related to your rogue image problem (if I understand it correctly). For example, your reformated image still falls outside the criteria.

My issue when I transferred hosts from Digital Ocean To Hetzner was that I had some images (and avatars) that simply were not appearing (similar to your first screenshot). My solution was FTP move them (crude but it worked). Not 100% sure why. At first I moved only the optimized images and it didn’t fix the problem, but when I moved all the images it resolved my issue.

Apologies about folder mistake I was doing it from memory. The folder structure is: VAR/Discours/Shared/Standalone/Upload

Its very easy to login with an FTP client with Digital ocean

Host ; IP address of Discourse
Port : 22
Username: root
Password: your SSH password

As I said I am far from an expert, more someone that has tinkered with it the last couple of months.

There are people here that can offer better advice (and insight). But if all else fails it might be worth giving my approach a go.

[anon23840344]

My solution was FTP move them (crude but it worked). Not 100% sure why. At first I moved only the optimized images and it didn’t fix the problem, but when I moved all the images it resolved my issue.

Thanks for the more detail instruction on how to FTP into my two Discourses. The uploads file was about 125 MB on the old Discourse, 60 MB on the new one. So I copied the old one to my laptop’s Desktop. Then I copied its contents, folder-by-folder, to the new Discourse, skipping any duplications.

And to my surprise, it appears to have solved the problem. All image uploads appear to be fixed, both in threads that pre-date my Discourse migration, and in threads that I created today in troubleshooting.

I might theorize that Discourse was re-using pre-existing pointers to content that somehow was lost in the move. So if I re-uploaded the exact same image file, it would re-use a broken pointer and repeatedly fail. But when I resaved it as a new file, it succeeded because it stored a new copy with a new pointer. Perhaps.

Thanks again very much.

[gerhard (Gerhard Schlager)]

By default backups do not contain optimised images as they can be recreated. Including them would waste bandwidth and disk space.

In your case it would have helped to include them by enabling the include_thumbnails_in_backups site setting before creating the backup.

As an alternative, running rake posts:rebake_uncooked_posts after the restore would have worked as well. Otherwise the optimised images are regenerated by a background job, but that takes some time… There’s a log message at the end of the restore which tells you exactly that.