Avatars lost after restore. How to get them back?

Hey @ariznaf … not exactly…

In the beginning, we had a standalone app. I seperated that app into two different containers (data and web-only) and then I made a restore from the big backup file with the uploads.

All that went well…

Then, I created a new container, socket-only and configured it to use a reverse proxy.

I did NOT do a restore in the new socket-only container (because the data container already had the DB data in tact) but I failed to manually copy over the uploads (that was my error). If I had done a normal restore process that would not have been necessary.

But there is no reason to do a manually DB restore again in the new container because that is the reason to have the data in it’s on cute little container. So, in this situation, the uploads must be copied over to the new container. It’s actually nicely done.

Does that help?

Not what I said, I said the backend cannot reach itself through the front nginx. What you’re saying is the other way around.

In order to optimize an upload, the Sidekiq job retrieves it using http(s).

2 Likes

No you can disable the ssl template but will need to manually enable force_https.

2 Likes

Good Morning @ariznaf,

Before coming to that conclusion, can you please go to all your installations and go to the shared folder of each of your installs and do this command for each install and post the results back?

# du -sh uploads

For example, on our installs, I did this:

Original Install:

# du -sh uploads
2.5G uploads

Socket-Only Install (before fixing the problem):

# du -sh uploads
444K uploads

This provided me the clue I needed to see that I needed to manually copy over the uploads folder; and then knowing the problem, the solution is easy.

If you check all your different uploads folders with du -sh uploads in all shared directories, this will provide a valuable clue as to what is going on.

If they are different, then you know some uploads are missing and you can manually correct this.

If they are all the same (unlikely), then the problem becomes more interesting :slight_smile:

I have found the problem (not the solution).
The problem is not the restoring itself, but changing the name of the server.

Let me explain how I did the restore (just in case there is some missing step).

I have downloaded a fresh copy discourse from github.
I run the docker-setup process.
Before running the app and proceeding to the restore, I edited app.yml to configure socket access.
And changed the name of the host to b.domain.com (in the original a.domain.com).

Configured ningx reverseproxy using SSL to direct 443 https traffic to the discourse socket.
Then I rebuild (launcher rebuild app) and restart nginx (service nginx restart).
I acessed https://b.domain.com to make the first config of discourse.
Configured it to restore form S3 and restored the las backup made form database and uploads (no thumbnail).
After restoring you are logged out.
I edited app.yml to copy the app.yml from the old site (in order to get the same config and plugins).
Changed the host name to b.domian.com in app.yml

Again a rebuild and nginx restart.

The PROBLEM persists: the profile images (miniatures) of everybody that has changed his profile is changed by that white profile image.
Our logo at the upper left corner is missing too.

@Stephen force_https was on (as it was in the original server, with not problems). I tried to activate and deactivate it, with no effect. I have left it on, as we want out site to be accessed using https (anyway http:80 traffic has a permanent redirect in our nginx config to https:443)

@riking I used sidekiq and trigger the avatarmissing job which seemed to finish OK in a few milleseconds. Just in case it was a lengthy process, I wait almost 24 hours to see if the profile images were reconstructed.
But today we have the same problem: no avatar images (for people taht upload an imagen profile) and no logo image.

After that I tried to see if the problem was the name change, as @Stephen suggested.

I changed back the name of the host in app.yml and nginx to a.domain.com (the original one) and rebuild an restarted nginx.
I have pointed my local hosts file to point a.domain.com to the IP of the new server, and tried a ping to see that it was accessing the new IP.

AND VOILÀ There we have the avatars and out profile.

So the problem is not the restoring itself, the problem is that somehow the complete path of the url are saved and it is trying to access them from the wrong place.
It is strange anyway, as the original server is up and running (it should have found the images even from the wrong place, the original server instead of the new one).

But anyway the problem is not with the restore process or having to reupload the images due to some kind of corruption.

The problem is the change in the server name.

No the question is how do you move a discourse forum from one domain/hostname to another?

I have tried to change again the name of the host to b.domain.com.

No luck.

It seems that when I use the old name it works (but now I am suspecting that it is getting images and things from the old server that is still online, as I get new posts and notifications from new posts in the old server, even if I have changed the IP for a.domain.com in my hosts file).

I have followed the instructions in this post to change the name of the host

I had thought that making the discourse remap a.domain.com to b.domain.com would solve the problem.
Even I have made the rake postes:rebake, but the result is the same.

I have lost the avatars and the logo and the images inserted in posts are lost too.

Finally, as @neounix suggested, I untar all the uploads again to replace the destionation in shared/standalone/uploads/ but with no luck, results are the same.

Is there actually any valuable information in your database? It might be easier to just start over fresh without worrying about server moves at all.

All the data and posts from the beginning of the forum months ago?

As I already said, I have been able to move the server to another server stopping the server, just copying all data to a new one and reestarting it.

But I was told by support that the correct way of doing things is using the standard backup and restoring the database.

I am trying to do that but each time I have some kind of problem.
I need a server to do tests too, in order to test plugins, changes or upgrades effects before doing it in the production.

I cannot wait to have a crash in the server to see if I can restore it or not.

Restore tests till now have ended with some kind of problem, and a non functional system, were images or other things are lost.

Following this saga:

https://meta.discourse.org/t/postgresql-12-update/151236/193

I tried the “backup & restore on a different discourse instance” method and now I’m facing this. Tried every trick in the book (the Sidekiq Job, the Rebake)… is there a “lead” on what may be causing this? Just for me to try to find out something.

(One thing I have to give you all, with all this I’m moving from “yeah I kind of get this” to PostGreSQL PhD, Redis PhD,… :stuck_out_tongue: Just need to get the hang of Ruby and local dev envs and I may turn useful to the Community :stuck_out_tongue: )

Have all avatars disappeared or just part of them? Custom avatars are basically Uploads, do other uploads work as expected?

Go to the rails console and check the record of non-functional avatar in database… Do they have correct URL, filesize, width, height, extension?

User.find_by_username('Overgrow').user_avatar
User.find_by_username('Overgrow').uploaded_avatar

They also need to have their optimized versions present… You can check that by:

OptimizedImage.where(upload_id: upload_id).where(version: 2)
2 Likes

First of all, thank you very much for your help @Overgrow.

All of the avatars and emojis (and even site images, like headers, etc) were “there” but invisible. For non-avatar stuff they appear like broken, for avatars, the gray placeholder. Some people have been able to just upload a new one and those ones you can see.

First tries running the command I got:

FATAL:  the database system is in recovery mode

So… there is that :eyes: (I do have a lot of “disconnects”, so I’m assuming it has something to do with the DB, maybe?)

But after persisting eventually:

User.find_by_username(‘Overgrow’).user_avatar

=> #<UserAvatar:0x000055702722d200
 id: 4,
 user_id: 3,
 custom_upload_id: 20504,
 gravatar_upload_id: 12240,
 last_gravatar_download_attempt: Thu, 21 May 2020 10:16:55 UTC +00:00,
 created_at: Sat, 30 May 2019 16:33:16 UTC +00:00,
 updated_at: Thu, 21 May 2020 10:16:55 UTC +00:00>

(Tried to upload a new one today but It doesn’t work).

User.find_by_username(‘Overgrow’).uploaded_avatar

=> #<Upload:0x00005555cd911b58
 id: 20504,
 user_id: 3,
 original_filename: "16_2.png.jpg",
 filesize: 56220,
 width: 360,
 height: 360,
 url: "/uploads/default/original/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8.jpeg",
 created_at: Thu, 15 Aug 2019 20:02:47 UTC +00:00,
 updated_at: Thu, 15 Aug 2019 20:02:47 UTC +00:00,
 sha1: "63347a46c0ca945f53613722a73c233484d642c8",
 origin: nil,
 retain_hours: nil,
 extension: "jpeg",
 thumbnail_width: 360,
 thumbnail_height: 360,
 etag: nil,
 secure: false,
 access_control_post_id: nil,
 original_sha1: nil>

OptimizedImage.where(upload_id: 20504).where(version: 2)

=> [#<OptimizedImage:0x000056366a01c1a0
  id: 95962,
  sha1: "5a32b5cc3e6f5c58d88a3c92a23076980a8ce840",
  extension: ".jpeg",
  width: 200,
  height: 200,
  upload_id: 20504,
  url: "/uploads/default/optimized/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8_2_200x200.jpeg",
  filesize: 28916,
  etag: nil,
  version: 2>,
 #<OptimizedImage:0x000056366a0741e8
  id: 95942,
  sha1: "ee353c9e23511b471e1a59c1f71a2ded3e366b1e",
  extension: ".jpeg",
  width: 20,
  height: 20,
  upload_id: 20504,
  url: "/uploads/default/optimized/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8_2_20x20.jpeg",
  filesize: 1270,
  etag: nil,
  version: 2>,
 #<OptimizedImage:0x000056366a074120
  id: 95943,
  sha1: "944fa9fc542a79a5c50394c75022bf84ace297e5",
  extension: ".jpeg",
  width: 30,
  height: 30,
  upload_id: 20504,
  url: "/uploads/default/optimized/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8_2_30x30.jpeg",
  filesize: 1952,
  etag: nil,
  version: 2>,
 #<OptimizedImage:0x000056366a074058
  id: 95944,
  sha1: "983490e58bed58c971ffa44e440b02ce3ea72bba",
  extension: ".jpeg",
  width: 40,
  height: 40,
  upload_id: 20504,
  url: "/uploads/default/optimized/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8_2_40x40.jpeg",
  filesize: 2695,
  etag: nil,
  version: 2>,
 #<OptimizedImage:0x000056366a07bf60

So apparently the images are there, but they won’t show. Just the gray default avatar placeholder.

Everything looks ok on db record level. You can move into levels above when investigating.

What do you get when you manually follow URLs of uploads you listed?

1 Like

If I put the /uploads/default/optimized/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8_2_200x200.jpeg (ie) after my Discourse’s URL? I get a 404, not found.

So… they don’t exist? (I’m asking with hope :stuck_out_tongue: )

Check also some file URLs from /uploads/default/original not only from /uploads/default/optimized.

404 … That means that you need to check your uploads folder inside /var/discourse/shared/standalone on filesystem and look up where the actual old files are (if they exist). When you find them, try to compare location to newly uploaded files (those that work).

You can also restore them from the backup manually.

Thanks for the explanation.

Just went in there, double checked, some of the paths listed do not exists.

The weird part is that I have people trying to upload new ones and those new ones don’t work also, when you go look with the commands you gave me you can see a path that also doesn’t exist. How does Discourse “map” this? Because one thing that I can quite get is that the files are missing (even though the backup was supposed to carry them over), but new uploads going to phantom paths?

Check the tombstone folder inside your uploads folder - are some of the missing files there?

1 Like

The only folder I see inside uploads is default… the tombstone folder is for deprecated files or something like that?

Also, an extra piece of Info: Turns out that if a user tries to upload the same image they already had (even if they change the name of the file, I assume based on what I can see with those queries that is based on the hash) the image won’t load, will appear empty, once you save you will see the grey placeholder.

Apparently if you modify the image somehow (even if just saving it as a different format in Photoshop) then you can re-upload it.

That is normal behaviour. Hash of the data file is stored in db to avoid duplicate images.

What happens if you upload picture through compose editor? Does it complete upload? Does it appear in preview pane?

If I write a message and include an image on it? It will show on the preview panel and will complete upload, yes, normal behavior.

So in which exact point that image stops showing?

Check url of that image you see and track it down to the filesystem.

Check url of images that don’t work (through web developer tools in browser). What is the difference?

Maybe they are pointing to different domain…

1 Like