Avatars lost after restore. How to get them back?

[neounix (Dark Matter)]

@ariznaf

If that is the case (I will try) it seem clear to me that there is some kind of bug in the restoring job.

Good Morning @ariznaf,

Before coming to that conclusion, can you please go to all your installations and go to the shared folder of each of your installs and do this command for each install and post the results back?

# du -sh uploads

For example, on our installs, I did this:

Original Install:

# du -sh uploads
2.5G uploads

Socket-Only Install (before fixing the problem):

# du -sh uploads
444K uploads

This provided me the clue I needed to see that I needed to manually copy over the uploads folder; and then knowing the problem, the solution is easy.

If you check all your different uploads folders with du -sh uploads in all shared directories, this will provide a valuable clue as to what is going on.

If they are different, then you know some uploads are missing and you can manually correct this.

If they are all the same (unlikely), then the problem becomes more interesting

[ariznaf (fernando)]

I have found the problem (not the solution).
The problem is not the restoring itself, but changing the name of the server.

Let me explain how I did the restore (just in case there is some missing step).

I have downloaded a fresh copy discourse from github.
I run the docker-setup process.
Before running the app and proceeding to the restore, I edited app.yml to configure socket access.
And changed the name of the host to b.domain.com (in the original a.domain.com).

Configured ningx reverseproxy using SSL to direct 443 https traffic to the discourse socket.
Then I rebuild (launcher rebuild app) and restart nginx (service nginx restart).
I acessed https://b.domain.com to make the first config of discourse.
Configured it to restore form S3 and restored the las backup made form database and uploads (no thumbnail).
After restoring you are logged out.
I edited app.yml to copy the app.yml from the old site (in order to get the same config and plugins).
Changed the host name to b.domian.com in app.yml

Again a rebuild and nginx restart.

The PROBLEM persists: the profile images (miniatures) of everybody that has changed his profile is changed by that white profile image.
Our logo at the upper left corner is missing too.

@Stephen force_https was on (as it was in the original server, with not problems). I tried to activate and deactivate it, with no effect. I have left it on, as we want out site to be accessed using https (anyway http:80 traffic has a permanent redirect in our nginx config to https:443)

@riking I used sidekiq and trigger the avatarmissing job which seemed to finish OK in a few milleseconds. Just in case it was a lengthy process, I wait almost 24 hours to see if the profile images were reconstructed.
But today we have the same problem: no avatar images (for people taht upload an imagen profile) and no logo image.

After that I tried to see if the problem was the name change, as @Stephen suggested.

I changed back the name of the host in app.yml and nginx to a.domain.com (the original one) and rebuild an restarted nginx.
I have pointed my local hosts file to point a.domain.com to the IP of the new server, and tried a ping to see that it was accessing the new IP.

AND VOILÀ There we have the avatars and out profile.

So the problem is not the restoring itself, the problem is that somehow the complete path of the url are saved and it is trying to access them from the wrong place.
It is strange anyway, as the original server is up and running (it should have found the images even from the wrong place, the original server instead of the new one).

But anyway the problem is not with the restore process or having to reupload the images due to some kind of corruption.

The problem is the change in the server name.

No the question is how do you move a discourse forum from one domain/hostname to another?

[ariznaf (fernando)]

I have tried to change again the name of the host to b.domain.com.

No luck.

It seems that when I use the old name it works (but now I am suspecting that it is getting images and things from the old server that is still online, as I get new posts and notifications from new posts in the old server, even if I have changed the IP for a.domain.com in my hosts file).

I have followed the instructions in this post to change the name of the host

Change the domain name or rename your Discourse

This guide is for self-hosted Discourse instances that were installed using our installation guide on GitHub. If you are hosted by Discourse, please see Configure your domain name for hosted Discourse for specific instructions. You may eventually want to change the domain name of your Discourse instance, from say =BEFORE= → =AFTER= Let’s get started. Set the TTL for your DNS low, and in advance You need to let everyone know that the domain is about to change, so change the TTL f…

I had thought that making the discourse remap a.domain.com to b.domain.com would solve the problem.
Even I have made the rake postes:rebake, but the result is the same.

I have lost the avatars and the logo and the images inserted in posts are lost too.

Finally, as @neounix suggested, I untar all the uploads again to replace the destionation in shared/standalone/uploads/ but with no luck, results are the same.

[riking (Kane York)]

Is there actually any valuable information in your database? It might be easier to just start over fresh without worrying about server moves at all.

[ariznaf (fernando)]

All the data and posts from the beginning of the forum months ago?

As I already said, I have been able to move the server to another server stopping the server, just copying all data to a new one and reestarting it.

But I was told by support that the correct way of doing things is using the standard backup and restoring the database.

I am trying to do that but each time I have some kind of problem.
I need a server to do tests too, in order to test plugins, changes or upgrades effects before doing it in the production.

I cannot wait to have a crash in the server to see if I can restore it or not.

Restore tests till now have ended with some kind of problem, and a non functional system, were images or other things are lost.

[Iceman]

Following this saga:

Improving Instance Performance (Megatopics, Database Size and Extreme Load)

Hi Discourse Community! Recently I have been trying to improve the performance of my Discourse installation and clean up a little since my site has been growing exponentially lately. I have identified two issues, searched about them here but I don’t see a clear answer for some specifics, so I hope that this doesn’t represent much of a bother to anyone. The first issue I have is the DB Size, which is pretty big. I’m running a 4GB Mem/80GB Disk instance on Digital Ocean and the DB is choking t…

https://meta.discourse.org/t/postgresql-12-update/151236/193

I tried the “backup & restore on a different discourse instance” method and now I’m facing this. Tried every trick in the book (the Sidekiq Job, the Rebake)… is there a “lead” on what may be causing this? Just for me to try to find out something.

(One thing I have to give you all, with all this I’m moving from “yeah I kind of get this” to PostGreSQL PhD, Redis PhD,… Just need to get the hang of Ruby and local dev envs and I may turn useful to the Community )

[Overgrow (OG)]

Have all avatars disappeared or just part of them? Custom avatars are basically Uploads, do other uploads work as expected?

Go to the rails console and check the record of non-functional avatar in database… Do they have correct URL, filesize, width, height, extension?

User.find_by_username('Overgrow').user_avatar
User.find_by_username('Overgrow').uploaded_avatar

They also need to have their optimized versions present… You can check that by:

OptimizedImage.where(upload_id: upload_id).where(version: 2)

[Iceman]

First of all, thank you very much for your help @Overgrow.

All of the avatars and emojis (and even site images, like headers, etc) were “there” but invisible. For non-avatar stuff they appear like broken, for avatars, the gray placeholder. Some people have been able to just upload a new one and those ones you can see.

First tries running the command I got:

FATAL:  the database system is in recovery mode

So… there is that (I do have a lot of “disconnects”, so I’m assuming it has something to do with the DB, maybe?)

But after persisting eventually:

User.find_by_username(‘Overgrow’).user_avatar

=> #<UserAvatar:0x000055702722d200
 id: 4,
 user_id: 3,
 custom_upload_id: 20504,
 gravatar_upload_id: 12240,
 last_gravatar_download_attempt: Thu, 21 May 2020 10:16:55 UTC +00:00,
 created_at: Sat, 30 May 2019 16:33:16 UTC +00:00,
 updated_at: Thu, 21 May 2020 10:16:55 UTC +00:00>

(Tried to upload a new one today but It doesn’t work).

User.find_by_username(‘Overgrow’).uploaded_avatar

=> #<Upload:0x00005555cd911b58
 id: 20504,
 user_id: 3,
 original_filename: "16_2.png.jpg",
 filesize: 56220,
 width: 360,
 height: 360,
 url: "/uploads/default/original/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8.jpeg",
 created_at: Thu, 15 Aug 2019 20:02:47 UTC +00:00,
 updated_at: Thu, 15 Aug 2019 20:02:47 UTC +00:00,
 sha1: "63347a46c0ca945f53613722a73c233484d642c8",
 origin: nil,
 retain_hours: nil,
 extension: "jpeg",
 thumbnail_width: 360,
 thumbnail_height: 360,
 etag: nil,
 secure: false,
 access_control_post_id: nil,
 original_sha1: nil>

OptimizedImage.where(upload_id: 20504).where(version: 2)

=> [#<OptimizedImage:0x000056366a01c1a0
  id: 95962,
  sha1: "5a32b5cc3e6f5c58d88a3c92a23076980a8ce840",
  extension: ".jpeg",
  width: 200,
  height: 200,
  upload_id: 20504,
  url: "/uploads/default/optimized/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8_2_200x200.jpeg",
  filesize: 28916,
  etag: nil,
  version: 2>,
 #<OptimizedImage:0x000056366a0741e8
  id: 95942,
  sha1: "ee353c9e23511b471e1a59c1f71a2ded3e366b1e",
  extension: ".jpeg",
  width: 20,
  height: 20,
  upload_id: 20504,
  url: "/uploads/default/optimized/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8_2_20x20.jpeg",
  filesize: 1270,
  etag: nil,
  version: 2>,
 #<OptimizedImage:0x000056366a074120
  id: 95943,
  sha1: "944fa9fc542a79a5c50394c75022bf84ace297e5",
  extension: ".jpeg",
  width: 30,
  height: 30,
  upload_id: 20504,
  url: "/uploads/default/optimized/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8_2_30x30.jpeg",
  filesize: 1952,
  etag: nil,
  version: 2>,
 #<OptimizedImage:0x000056366a074058
  id: 95944,
  sha1: "983490e58bed58c971ffa44e440b02ce3ea72bba",
  extension: ".jpeg",
  width: 40,
  height: 40,
  upload_id: 20504,
  url: "/uploads/default/optimized/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8_2_40x40.jpeg",
  filesize: 2695,
  etag: nil,
  version: 2>,
 #<OptimizedImage:0x000056366a07bf60

So apparently the images are there, but they won’t show. Just the gray default avatar placeholder.

[Overgrow (OG)]

Everything looks ok on db record level. You can move into levels above when investigating.

What do you get when you manually follow URLs of uploads you listed?

[Iceman]

@Iceman

/uploads/default/optimized/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8_2_200x200.jpeg

If I put the /uploads/default/optimized/3X/6/3/63347a46c0ca945f53613722a73c233484d642c8_2_200x200.jpeg (ie) after my Discourse’s URL? I get a 404, not found.

So… they don’t exist? (I’m asking with hope )

[Overgrow (OG)]

Check also some file URLs from /uploads/default/original not only from /uploads/default/optimized.

404 … That means that you need to check your uploads folder inside /var/discourse/shared/standalone on filesystem and look up where the actual old files are (if they exist). When you find them, try to compare location to newly uploaded files (those that work).

You can also restore them from the backup manually.

[Iceman]

Thanks for the explanation.

Just went in there, double checked, some of the paths listed do not exists.

The weird part is that I have people trying to upload new ones and those new ones don’t work also, when you go look with the commands you gave me you can see a path that also doesn’t exist. How does Discourse “map” this? Because one thing that I can quite get is that the files are missing (even though the backup was supposed to carry them over), but new uploads going to phantom paths?

[riking (Kane York)]

Check the tombstone folder inside your uploads folder - are some of the missing files there?

[Iceman]

The only folder I see inside uploads is default… the tombstone folder is for deprecated files or something like that?

Also, an extra piece of Info: Turns out that if a user tries to upload the same image they already had (even if they change the name of the file, I assume based on what I can see with those queries that is based on the hash) the image won’t load, will appear empty, once you save you will see the grey placeholder.

Apparently if you modify the image somehow (even if just saving it as a different format in Photoshop) then you can re-upload it.

[Overgrow (OG)]

That is normal behaviour. Hash of the data file is stored in db to avoid duplicate images.

What happens if you upload picture through compose editor? Does it complete upload? Does it appear in preview pane?

[Iceman]

If I write a message and include an image on it? It will show on the preview panel and will complete upload, yes, normal behavior.

[Overgrow (OG)]

@Iceman

I have people trying to upload new ones and those new ones don’t work also, when you go look with the commands you gave me you can see a path that also doesn’t exist.

@Iceman

If I write a message and include an image on it? It will show on the preview panel and will complete upload, yes, normal behavior.

So in which exact point that image stops showing?

Check url of that image you see and track it down to the filesystem.

Check url of images that don’t work (through web developer tools in browser). What is the difference?

Maybe they are pointing to different domain…

[Iceman]

On the first message I meant avatars specifically (through the user’s profile), the second is on the editor.

So on a normal message, if you drag and drop or press the “upload image” button, it will work seamlessly, as normal.

In short:

• Avatars don’t show, just the placeholder.
• Custom Emoji also don’t show.
• Site Images (logo, etc) were also not showing.
• If you upload images on the composer it works ok.
• If you try to upload the same avatar you had before the incident, it wont work. The behavior is: Will upload, then in the box where you select if you want a default letter, gravatar or upload will be shown as a blank square. When you confirm your choice and the page reloads, you will have the grey placeholder.

Also:

• No Tombstone folder.
• The old images are in directories (as shown with the queries you gave me) that don’t exist.

Let me check the domain thing. As a note, when I was reading for info I tried:

• Triggering the CreateMissingAvatars job from Sidekiq -> No success.
• Rebake all posts (yeah, kind of a stretch) -> No success.
• Based on this thread since I did use a different domain (subdomain actually) to test the restore from backup while the main site was offline I thought that maybe some URLs may be wrong, so I ran this: discourse remap talk.foo.com talk.bar.com -> No success.

[neounix (Dark Matter)]

@Iceman

This is mostly FYI, and maybe it is TMI, but it might help you in some small way gain additional insight into some of the interesting things we experience running three containers (two web application and one data container) at the same time (and how it also effects user avatars).

It’s is very interesting (and very cool, in my view), how the Redis / Sidekiq job scheduler works when they are running in parallel, but only one “active on the user web side”:

Hope you find this short discussion with real-world example interesting. It might provide a small amount of insight into the Discourse job scheduler, image optimization, and avatars based on our configuration:

Docker Container Magic with Redis and Sidekiq Job Scheduling

I am a big fan of how Discourse uses Redis / Sidekiq to schedule background jobs; and consider this one of the key strengths and benefits of the Discourse S/W architecture.

Note: These concepts also apply, in various subtle ways, to different stages in the backup and restore process and other processes (time dependent), so it is a good idea to understand how and why sidekiq schedules jobs in the background.

[Iceman]

Thanks for the info @neounix, it’s very helpful in order to understand better the “insides” of Discourse from the perspective of “I’m learning Rails so that I can help but by God it’s being a stiff learning curve while trying to fix your own installation”

I’m focusing on Redis/Sidekiq now to try to understand why some embeds are not working since I think that can be “Bake” related, but I can’t sign under that since I’m still getting there in terms of debug (hopefully).

Regarding my issue here, thanks to @Overgrow I could determine that:

• In fact, the Backup Process did copy the files on the backup but did not restore them into the Discourse Installation. Based on the status and/or fixes of my other three issues I may need to recover another backup on another installation and test again the backup process, but it may be an issue for everyone, don’t know yet.

• Because of that, the weird behavior was happening.

• Ended up opening the backup and injecting the missing files onto the instalaltion. It recovered everything without need for Rebaking.

However, the other issues persist (not being able to rebuild the Data container, which I’m clueless about and two issues that are making me focusing on Sidekiq and events, because they may be solved that way: Some Oneboxes (YouTube, specifically) not working and some notifications about “false” edits happening recursively to some users, even though no edits happened. So I think I the new installation may have some issues on its events, trying to figure it out.