So many broken users due to corrupt database

I have a real issue with many of my users. The users can be seen in admin panel like this:

But when I try to reveal their email address, It wont show anything to me. And also their public profile gives a 404 error.

As far as I see, all of these users are almost inactive (they are active in terms of activation but inactive in terms of activity in forum). So I think maybe its been caused by a broken automatic inactive user removal in the very long past.

1 Like

Another thing I notice:

If I inactivate, and then activate them, they get fixed!

2 Likes

If a user is inactive for a certain length of time (730 days by default) they are automatically inactivated. The setting is in your dashboard under Settings/Users and scroll almost down to the bottom. That is where you’ll see that. But, it’s not necessary to make any changes for the sake of making a change. If those users haven’t logged in for 2 years, it makes no sense to reactivate them unless they start showing up again. :wink:

4 Likes

No. My problem is not this. The dashboard says the users are activated.

Seems similar to the issue I’m having. Could you check if the affected names have duplicates or ‘very near matches’ in your database? For example user.user and useruser.

1 Like

Yes affected users are among those usernames that are very popular. so discourse suggest a near match username. I signup users with api. so I get a username from user. then I check it against discourse api and if its already taken, I would automatically use whatever discourse suggest.

1 Like

btw with help of @RGJ we nailed the problem down to this two condisions:

3 Likes

To add to that, the issue @hosna is having is clearly an issue on the database level. It seems like there is some corruption in the users table. Copying the contents to a new table solves these issues.

That said, I did see two occurrences of @bartv’s issue in @hosna’s database (those were the two duplicates before Sept 22 and they both had a dot in their username) but I am not sure if these two issues are related. They just have the same symptoms.

4 Likes

Sounds like a corrupt database index. REINDEX TABLE users should solve the problem.

2 Likes

What about the duplicate usernames? I have many same usernames being used for 2 distinct users.

The issue is acknowledged here:

1 Like

That’s probably a side effect of a corrupt index. You might need to clean that up manually before the reindexing works.

2 Likes

Can you explain how a corrupt index might happen? to prevent it in the future?

Hardware failure, bug in Postgres,… hard to tell. It happens.

Except that that is impossible since the index is broken.
This does the trick:

# create a temporary table without constraints and copy the contents into it
create table users_test (like users);
insert into users_test select * from users;

# remove duplicate usernames case sensitive, the dupes are after Sept 22
delete from users_test where username in (
  select username 
  from users_test 
  group by username 
  having count(username)>1
) and created_at>'2019-09-22' ;

# remove duplicate usernames case insensitive, the dupes are after Sept 22
delete from users_test where lower(username) in (
  select lower(username) 
  from users_test 
  group by lower(username) 
  having count(lower(username))>1
) and created_at > '2019-09-22' ;

# two more problems left, delete them individually
delete from users_test where id in (184534,130826);

# create a new table with constraints and copy the users
create table users_clean (like users including indexes);
insert into users_clean select * from users_test;

and then rename users to users_old and users_clean to users.

3 Likes

I’d like to drop in here and say this might break your database even more than the broken users will!

We now are stuck halfway between upgrades as a lot of constraints are still relying on users_old since we renamed the table, and this was only shown as an issue a few days after applying this apparently incomplete fix, also like users including indexes is not enough (it’ll ignore the id sequence for example).

You are absolutely right, I indeed remember having to recreate the constraints after renaming the tables.
My apologies for this important omission.

From my notes:

alter table poll_votes drop constraint fk_rails_b64de9b025;
alter table poll_votes add constraint fk_rails_b64de9b025 FOREIGN KEY (user_id) REFERENCES users(id);

alter table user_security_keys drop constraint fk_rails_90999b0454;
alter table user_security_keys add  constraint fk_rails_90999b0454 FOREIGN KEY (user_id) REFERENCES users(id);

and nowadays also

alter table bookmarks drop constraint fk_rails_c1ff6fa4ac;
alter table bookmarks add  constraint fk_rails_c1ff6fa4ac FOREIGN KEY (user_id) REFERENCES users(id);

And as an important disclaimer: only use this when you absolutely know what you are doing!

1 Like

This does indeed seem to match what we found after querying pg_catalog for constraints affecting users_old.

Also, I recall including defaults being at minimum required to not break registration.

Thanks for the rectification!

1 Like