Our discourse overload because of a custom plugin. Please help me

Oh Robert,
We migrate to a new vps with 4 core(Ryzen 9 5950x),
Cpu usage is still almost 100%. I really don’t know what is wrong.
屏幕快照 2022-08-22 16.32.34


I thought you were going to move to an 8 core Scaleway server?

What is the claimed bandwidth of this server? I wonder if low bandwidth could cause slow connections/disconnections to become a bottleneck?

Scaleway have VPS with bandwidths of hundreds of megabits per second, not 30.

30 seems extremely low to me.

This is the spec.

4096MB DDR3 ECC RAM
4 CPU vCores (3.4+ GHz E3-1231 v3/1241 v3/1240 v5)
200GB Raid 10 HDD Storage
5TB bandwidth @ 1Gb/s (speed limited to 5Mb/s after allocation used)
1 IPv4, /64 IPv6
20Gb/s TCP DDoS Protection, 200Gb/s upgrade available
Seattle, Washington, US
Virtualizor Control Panel
Unmanaged

Because of time delay, we finally chose this cloud service provider. They did the optimization of the network for visitors from our country.

By the way, output of htop is


100% CPU usage is abnormal. :fearful:
Everything is fine except load and cpu usage.

Anything strange in access logs?

Oh now it seems like it is calming down now.


We guess restoring from a backup causes a spike in cpu usage.
Anyway, thank you very much Robert!

2 Likes

Oh yes that would do it!

Hum, you should aim for ssd

2 Likes

Good catch Benjamin!

Scaleway and others have NVMe SSDs. Much better!

CPU could be waiting on disk.

1 Like

oh sorry I just pasted a wrong spec.
The spec is

4096MB DDR4 ECC RAM
4 CPU vCores (3.4GHz AMD Ryzen 5950X)
100GB Raid NVMe SSD Storage
5TB bandwidth @ 10Gb/s (speed limited to 5Mb/s after allocation used)
1 IPv4, /64 IPv6
20Gb/s TCP DDoS Protection, 200Gb/s upgrade available
Seattle, Washington, US
Virtualizor Control Panel
Unmanaged

It is NVMe. :flushed:

3 Likes

I think it is possible the plugins we installed have increased the load.
The plugin made such a change to the user model that when we go to discovery route, we always calculate the points of user model in topic model for many times

add_to_class(:user, :total_earned_points) do
    self.user_points.sum(:reward_points)
  end

  add_to_class(:user, :available_points) do
    self.total_earned_points - self.user_rewards.sum(:points)
  end

  add_to_class(:user, :rewards) do
    DiscourseRewards::Reward.where(created_by_id: self.id)
  end

  add_to_serializer(:basic_user, :total_earned_points) do
    user&.total_earned_points
  end

  add_to_serializer(:basic_user, :available_points) do
    user&.available_points
  end

  add_to_serializer(:current_user, :total_earned_points) do
    scope.user.total_earned_points
  end

  add_to_serializer(:current_user, :available_points) do
    scope.user.available_points
  end

  add_to_serializer(:current_user, :user_rewards) do
    scope.user.user_rewards
  end

  add_to_serializer(:current_user, :rewards) do
    scope.user.rewards
  end

I thought you’d said you’d removed all plugins and it was behaving the same?

Does this have an index?

How many rows is user_rewards table? Interesting it is not distinguishing by user … (but I assume it must be somehow beneath this level).

Oh I am afraid it doesn’t set an index.
The model is shown below.

module DiscourseRewards
  class Reward < ActiveRecord::Base
    self.table_name = 'discourse_rewards_rewards'

    has_many :user_rewards
    belongs_to :created_by, class_name: 'User'

    default_scope { where(deleted_at: nil) }
  end
end
module DiscourseRewards
  class UserPoint < ActiveRecord::Base
    self.table_name = 'discourse_rewards_user_points'

    belongs_to :user
    belongs_to :user_badge
    belongs_to :user_points_category

    def self.user_total_points(user)
      UserPoint.where(user_id: user.id).sum(:reward_points)
    end
  end
end
module DiscourseRewards
  class UserReward < ActiveRecord::Base
    self.table_name = 'discourse_rewards_user_rewards'

    belongs_to :user
    belongs_to :reward

    enum status: [:applied, :granted]
  end
end

I disable all plugin on safe mode. However you know, the database didn’t change. :smiling_face_with_tear:

Something to take up with the author I suspect.

Looks like it might not be scaling for your high number of users?

You may need to rebuild without the plugin. It would be a good comparison and maybe confirm if the issues are caused by the plugin or not.

The presence of tables won’t matter if the queries aren’t running.

2 Likes

discourse_rewards_user_points | 233525 | 27 MB | 5152 kB | 32 MB
discourse_rewards_user_rewards | 61 | 16 kB | 16 kB | 32 kB
discourse_rewards_user_points_categories | 9 | 16 kB | 16 kB | 32 kB
discourse_rewards_rewards | 4 | 16 kB | 16 kB | 32 kB

discourse_rewards_user_points table is huge.

1 Like

Completely remove it (from build, no need to destroy the tables), see what happens.

2 Likes

Thank you, I 'll try it right now.

1 Like

Oh Robert, I rebuild without the plugin, response time for pages of discovery route is reduced to less than 200ms from 2-3s before. :fearful:

1 Like

Well there you have it. I would discuss optimisation with the author, or if you have the skills, PR?

1 Like

I am asking for our friend who is familiar with database for help.
But he is working on JAVA. I think it will takes some time.

1 Like