Discourse overloaded real traffic or DDOS? 100% CPU usage despite of decent traffic and high specs server

All 8 vCPUs usage on our site is 100%.

Our users get this message often,

Due to extreme load, this is temporarily being shown to everyone as a logged out user would see it.

Our droplet specs are:

CPU-Optimized / 16 GB / 8 vCPUs , Intel Dedicated Digital Ocean Droplet

Traffic:

Online real time users on Google analytics as of writing are approx 70 to 150.

htop snap for CPU and RAM usage is below,

Last 14 days of graph usage from digital ocean:

App.yaml looks like,

db_shared_buffers: “4096MB”
db_work_mem: “400MB”
UNICORN_WORKERS: 16
#- “templates/web.ratelimited.template.yml” #removed this to avoid 429 too many requests errors Remove the ip limits

As you can see, we have commented the ratelimited template because it was giving 429 nginx too many request errors to our normal users despite of very low actual traffic on our site.

CF firewall page looks like,

Launcher log looks like, ./launcher logs app

Prometheus Metrics look like,

# HELP discourse_collector_working Is the master process collector able to collect metrics
# TYPE discourse_collector_working gauge
discourse_collector_working 1


# HELP discourse_collector_rss total memory used by collector process
# TYPE discourse_collector_rss gauge
discourse_collector_rss 46530560


# HELP discourse_collector_metrics_total Total metrics processed by exporter web.
# TYPE discourse_collector_metrics_total counter
discourse_collector_metrics_total 215465


# HELP discourse_collector_sessions_total Total send_metric sessions processed by exporter web.
# TYPE discourse_collector_sessions_total counter
discourse_collector_sessions_total 7499


# HELP discourse_collector_bad_metrics_total Total mis-handled metrics by collector.
# TYPE discourse_collector_bad_metrics_total counter
discourse_collector_bad_metrics_total 0

# HELP discourse_page_views Page views reported by admin dashboard
# TYPE discourse_page_views counter
discourse_page_views{db="default",type="crawler",device="crawler"} 128620
discourse_page_views{db="default",type="anon",device="mobile"} 2840
discourse_page_views{db="default",type="anon",device="desktop"} 4094
discourse_page_views{db="default",type="logged_in",device="mobile"} 55
discourse_page_views{db="default",type="logged_in",device="desktop"} 64

# HELP discourse_http_requests Total HTTP requests from web app
# TYPE discourse_http_requests counter
discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="200"} 166429
discourse_http_requests{db="default",api="web",verb="POST",type="background",status="429"} 19628
discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="404"} 3945
discourse_http_requests{db="default",api="web",verb="HEAD",type="regular",status="200"} 154
discourse_http_requests{db="default",api="web",verb="POST",type="regular",status="200"} 2232
discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="301"} 152
discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="304"} 2295
discourse_http_requests{db="default",api="admin",verb="POST",type="regular",status="200"} 97
discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="409"} 141
discourse_http_requests{db="default",api="web",verb="POST",type="background",status="200"} 7540
discourse_http_requests{db="default",api="web",verb="POST",type="background",status="-1"} 1985
discourse_http_requests{db="default",api="admin",verb="POST",type="regular",status="422"} 3
discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="500"} 1
discourse_http_requests{db="default",api="web",verb="POST",type="regular",status="302"} 3
discourse_http_requests{db="default",api="web",verb="PUT",type="regular",status="200"} 7
discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="302"} 2

# HELP discourse_http_duration_seconds Time spent in HTTP reqs in seconds
# TYPE discourse_http_duration_seconds summary
discourse_http_duration_seconds{controller="other",action="other",quantile="0.99"} 0.942478418815881
discourse_http_duration_seconds{controller="other",action="other",quantile="0.9"} 0.05924026109278202
discourse_http_duration_seconds{controller="other",action="other",quantile="0.5"} 0.000657188706099987
discourse_http_duration_seconds{controller="other",action="other",quantile="0.1"} 0.0004582637920975685
discourse_http_duration_seconds{controller="other",action="other",quantile="0.01"} 0.0004117507487535477
discourse_http_duration_seconds_sum{controller="other",action="other"} 3942.3923240867443
discourse_http_duration_seconds_count{controller="other",action="other"} 58870
discourse_http_duration_seconds{controller="topics",action="show",quantile="0.99"} 4.58280345518142
discourse_http_duration_seconds{controller="topics",action="show",quantile="0.9"} 2.827586194965988
discourse_http_duration_seconds{controller="topics",action="show",quantile="0.5"} 0.01292009698227048
discourse_http_duration_seconds{controller="topics",action="show",quantile="0.1"} 0.002048521768301725
discourse_http_duration_seconds{controller="topics",action="show",quantile="0.01"} 0.001819557975977659
discourse_http_duration_seconds_sum{controller="topics",action="show"} 128537.17967935232
discourse_http_duration_seconds_count{controller="topics",action="show"} 143926
discourse_http_duration_seconds{controller="list",action="latest",quantile="0.99"} 5.726923605892807
discourse_http_duration_seconds{controller="list",action="latest",quantile="0.9"} 5.168957637157291
discourse_http_duration_seconds{controller="list",action="latest",quantile="0.5"} 0.009740645997226238
discourse_http_duration_seconds{controller="list",action="latest",quantile="0.1"} 0.003163302782922983
discourse_http_duration_seconds{controller="list",action="latest",quantile="0.01"} 0.002744518686085939
discourse_http_duration_seconds_sum{controller="list",action="latest"} 3052.3646773528308
discourse_http_duration_seconds_count{controller="list",action="latest"} 1676
discourse_http_duration_seconds{controller="users",action="show",quantile="0.99"} 0.1768659572117031
discourse_http_duration_seconds{controller="users",action="show",quantile="0.9"} 0.1768659572117031
discourse_http_duration_seconds{controller="users",action="show",quantile="0.5"} 0.1768659572117031
discourse_http_duration_seconds{controller="users",action="show",quantile="0.1"} 0.1768659572117031
discourse_http_duration_seconds{controller="users",action="show",quantile="0.01"} 0.1768659572117031
discourse_http_duration_seconds_sum{controller="users",action="show"} 22.969309298321605
discourse_http_duration_seconds_count{controller="users",action="show"} 142

# HELP discourse_http_redis_duration_seconds Time spent in HTTP reqs in redis seconds
# TYPE discourse_http_redis_duration_seconds summary
discourse_http_redis_duration_seconds{controller="other",action="other",quantile="0.99"} 0.02405449468642473
discourse_http_redis_duration_seconds{controller="other",action="other",quantile="0.9"} 0.0092271463945508
discourse_http_redis_duration_seconds{controller="other",action="other",quantile="0.5"} 0.0
discourse_http_redis_duration_seconds{controller="other",action="other",quantile="0.1"} 0.0
discourse_http_redis_duration_seconds{controller="other",action="other",quantile="0.01"} 0.0
discourse_http_redis_duration_seconds_sum{controller="other",action="other"} 177.52055060770363
discourse_http_redis_duration_seconds_count{controller="other",action="other"} 58870
discourse_http_redis_duration_seconds{controller="topics",action="show",quantile="0.99"} 0.064909300301224
discourse_http_redis_duration_seconds{controller="topics",action="show",quantile="0.9"} 0.03797017782926559
discourse_http_redis_duration_seconds{controller="topics",action="show",quantile="0.5"} 0.007125054951757193
discourse_http_redis_duration_seconds{controller="topics",action="show",quantile="0.1"} 0.0005201729945838451
discourse_http_redis_duration_seconds{controller="topics",action="show",quantile="0.01"} 0.0004380182363092899
discourse_http_redis_duration_seconds_sum{controller="topics",action="show"} 1814.5181179852225
discourse_http_redis_duration_seconds_count{controller="topics",action="show"} 143924
discourse_http_redis_duration_seconds{controller="list",action="latest",quantile="0.99"} 0.03446856886148453
discourse_http_redis_duration_seconds{controller="list",action="latest",quantile="0.9"} 0.02759678149595857
discourse_http_redis_duration_seconds{controller="list",action="latest",quantile="0.5"} 0.004360581748187542
discourse_http_redis_duration_seconds{controller="list",action="latest",quantile="0.1"} 0.001454399898648262
discourse_http_redis_duration_seconds{controller="list",action="latest",quantile="0.01"} 0.00108026061207056
discourse_http_redis_duration_seconds_sum{controller="list",action="latest"} 20.32011368824169
discourse_http_redis_duration_seconds_count{controller="list",action="latest"} 1676
discourse_http_redis_duration_seconds{controller="users",action="show",quantile="0.99"} 0.01075387001037598
discourse_http_redis_duration_seconds{controller="users",action="show",quantile="0.9"} 0.01075387001037598
discourse_http_redis_duration_seconds{controller="users",action="show",quantile="0.5"} 0.01075387001037598
discourse_http_redis_duration_seconds{controller="users",action="show",quantile="0.1"} 0.01075387001037598
discourse_http_redis_duration_seconds{controller="users",action="show",quantile="0.01"} 0.01075387001037598
discourse_http_redis_duration_seconds_sum{controller="users",action="show"} 1.3520850692875683
discourse_http_redis_duration_seconds_count{controller="users",action="show"} 142

# HELP discourse_http_sql_duration_seconds Time spent in HTTP reqs in SQL in seconds
# TYPE discourse_http_sql_duration_seconds summary
discourse_http_sql_duration_seconds{controller="other",action="other",quantile="0.99"} 0.1874205535277724
discourse_http_sql_duration_seconds{controller="other",action="other",quantile="0.9"} 0.009177460800856352
discourse_http_sql_duration_seconds{controller="other",action="other",quantile="0.5"} 0.0
discourse_http_sql_duration_seconds{controller="other",action="other",quantile="0.1"} 0.0
discourse_http_sql_duration_seconds{controller="other",action="other",quantile="0.01"} 0.0
discourse_http_sql_duration_seconds_sum{controller="other",action="other"} 375.05954352673143
discourse_http_sql_duration_seconds_count{controller="other",action="other"} 58870
discourse_http_sql_duration_seconds{controller="topics",action="show",quantile="0.99"} 0.4393073697574437
discourse_http_sql_duration_seconds{controller="topics",action="show",quantile="0.9"} 0.2883010846562684
discourse_http_sql_duration_seconds{controller="topics",action="show",quantile="0.5"} 0.0
discourse_http_sql_duration_seconds{controller="topics",action="show",quantile="0.1"} 0.0
discourse_http_sql_duration_seconds{controller="topics",action="show",quantile="0.01"} 0.0
discourse_http_sql_duration_seconds_sum{controller="topics",action="show"} 13847.843351929914
discourse_http_sql_duration_seconds_count{controller="topics",action="show"} 143926
discourse_http_sql_duration_seconds{controller="list",action="latest",quantile="0.99"} 0.6733640311285853
discourse_http_sql_duration_seconds{controller="list",action="latest",quantile="0.9"} 0.5781291825696826
discourse_http_sql_duration_seconds{controller="list",action="latest",quantile="0.5"} 0.0
discourse_http_sql_duration_seconds{controller="list",action="latest",quantile="0.1"} 0.0
discourse_http_sql_duration_seconds{controller="list",action="latest",quantile="0.01"} 0.0
discourse_http_sql_duration_seconds_sum{controller="list",action="latest"} 324.1082574711181
discourse_http_sql_duration_seconds_count{controller="list",action="latest"} 1676
discourse_http_sql_duration_seconds{controller="users",action="show",quantile="0.99"} 0.07832226948812604
discourse_http_sql_duration_seconds{controller="users",action="show",quantile="0.9"} 0.07832226948812604
discourse_http_sql_duration_seconds{controller="users",action="show",quantile="0.5"} 0.07832226948812604
discourse_http_sql_duration_seconds{controller="users",action="show",quantile="0.1"} 0.07832226948812604
discourse_http_sql_duration_seconds{controller="users",action="show",quantile="0.01"} 0.07832226948812604
discourse_http_sql_duration_seconds_sum{controller="users",action="show"} 9.40852727741003
discourse_http_sql_duration_seconds_count{controller="users",action="show"} 142

# HELP discourse_http_net_duration_seconds Time spent in external network requests
# TYPE discourse_http_net_duration_seconds summary
discourse_http_net_duration_seconds{controller="other",action="other",quantile="0.99"} 0.0
discourse_http_net_duration_seconds{controller="other",action="other",quantile="0.9"} 0.0
discourse_http_net_duration_seconds{controller="other",action="other",quantile="0.5"} 0.0
discourse_http_net_duration_seconds{controller="other",action="other",quantile="0.1"} 0.0
discourse_http_net_duration_seconds{controller="other",action="other",quantile="0.01"} 0.0
discourse_http_net_duration_seconds_sum{controller="other",action="other"} 349.2784005794674
discourse_http_net_duration_seconds_count{controller="other",action="other"} 58870
discourse_http_net_duration_seconds{controller="topics",action="show",quantile="0.99"} 0.0
discourse_http_net_duration_seconds{controller="topics",action="show",quantile="0.9"} 0.0
discourse_http_net_duration_seconds{controller="topics",action="show",quantile="0.5"} 0.0
discourse_http_net_duration_seconds{controller="topics",action="show",quantile="0.1"} 0.0
discourse_http_net_duration_seconds{controller="topics",action="show",quantile="0.01"} 0.0
discourse_http_net_duration_seconds_sum{controller="topics",action="show"} 0.0
discourse_http_net_duration_seconds_count{controller="topics",action="show"} 143926
discourse_http_net_duration_seconds{controller="list",action="latest",quantile="0.99"} 0.0
discourse_http_net_duration_seconds{controller="list",action="latest",quantile="0.9"} 0.0
discourse_http_net_duration_seconds{controller="list",action="latest",quantile="0.5"} 0.0
discourse_http_net_duration_seconds{controller="list",action="latest",quantile="0.1"} 0.0
discourse_http_net_duration_seconds{controller="list",action="latest",quantile="0.01"} 0.0
discourse_http_net_duration_seconds_sum{controller="list",action="latest"} 0.0
discourse_http_net_duration_seconds_count{controller="list",action="latest"} 1676
discourse_http_net_duration_seconds{controller="users",action="show",quantile="0.99"} 0.0
discourse_http_net_duration_seconds{controller="users",action="show",quantile="0.9"} 0.0
discourse_http_net_duration_seconds{controller="users",action="show",quantile="0.5"} 0.0
discourse_http_net_duration_seconds{controller="users",action="show",quantile="0.1"} 0.0
discourse_http_net_duration_seconds{controller="users",action="show",quantile="0.01"} 0.0
discourse_http_net_duration_seconds_sum{controller="users",action="show"} 0.0
discourse_http_net_duration_seconds_count{controller="users",action="show"} 142

# HELP discourse_http_queue_duration_seconds Time spent queueing requests between NGINX and Ruby
# TYPE discourse_http_queue_duration_seconds summary
discourse_http_queue_duration_seconds{controller="other",action="other",quantile="0.99"} 7.055693626403809
discourse_http_queue_duration_seconds{controller="other",action="other",quantile="0.9"} 5.77228569984436
discourse_http_queue_duration_seconds{controller="other",action="other",quantile="0.5"} 3.628639221191406
discourse_http_queue_duration_seconds{controller="other",action="other",quantile="0.1"} 1.691039323806763
discourse_http_queue_duration_seconds{controller="other",action="other",quantile="0.01"} 0.1517226696014404
discourse_http_queue_duration_seconds_sum{controller="other",action="other"} 128979.19509196281
discourse_http_queue_duration_seconds_count{controller="other",action="other"} 58870
discourse_http_queue_duration_seconds{controller="topics",action="show",quantile="0.99"} 6.773485660552979
discourse_http_queue_duration_seconds{controller="topics",action="show",quantile="0.9"} 5.540528535842896
discourse_http_queue_duration_seconds{controller="topics",action="show",quantile="0.5"} 3.386407375335693
discourse_http_queue_duration_seconds{controller="topics",action="show",quantile="0.1"} 1.529008388519287
discourse_http_queue_duration_seconds{controller="topics",action="show",quantile="0.01"} 0.3203737735748291
discourse_http_queue_duration_seconds_sum{controller="topics",action="show"} 358491.9590744972
discourse_http_queue_duration_seconds_count{controller="topics",action="show"} 143926
discourse_http_queue_duration_seconds{controller="list",action="latest",quantile="0.99"} 7.470051765441895
discourse_http_queue_duration_seconds{controller="list",action="latest",quantile="0.9"} 5.970950841903687
discourse_http_queue_duration_seconds{controller="list",action="latest",quantile="0.5"} 3.233286619186401
discourse_http_queue_duration_seconds{controller="list",action="latest",quantile="0.1"} 1.421695709228516
discourse_http_queue_duration_seconds{controller="list",action="latest",quantile="0.01"} 0.4568729400634766
discourse_http_queue_duration_seconds_sum{controller="list",action="latest"} 5290.475468635559
discourse_http_queue_duration_seconds_count{controller="list",action="latest"} 1676
discourse_http_queue_duration_seconds{controller="users",action="show",quantile="0.99"} 5.078022718429565
discourse_http_queue_duration_seconds{controller="users",action="show",quantile="0.9"} 5.078022718429565
discourse_http_queue_duration_seconds{controller="users",action="show",quantile="0.5"} 5.078022718429565
discourse_http_queue_duration_seconds{controller="users",action="show",quantile="0.1"} 5.078022718429565
discourse_http_queue_duration_seconds{controller="users",action="show",quantile="0.01"} 5.078022718429565
discourse_http_queue_duration_seconds_sum{controller="users",action="show"} 381.56648421287537
discourse_http_queue_duration_seconds_count{controller="users",action="show"} 142

# HELP discourse_http_forced_anon_count Total count of logged in requests forced into anonymous mode
# TYPE discourse_http_forced_anon_count counter
discourse_http_forced_anon_count{db="default",api="web",verb="GET",type="regular",status="200"} 38907
discourse_http_forced_anon_count{db="default",api="web",verb="GET",type="regular",status="404"} 1236

# HELP discourse_http_sql_calls_per_request How many SQL statements ran per request
# TYPE discourse_http_sql_calls_per_request gauge
discourse_http_sql_calls_per_request{controller="users",action="show",logged_in="false"} 30
discourse_http_sql_calls_per_request{controller="list",action="latest",logged_in="true"} 271
discourse_http_sql_calls_per_request{controller="topics",action="show",logged_in="true"} 164
discourse_http_sql_calls_per_request{controller="users",action="show",logged_in="true"} 98
discourse_http_sql_calls_per_request{controller="list",action="latest",logged_in="false"} 223
discourse_http_sql_calls_per_request{controller="other",action="other",logged_in="false"} 6
discourse_http_sql_calls_per_request{controller="topics",action="show",logged_in="false"} 136

# HELP discourse_http_anon_cache_store How many a payload is stored in redis for anonymous cache
# TYPE discourse_http_anon_cache_store counter
discourse_http_anon_cache_store{controller="topics",action="show"} 21358
discourse_http_anon_cache_store{controller="other",action="other"} 260
discourse_http_anon_cache_store{controller="list",action="latest"} 286

# HELP discourse_http_anon_cache_hit How many a payload from redis is used for anonymous cache
# TYPE discourse_http_anon_cache_hit counter
discourse_http_anon_cache_hit{controller="other",action="other"} 7507
discourse_http_anon_cache_hit{controller="topics",action="show"} 82761
discourse_http_anon_cache_hit{controller="list",action="latest"} 929

# HELP discourse_heap_free_slots Free ruby heap slots
# TYPE discourse_heap_free_slots gauge
discourse_heap_free_slots{type="sidekiq",pid="131"} 12166
discourse_heap_free_slots{type="web",pid="163"} 119712
discourse_heap_free_slots{type="web",pid="183"} 69322
discourse_heap_free_slots{type="web",pid="454"} 31247
discourse_heap_free_slots{type="web",pid="201"} 162260
discourse_heap_free_slots{type="web",pid="503"} 83322
discourse_heap_free_slots{type="web",pid="308"} 72679
discourse_heap_free_slots{type="web",pid="328"} 79991
discourse_heap_free_slots{type="web",pid="396"} 204322
discourse_heap_free_slots{type="web",pid="219"} 221194
discourse_heap_free_slots{type="web",pid="374"} 122163
discourse_heap_free_slots{type="web",pid="270"} 43722
discourse_heap_free_slots{type="web",pid="418"} 155036
discourse_heap_free_slots{type="web",pid="434"} 158153
discourse_heap_free_slots{type="web",pid="479"} 116781
discourse_heap_free_slots{type="web",pid="350"} 167763
discourse_heap_free_slots{type="web",pid="287"} 95786

# HELP discourse_heap_live_slots Used ruby heap slots
# TYPE discourse_heap_live_slots gauge
discourse_heap_live_slots{type="sidekiq",pid="131"} 883728
discourse_heap_live_slots{type="web",pid="163"} 816121
discourse_heap_live_slots{type="web",pid="183"} 866514
discourse_heap_live_slots{type="web",pid="454"} 904588
discourse_heap_live_slots{type="web",pid="201"} 773576
discourse_heap_live_slots{type="web",pid="503"} 852518
discourse_heap_live_slots{type="web",pid="308"} 903104
discourse_heap_live_slots{type="web",pid="328"} 855035
discourse_heap_live_slots{type="web",pid="396"} 731505
discourse_heap_live_slots{type="web",pid="219"} 714638
discourse_heap_live_slots{type="web",pid="374"} 810818
discourse_heap_live_slots{type="web",pid="270"} 892104
discourse_heap_live_slots{type="web",pid="418"} 780793
discourse_heap_live_slots{type="web",pid="434"} 777678
discourse_heap_live_slots{type="web",pid="479"} 819045
discourse_heap_live_slots{type="web",pid="350"} 768077
discourse_heap_live_slots{type="web",pid="287"} 840051

# HELP discourse_v8_heap_size Total JavaScript V8 heap size (bytes)
# TYPE discourse_v8_heap_size gauge
discourse_v8_heap_size{type="sidekiq",pid="131"} 16392192
discourse_v8_heap_size{type="web",pid="163"} 15900672
discourse_v8_heap_size{type="web",pid="183"} 16424960
discourse_v8_heap_size{type="web",pid="454"} 17702912
discourse_v8_heap_size{type="web",pid="201"} 16433152
discourse_v8_heap_size{type="web",pid="503"} 17440768
discourse_v8_heap_size{type="web",pid="308"} 17178624
discourse_v8_heap_size{type="web",pid="328"} 16424960
discourse_v8_heap_size{type="web",pid="396"} 16162816
discourse_v8_heap_size{type="web",pid="219"} 16171008
discourse_v8_heap_size{type="web",pid="374"} 16171008
discourse_v8_heap_size{type="web",pid="270"} 17219584
discourse_v8_heap_size{type="web",pid="418"} 16957440
discourse_v8_heap_size{type="web",pid="434"} 15908864
discourse_v8_heap_size{type="web",pid="479"} 16654336
discourse_v8_heap_size{type="web",pid="350"} 16949248
discourse_v8_heap_size{type="web",pid="287"} 16654336

# HELP discourse_v8_used_heap_size Total used JavaScript V8 heap size (bytes)
# TYPE discourse_v8_used_heap_size gauge
discourse_v8_used_heap_size{type="sidekiq",pid="131"} 12932148
discourse_v8_used_heap_size{type="web",pid="163"} 12252448
discourse_v8_used_heap_size{type="web",pid="183"} 12526820
discourse_v8_used_heap_size{type="web",pid="454"} 12993364
discourse_v8_used_heap_size{type="web",pid="201"} 12462852
discourse_v8_used_heap_size{type="web",pid="503"} 12626676
discourse_v8_used_heap_size{type="web",pid="308"} 12702604
discourse_v8_used_heap_size{type="web",pid="328"} 12126824
discourse_v8_used_heap_size{type="web",pid="396"} 12269764
discourse_v8_used_heap_size{type="web",pid="219"} 12293912
discourse_v8_used_heap_size{type="web",pid="374"} 12210560
discourse_v8_used_heap_size{type="web",pid="270"} 12628080
discourse_v8_used_heap_size{type="web",pid="418"} 12538324
discourse_v8_used_heap_size{type="web",pid="434"} 12179620
discourse_v8_used_heap_size{type="web",pid="479"} 12715960
discourse_v8_used_heap_size{type="web",pid="350"} 12362024
discourse_v8_used_heap_size{type="web",pid="287"} 12654196

# HELP discourse_v8_physical_size Physical size consumed by V8 heaps
# TYPE discourse_v8_physical_size gauge
discourse_v8_physical_size{type="sidekiq",pid="131"} 14118508
discourse_v8_physical_size{type="web",pid="163"} 13315992
discourse_v8_physical_size{type="web",pid="183"} 13639620
discourse_v8_physical_size{type="web",pid="454"} 14201860
discourse_v8_physical_size{type="web",pid="201"} 13543736
discourse_v8_physical_size{type="web",pid="503"} 13779544
discourse_v8_physical_size{type="web",pid="308"} 13836520
discourse_v8_physical_size{type="web",pid="328"} 13250748
discourse_v8_physical_size{type="web",pid="396"} 13379408
discourse_v8_physical_size{type="web",pid="219"} 13401800
discourse_v8_physical_size{type="web",pid="374"} 13285780
discourse_v8_physical_size{type="web",pid="270"} 13776112
discourse_v8_physical_size{type="web",pid="418"} 13634224
discourse_v8_physical_size{type="web",pid="434"} 13221264
discourse_v8_physical_size{type="web",pid="479"} 13876884
discourse_v8_physical_size{type="web",pid="350"} 13471300
discourse_v8_physical_size{type="web",pid="287"} 13738052

# HELP discourse_v8_heap_count Number of V8 contexts running
# TYPE discourse_v8_heap_count gauge
discourse_v8_heap_count{type="sidekiq",pid="131"} 2
discourse_v8_heap_count{type="web",pid="163"} 2
discourse_v8_heap_count{type="web",pid="183"} 2
discourse_v8_heap_count{type="web",pid="454"} 2
discourse_v8_heap_count{type="web",pid="201"} 2
discourse_v8_heap_count{type="web",pid="503"} 2
discourse_v8_heap_count{type="web",pid="308"} 2
discourse_v8_heap_count{type="web",pid="328"} 2
discourse_v8_heap_count{type="web",pid="396"} 2
discourse_v8_heap_count{type="web",pid="219"} 2
discourse_v8_heap_count{type="web",pid="374"} 2
discourse_v8_heap_count{type="web",pid="270"} 2
discourse_v8_heap_count{type="web",pid="418"} 2
discourse_v8_heap_count{type="web",pid="434"} 2
discourse_v8_heap_count{type="web",pid="479"} 2
discourse_v8_heap_count{type="web",pid="350"} 2
discourse_v8_heap_count{type="web",pid="287"} 2

# HELP discourse_rss Total RSS used by process
# TYPE discourse_rss gauge
discourse_rss{type="sidekiq",pid="131"} 412172288
discourse_rss{type="web",pid="163"} 481796096
discourse_rss{type="web",pid="183"} 485736448
discourse_rss{type="web",pid="454"} 432590848
discourse_rss{type="web",pid="201"} 408793088
discourse_rss{type="web",pid="503"} 400375808
discourse_rss{type="web",pid="308"} 444911616
discourse_rss{type="web",pid="328"} 418308096
discourse_rss{type="web",pid="396"} 413487104
discourse_rss{type="web",pid="219"} 400482304
discourse_rss{type="web",pid="374"} 440221696
discourse_rss{type="web",pid="270"} 501215232
discourse_rss{type="web",pid="418"} 479678464
discourse_rss{type="web",pid="434"} 431116288
discourse_rss{type="web",pid="479"} 428220416
discourse_rss{type="web",pid="350"} 430444544
discourse_rss{type="web",pid="287"} 428658688

# HELP discourse_thread_count Total number of active threads per process
# TYPE discourse_thread_count gauge
discourse_thread_count{type="sidekiq",pid="131"} 22
discourse_thread_count{type="web",pid="163"} 8
discourse_thread_count{type="web",pid="183"} 8
discourse_thread_count{type="web",pid="454"} 8
discourse_thread_count{type="web",pid="201"} 8
discourse_thread_count{type="web",pid="503"} 8
discourse_thread_count{type="web",pid="308"} 8
discourse_thread_count{type="web",pid="328"} 8
discourse_thread_count{type="web",pid="396"} 8
discourse_thread_count{type="web",pid="219"} 8
discourse_thread_count{type="web",pid="374"} 8
discourse_thread_count{type="web",pid="270"} 8
discourse_thread_count{type="web",pid="418"} 8
discourse_thread_count{type="web",pid="434"} 8
discourse_thread_count{type="web",pid="479"} 8
discourse_thread_count{type="web",pid="350"} 8
discourse_thread_count{type="web",pid="287"} 8

# HELP discourse_deferred_jobs_queued Number of jobs queued in the deferred job queue
# TYPE discourse_deferred_jobs_queued gauge
discourse_deferred_jobs_queued{type="sidekiq",pid="131"} 0
discourse_deferred_jobs_queued{type="web",pid="163"} 0
discourse_deferred_jobs_queued{type="web",pid="183"} 0
discourse_deferred_jobs_queued{type="web",pid="454"} 0
discourse_deferred_jobs_queued{type="web",pid="201"} 0
discourse_deferred_jobs_queued{type="web",pid="503"} 0
discourse_deferred_jobs_queued{type="web",pid="308"} 0
discourse_deferred_jobs_queued{type="web",pid="328"} 0
discourse_deferred_jobs_queued{type="web",pid="396"} 0
discourse_deferred_jobs_queued{type="web",pid="219"} 0
discourse_deferred_jobs_queued{type="web",pid="374"} 0
discourse_deferred_jobs_queued{type="web",pid="270"} 0
discourse_deferred_jobs_queued{type="web",pid="418"} 0
discourse_deferred_jobs_queued{type="web",pid="434"} 0
discourse_deferred_jobs_queued{type="web",pid="479"} 0
discourse_deferred_jobs_queued{type="web",pid="350"} 0
discourse_deferred_jobs_queued{type="web",pid="287"} 0

# HELP discourse_active_record_connections_count Total number of connections in ActiveRecord's connection pools
# TYPE discourse_active_record_connections_count gauge
discourse_active_record_connections_count{type="sidekiq",pid="131",status="busy"} 2
discourse_active_record_connections_count{type="sidekiq",pid="131",status="dead"} 0
discourse_active_record_connections_count{type="sidekiq",pid="131",status="idle"} 4
discourse_active_record_connections_count{type="sidekiq",pid="131",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="163",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="163",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="163",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="163",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="183",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="183",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="183",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="183",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="454",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="454",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="454",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="454",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="201",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="201",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="201",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="201",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="503",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="503",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="503",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="503",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="308",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="308",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="308",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="308",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="328",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="328",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="328",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="328",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="396",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="396",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="396",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="396",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="219",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="219",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="219",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="219",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="374",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="374",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="374",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="374",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="270",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="270",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="270",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="270",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="418",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="418",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="418",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="418",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="434",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="434",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="434",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="434",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="479",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="479",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="479",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="479",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="350",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="350",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="350",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="350",status="waiting"} 0
discourse_active_record_connections_count{type="web",pid="287",status="busy"} 1
discourse_active_record_connections_count{type="web",pid="287",status="dead"} 0
discourse_active_record_connections_count{type="web",pid="287",status="idle"} 1
discourse_active_record_connections_count{type="web",pid="287",status="waiting"} 0

# HELP discourse_active_record_failover_count Count of ActiveRecord databases in a failover state
# TYPE discourse_active_record_failover_count gauge
discourse_active_record_failover_count{type="sidekiq",pid="131"} 0
discourse_active_record_failover_count{type="web",pid="163"} 0
discourse_active_record_failover_count{type="web",pid="183"} 0
discourse_active_record_failover_count{type="web",pid="454"} 0
discourse_active_record_failover_count{type="web",pid="201"} 0
discourse_active_record_failover_count{type="web",pid="503"} 0
discourse_active_record_failover_count{type="web",pid="308"} 0
discourse_active_record_failover_count{type="web",pid="328"} 0
discourse_active_record_failover_count{type="web",pid="396"} 0
discourse_active_record_failover_count{type="web",pid="219"} 0
discourse_active_record_failover_count{type="web",pid="374"} 0
discourse_active_record_failover_count{type="web",pid="270"} 0
discourse_active_record_failover_count{type="web",pid="418"} 0
discourse_active_record_failover_count{type="web",pid="434"} 0
discourse_active_record_failover_count{type="web",pid="479"} 0
discourse_active_record_failover_count{type="web",pid="350"} 0
discourse_active_record_failover_count{type="web",pid="287"} 0

# HELP discourse_redis_failover_count Count of Redis servers in a failover state
# TYPE discourse_redis_failover_count gauge
discourse_redis_failover_count{type="sidekiq",pid="131"} 0
discourse_redis_failover_count{type="web",pid="163"} 0
discourse_redis_failover_count{type="web",pid="183"} 0
discourse_redis_failover_count{type="web",pid="454"} 0
discourse_redis_failover_count{type="web",pid="201"} 0
discourse_redis_failover_count{type="web",pid="503"} 0
discourse_redis_failover_count{type="web",pid="308"} 0
discourse_redis_failover_count{type="web",pid="328"} 0
discourse_redis_failover_count{type="web",pid="396"} 0
discourse_redis_failover_count{type="web",pid="219"} 0
discourse_redis_failover_count{type="web",pid="374"} 0
discourse_redis_failover_count{type="web",pid="270"} 0
discourse_redis_failover_count{type="web",pid="418"} 0
discourse_redis_failover_count{type="web",pid="434"} 0
discourse_redis_failover_count{type="web",pid="479"} 0
discourse_redis_failover_count{type="web",pid="350"} 0
discourse_redis_failover_count{type="web",pid="287"} 0

# HELP discourse_major_gc_count Major GC operations by process
# TYPE discourse_major_gc_count counter
discourse_major_gc_count{type="sidekiq",pid="131"} 39
discourse_major_gc_count{type="web",pid="163"} 902
discourse_major_gc_count{type="web",pid="183"} 848
discourse_major_gc_count{type="web",pid="454"} 881
discourse_major_gc_count{type="web",pid="201"} 896
discourse_major_gc_count{type="web",pid="503"} 849
discourse_major_gc_count{type="web",pid="308"} 878
discourse_major_gc_count{type="web",pid="328"} 888
discourse_major_gc_count{type="web",pid="396"} 882
discourse_major_gc_count{type="web",pid="219"} 897
discourse_major_gc_count{type="web",pid="374"} 919
discourse_major_gc_count{type="web",pid="270"} 883
discourse_major_gc_count{type="web",pid="418"} 921
discourse_major_gc_count{type="web",pid="434"} 853
discourse_major_gc_count{type="web",pid="479"} 923
discourse_major_gc_count{type="web",pid="350"} 909
discourse_major_gc_count{type="web",pid="287"} 823

# HELP discourse_minor_gc_count Minor GC operations by process
# TYPE discourse_minor_gc_count counter
discourse_minor_gc_count{type="sidekiq",pid="131"} 251
discourse_minor_gc_count{type="web",pid="163"} 8220
discourse_minor_gc_count{type="web",pid="183"} 8171
discourse_minor_gc_count{type="web",pid="454"} 8435
discourse_minor_gc_count{type="web",pid="201"} 8249
discourse_minor_gc_count{type="web",pid="503"} 8252
discourse_minor_gc_count{type="web",pid="308"} 8356
discourse_minor_gc_count{type="web",pid="328"} 8276
discourse_minor_gc_count{type="web",pid="396"} 8377
discourse_minor_gc_count{type="web",pid="219"} 8311
discourse_minor_gc_count{type="web",pid="374"} 8521
discourse_minor_gc_count{type="web",pid="270"} 8256
discourse_minor_gc_count{type="web",pid="418"} 8503
discourse_minor_gc_count{type="web",pid="434"} 8332
discourse_minor_gc_count{type="web",pid="479"} 8577
discourse_minor_gc_count{type="web",pid="350"} 8480
discourse_minor_gc_count{type="web",pid="287"} 8241

# HELP discourse_total_allocated_objects Total number of allocateds objects by process
# TYPE discourse_total_allocated_objects counter
discourse_total_allocated_objects{type="sidekiq",pid="131"} 30060560
discourse_total_allocated_objects{type="web",pid="163"} 985232557
discourse_total_allocated_objects{type="web",pid="183"} 983805640
discourse_total_allocated_objects{type="web",pid="454"} 1008867616
discourse_total_allocated_objects{type="web",pid="201"} 988967755
discourse_total_allocated_objects{type="web",pid="503"} 998885290
discourse_total_allocated_objects{type="web",pid="308"} 1004069224
discourse_total_allocated_objects{type="web",pid="328"} 996554004
discourse_total_allocated_objects{type="web",pid="396"} 1013631939
discourse_total_allocated_objects{type="web",pid="219"} 999330059
discourse_total_allocated_objects{type="web",pid="374"} 1016924963
discourse_total_allocated_objects{type="web",pid="270"} 989042711
discourse_total_allocated_objects{type="web",pid="418"} 1024175277
discourse_total_allocated_objects{type="web",pid="434"} 1004407864
discourse_total_allocated_objects{type="web",pid="479"} 1025782126
discourse_total_allocated_objects{type="web",pid="350"} 1013371770
discourse_total_allocated_objects{type="web",pid="287"} 1003799360

# HELP discourse_scheduled_job_duration_seconds Total time spent in scheduled jobs
# TYPE discourse_scheduled_job_duration_seconds counter
discourse_scheduled_job_duration_seconds{job_name="Jobs::PendingUsersReminder"} 0.644000000000000011e0
discourse_scheduled_job_duration_seconds{job_name="Jobs::WhosOnlineGoingOffline"} 0.5267999999999998071e1
discourse_scheduled_job_duration_seconds{job_name="Jobs::PresenceChannelAutoLeave"} 0.6388000000000000119e1
discourse_scheduled_job_duration_seconds{job_name="Jobs::ProcessBadgeBacklog"} 0.25297000000000000432e2
discourse_scheduled_job_duration_seconds{job_name="Jobs::TopicTimerEnqueuer"} 0.5947000000000000053e1
discourse_scheduled_job_duration_seconds{job_name="Jobs::Heartbeat"} 0.1635000000000000038e1
discourse_scheduled_job_duration_seconds{job_name="Jobs::ProcessShelvedNotifications"} 0.1221000000000000029e1
discourse_scheduled_job_duration_seconds{job_name="Jobs::EnqueueOnceoffs"} 0.4557000000000000043e1
discourse_scheduled_job_duration_seconds{job_name="Jobs::BookmarkReminderNotifications"} 0.1019000000000000007e1
discourse_scheduled_job_duration_seconds{job_name="Jobs::CleanUpUploads"} 0.138000000000000002e0
discourse_scheduled_job_duration_seconds{job_name="Jobs::PollMailbox"} 0.851000000000000004e0
discourse_scheduled_job_duration_seconds{job_name="Jobs::TopRefreshToday"} 13.684000000000001
discourse_scheduled_job_duration_seconds{job_name="Jobs::DestroyOldDeletionStubs"} 0.272400000000000003e1
discourse_scheduled_job_duration_seconds{job_name="Jobs::ActivationReminderEmails"} 0.096
discourse_scheduled_job_duration_seconds{job_name="Jobs::UnsilenceUsers"} 0.8790000000000002
discourse_scheduled_job_duration_seconds{job_name="Jobs::PeriodicalUpdates"} 0.1145550000000000045e3
discourse_scheduled_job_duration_seconds{job_name="Jobs::MigrateUploadScheme"} 0.388000000000000004e0
discourse_scheduled_job_duration_seconds{job_name="Jobs::AboutStats"} 2.139
discourse_scheduled_job_duration_seconds{job_name="Jobs::CheckOutOfDateThemes"} 1.68
discourse_scheduled_job_duration_seconds{job_name="Jobs::EnqueueDigestEmails"} 0.156000000000000008e0
discourse_scheduled_job_duration_seconds{job_name="Jobs::DirectoryRefreshDaily"} 0.43910000000000002e1
discourse_scheduled_job_duration_seconds{job_name="Jobs::DashboardStats"} 0.153000000000000009e0
discourse_scheduled_job_duration_seconds{job_name="Jobs::PendingReviewablesReminder"} 0.060000000000000005
discourse_scheduled_job_duration_seconds{job_name="Jobs::CreateMissingAvatars"} 0.200000000000000005e0
discourse_scheduled_job_duration_seconds{job_name="Jobs::UpdateAnimatedUploads"} 0.133
discourse_scheduled_job_duration_seconds{job_name="Jobs::EnqueueSuspectUsers"} 0.12100000000000001
discourse_scheduled_job_duration_seconds{job_name="Jobs::UpdateSitemaps"} 3.831
discourse_scheduled_job_duration_seconds{job_name="Jobs::PendingQueuedPostsReminder"} 2.649
discourse_scheduled_job_duration_seconds{job_name="Jobs::ReindexSearch"} 1.431
discourse_scheduled_job_duration_seconds{job_name="Jobs::DisableBootstrapMode"} 0.038
discourse_scheduled_job_duration_seconds{job_name="Jobs::CleanUpUnusedStagedUsers"} 0.065

# HELP discourse_scheduled_job_count Total number of scheduled jobs executued
# TYPE discourse_scheduled_job_count counter
discourse_scheduled_job_count{job_name="Jobs::PendingUsersReminder"} 38
discourse_scheduled_job_count{job_name="Jobs::WhosOnlineGoingOffline"} 187
discourse_scheduled_job_count{job_name="Jobs::PresenceChannelAutoLeave"} 187
discourse_scheduled_job_count{job_name="Jobs::ProcessBadgeBacklog"} 188
discourse_scheduled_job_count{job_name="Jobs::TopicTimerEnqueuer"} 187
discourse_scheduled_job_count{job_name="Jobs::Heartbeat"} 63
discourse_scheduled_job_count{job_name="Jobs::ProcessShelvedNotifications"} 38
discourse_scheduled_job_count{job_name="Jobs::EnqueueOnceoffs"} 19
discourse_scheduled_job_count{job_name="Jobs::BookmarkReminderNotifications"} 38
discourse_scheduled_job_count{job_name="Jobs::CleanUpUploads"} 4
discourse_scheduled_job_count{job_name="Jobs::PollMailbox"} 38
discourse_scheduled_job_count{job_name="Jobs::TopRefreshToday"} 4
discourse_scheduled_job_count{job_name="Jobs::DestroyOldDeletionStubs"} 7
discourse_scheduled_job_count{job_name="Jobs::ActivationReminderEmails"} 2
discourse_scheduled_job_count{job_name="Jobs::UnsilenceUsers"} 13
discourse_scheduled_job_count{job_name="Jobs::PeriodicalUpdates"} 13
discourse_scheduled_job_count{job_name="Jobs::MigrateUploadScheme"} 19
discourse_scheduled_job_count{job_name="Jobs::AboutStats"} 6
discourse_scheduled_job_count{job_name="Jobs::CheckOutOfDateThemes"} 1
discourse_scheduled_job_count{job_name="Jobs::EnqueueDigestEmails"} 7
discourse_scheduled_job_count{job_name="Jobs::DirectoryRefreshDaily"} 3
discourse_scheduled_job_count{job_name="Jobs::DashboardStats"} 6
discourse_scheduled_job_count{job_name="Jobs::PendingReviewablesReminder"} 3
discourse_scheduled_job_count{job_name="Jobs::CreateMissingAvatars"} 3
discourse_scheduled_job_count{job_name="Jobs::UpdateAnimatedUploads"} 3
discourse_scheduled_job_count{job_name="Jobs::EnqueueSuspectUsers"} 2
discourse_scheduled_job_count{job_name="Jobs::UpdateSitemaps"} 3
discourse_scheduled_job_count{job_name="Jobs::PendingQueuedPostsReminder"} 3
discourse_scheduled_job_count{job_name="Jobs::ReindexSearch"} 1
discourse_scheduled_job_count{job_name="Jobs::DisableBootstrapMode"} 1
discourse_scheduled_job_count{job_name="Jobs::CleanUpUnusedStagedUsers"} 1

# HELP discourse_sidekiq_job_duration_seconds Total time spent in sidekiq jobs
# TYPE discourse_sidekiq_job_duration_seconds counter
discourse_sidekiq_job_duration_seconds{job_name="Jobs::RunHeartbeat"} 0.226169511675834654e0
discourse_sidekiq_job_duration_seconds{job_name="Jobs::NotifyMailingListSubscribers"} 0.8942126949783414593e1
discourse_sidekiq_job_duration_seconds{job_name="Jobs::PullHotlinkedImages"} 0.8282361692283302553e1
discourse_sidekiq_job_duration_seconds{job_name="Jobs::FeatureTopicUsers"} 0.1046757272724062204e2
discourse_sidekiq_job_duration_seconds{job_name="Jobs::BotInput"} 0.114331910074222833095e3
discourse_sidekiq_job_duration_seconds{job_name="Jobs::PostUpdateTopicTrackingState"} 0.17196641702670603947e2
discourse_sidekiq_job_duration_seconds{job_name="Jobs::CrawlTopicLink"} 0.1373126078015193336375e3
discourse_sidekiq_job_duration_seconds{job_name="Jobs::PostAlert"} 0.22092239051265641677e3
discourse_sidekiq_job_duration_seconds{job_name="Jobs::ProcessPost"} 0.30518020779779181136e3

# HELP discourse_sidekiq_job_count Total number of sidekiq jobs executed
# TYPE discourse_sidekiq_job_count counter
discourse_sidekiq_job_count{job_name="Jobs::RunHeartbeat"} 63
discourse_sidekiq_job_count{job_name="Jobs::NotifyMailingListSubscribers"} 94
discourse_sidekiq_job_count{job_name="Jobs::PullHotlinkedImages"} 100
discourse_sidekiq_job_count{job_name="Jobs::FeatureTopicUsers"} 101
discourse_sidekiq_job_count{job_name="Jobs::BotInput"} 217
discourse_sidekiq_job_count{job_name="Jobs::PostUpdateTopicTrackingState"} 101
discourse_sidekiq_job_count{job_name="Jobs::CrawlTopicLink"} 414
discourse_sidekiq_job_count{job_name="Jobs::PostAlert"} 106
discourse_sidekiq_job_count{job_name="Jobs::ProcessPost"} 106

# HELP discourse_postgres_readonly_mode Indicates whether site is in readonly mode due to PostgreSQL failover
# TYPE discourse_postgres_readonly_mode gauge
discourse_postgres_readonly_mode 0

# HELP discourse_redis_master_available Whether or not we have an active connection to the master Redis
# TYPE discourse_redis_master_available gauge
discourse_redis_master_available 1

# HELP discourse_redis_slave_available Whether or not we have an active connection a Redis slave
# TYPE discourse_redis_slave_available gauge
discourse_redis_slave_available 0

# HELP discourse_postgres_master_available Whether or not we have an active connection to the master PostgreSQL
# TYPE discourse_postgres_master_available gauge
discourse_postgres_master_available 1

# HELP discourse_postgres_replica_available Whether or not we have an active connection to the replica PostgreSQL
# TYPE discourse_postgres_replica_available gauge


# HELP discourse_active_app_reqs Number of active web requests in progress
# TYPE discourse_active_app_reqs gauge
discourse_active_app_reqs 16

# HELP discourse_queued_app_reqs Number of queued web requests
# TYPE discourse_queued_app_reqs gauge
discourse_queued_app_reqs 95

# HELP discourse_sidekiq_jobs_enqueued Number of jobs queued in the Sidekiq worker processes
# TYPE discourse_sidekiq_jobs_enqueued gauge
discourse_sidekiq_jobs_enqueued{queue="default"} 0
discourse_sidekiq_jobs_enqueued{queue="low"} 0
discourse_sidekiq_jobs_enqueued{queue="critical"} 0
discourse_sidekiq_jobs_enqueued{queue="ultra_low"} 0

# HELP discourse_sidekiq_processes Number of Sidekiq job processes
# TYPE discourse_sidekiq_processes gauge
discourse_sidekiq_processes 1

# HELP discourse_sidekiq_paused Whether or not Sidekiq is paused
# TYPE discourse_sidekiq_paused gauge


# HELP discourse_sidekiq_workers Total number of active sidekiq workers
# TYPE discourse_sidekiq_workers gauge
discourse_sidekiq_workers 5

# HELP discourse_sidekiq_jobs_stuck Number of sidekiq jobs which have been running for more than 120 minutes
# TYPE discourse_sidekiq_jobs_stuck gauge


# HELP discourse_scheduled_jobs_stuck Number of scheduled jobs which have been running for more than their expected duration
# TYPE discourse_scheduled_jobs_stuck gauge


# HELP discourse_missing_s3_uploads Number of missing uploads in S3
# TYPE discourse_missing_s3_uploads gauge


# HELP discourse_version_info Labelled with `revision` (current core commit hash), and `version` (Discourse::VERSION::STRING)
# TYPE discourse_version_info gauge
discourse_version_info{revision="e3164671698aeef94cf2e142903a30e4688e9dfe",version="2.8.0.beta6"} 1

# HELP discourse_readonly_sites Count of sites currently in readonly mode, grouped by the relevant key from Discourse::READONLY_KEYS
# TYPE discourse_readonly_sites gauge
discourse_readonly_sites{key="readonly_mode"} 0
discourse_readonly_sites{key="readonly_mode:postgres"} 0
discourse_readonly_sites{key="readonly_mode:user"} 0
discourse_readonly_sites{key="readonly_mode:postgres_force"} 0

Questions:

  1. Is it DDOS or bruteforce attack on the forum or some bots traffic?
    OR its genuine traffic and we need to upgrade our droplet specs, if so, what should we upgrade, by how much?

  2. What should we do ? right now, our site is loading very slow, and navigation from 1 topic to another, the loading transition is like 5 seconds+

  3. How to find out which traffic or resource is using this much traffic? Whether if its real human traffic or bots trying to hit down our site?

  4. What changes should we do in our app.yaml if we decide to upgrade our droplet specs?

  5. On users profile admin page, we see unique IP of each user, means CF is getting real IPs.? but they why we get 429 too many requests error when we uncomment the ratelimited template from app.yml

  6. How should we configure our CF DDOS settings?

  7. Is there any way to configure our server/discourse to work in multi threaded or parallel way to support our high traffic using our same current specs droplet? If yes, how to configure multithreaded/parallel scaling in discourse to use multi core/CPUs droplets effectively and in best optimized way?

We are using cloudflare and when we turn on CF I am under attack mode, then CPU usage goes down, but we cannot annoy our new users to solve captcha each time they use our site. Right now CF bot fight mode is ON and CF security level is high.
We have tried rebuilding but after rebuild is done, it goes to normal but after few minutes CPU usage goes back to 100%

Please someone help.

1 Like

Well, you need to decide whether the convenience or stable operation of your site is important to you.
During DDoS attacks, there will always be some compromises in the operation of your site.
You can try https://ddos-guard.net

2 Likes

How do i make sure that it is indeed a DDOS attack on my site? or its just real genuine traffic growth?

First i need to find the real problem and then go towards the right solution.

1 Like

try
netstat -ntu|awk '{print $5}'|cut -d: -f1 -s|sort|uniq -c|sort -nk1 -r
and compare with your online users

2 Likes

@valsha
I just ran the above command, and my google analytics users were 73 and out put of above command is below and it shows 596 lines, now what does this means?

3178 127.0.0.1
   3105 172.17.0.2
    434 172.64.200.21
     27 172.64.201.21
      8 162.158.89.144
      7 162.158.94.247
      7 162.158.92.123
      7 162.158.90.37
      7 162.158.90.253
      7 162.158.89.140
      7 162.158.88.163
      6 162.158.94.163
      6 162.158.90.239
      6 162.158.89.34
      6 162.158.89.246
      6 162.158.89.106
      6 162.158.89.100
      6 162.158.88.61
      6 162.158.88.119
      5 162.158.88.7
      4 172.69.71.157
      4 172.68.142.200
      3 172.70.142.205
      3 172.70.126.91
      3 172.70.126.7
      3 172.70.110.233
      3 172.70.110.197
      3 172.69.71.67
      3 172.69.35.204
      3 172.69.22.107
      3 172.69.22.101
      3 172.68.189.142
      3 172.68.143.57
      3 172.68.143.231
      3 172.68.133.21
      3 172.68.133.165
      3 172.68.132.34
      3 172.68.132.120
      3 162.158.63.241
      3 162.158.255.183
      3 108.162.245.79
      3 108.162.221.137
      3 108.162.219.187
      2 173.245.52.212
      2 172.70.83.8
      2 172.70.82.187
      2 172.70.82.115
      2 172.70.61.179
      2 172.70.61.135
      2 172.70.54.63
      2 172.70.178.199
      2 172.70.178.103
      2 172.70.130.59
      2 172.70.130.165
      2 172.70.126.9
      2 172.70.126.81
      2 172.70.114.73
      2 172.70.114.123
      2 172.70.110.125
      2 172.69.71.89
      2 172.69.71.87
      2 172.69.71.204
      2 172.69.69.182
      2 172.69.69.170
      2 172.69.68.229
      2 172.69.68.144
      2 172.69.55.161
      2 172.69.34.119
      2 172.69.22.173
      2 172.69.22.125
      2 172.68.143.21
      2 172.68.141.187
      2 172.68.141.175
      2 172.68.133.13
      2 162.158.74.198
      2 108.162.250.192
      2 108.162.246.22
      2 108.162.221.67
      2 108.162.221.207
      2 108.162.219.19
      2 108.162.219.165
      2 108.162.219.115
      2 108.162.215.107
      2 108.162.212.20
      2 108.162.210.199
      1 198.41.238.97
      1 198.41.231.66
      1 188.141.50.15
      1 173.245.52.222
      1 173.245.52.210
      1 173.245.52.204
      1 173.245.52.168
      1 173.245.52.166
      1 172.70.98.9
      1 172.70.98.141
      1 172.70.98.131
      1 172.70.82.99
      1 172.70.82.95
      1 172.70.82.85
      1 172.70.82.57
      1 172.70.82.29
      1 172.70.82.27
      1 172.70.82.249
      1 172.70.82.233
      1 172.70.82.229
      1 172.70.82.221
      1 172.70.82.213
      1 172.70.82.211
      1 172.70.82.203
      1 172.70.82.173
      1 172.70.82.159
      1 172.70.82.153
      1 172.70.82.151
      1 172.70.82.137
      1 172.70.82.135
      1 172.70.82.107
      1 172.70.61.193
      1 172.70.61.189
      1 172.70.61.185
      1 172.70.61.177
      1 172.70.61.173
      1 172.70.61.171
      1 172.70.61.169
      1 172.70.61.161
      1 172.70.61.157
      1 172.70.61.153
      1 172.70.61.151
      1 172.70.61.149
      1 172.70.61.139
      1 172.70.61.133
      1 172.70.57.187
      1 172.70.57.181
      1 172.70.54.97
      1 172.70.54.71
      1 172.70.54.55
      1 172.70.54.53
      1 172.70.54.45
      1 172.70.54.43
      1 172.70.54.139
      1 172.70.54.109
      1 172.70.54.105
      1 172.70.188.53
      1 172.70.188.159
      1 172.70.178.51
      1 172.70.178.237
      1 172.70.178.141
      1 172.70.143.58
      1 172.70.143.36
      1 172.70.142.55
      1 172.70.142.239
      1 172.70.142.235
      1 172.70.142.215
      1 172.70.142.129
      1 172.70.142.119
      1 172.70.134.17
      1 172.70.130.95
      1 172.70.130.19
      1 172.70.127.38
      1 172.70.127.34
      1 172.70.126.79
      1 172.70.126.3
      1 172.70.126.251
      1 172.70.126.25
      1 172.70.126.243
      1 172.70.126.23
      1 172.70.126.227
      1 172.70.126.225
      1 172.70.126.203
      1 172.70.126.201
      1 172.70.126.197
      1 172.70.126.179
      1 172.70.126.153
      1 172.70.126.143
      1 172.70.126.139
      1 172.70.126.131
      1 172.70.126.111
      1 172.70.122.217
      1 172.70.114.99
      1 172.70.114.175
      1 172.70.114.119
      1 172.70.110.85
      1 172.70.110.77
      1 172.70.110.49
      1 172.70.110.249
      1 172.70.110.221
      1 172.70.110.215
      1 172.70.110.211
      1 172.70.110.201
      1 172.70.110.185
      1 172.70.110.173
      1 172.70.110.167
      1 172.70.110.129
      1 172.69.71.97
      1 172.69.71.95
      1 172.69.71.93
      1 172.69.71.91
      1 172.69.71.85
      1 172.69.71.83
      1 172.69.71.81
      1 172.69.71.75
      1 172.69.71.73
      1 172.69.71.71
      1 172.69.71.69
      1 172.69.71.65
      1 172.69.71.63
      1 172.69.71.6
      1 172.69.71.57
      1 172.69.71.55
      1 172.69.71.53
      1 172.69.71.51
      1 172.69.71.49
      1 172.69.71.47
      1 172.69.71.45
      1 172.69.71.42
      1 172.69.71.32
      1 172.69.71.30
      1 172.69.71.202
      1 172.69.71.200
      1 172.69.71.199
      1 172.69.71.197
      1 172.69.71.195
      1 172.69.71.193
      1 172.69.71.191
      1 172.69.71.189
      1 172.69.71.187
      1 172.69.71.185
      1 172.69.71.183
      1 172.69.71.181
      1 172.69.71.179
      1 172.69.71.177
      1 172.69.71.175
      1 172.69.71.173
      1 172.69.71.171
      1 172.69.71.167
      1 172.69.71.165
      1 172.69.71.163
      1 172.69.71.161
      1 172.69.71.155
      1 172.69.71.153
      1 172.69.71.149
      1 172.69.71.147
      1 172.69.71.143
      1 172.69.71.141
      1 172.69.71.139
      1 172.69.71.137
      1 172.69.71.135
      1 172.69.71.133
      1 172.69.71.131
      1 172.69.71.129
      1 172.69.71.127
      1 172.69.71.125
      1 172.69.71.123
      1 172.69.71.121
      1 172.69.71.117
      1 172.69.71.115
      1 172.69.71.113
      1 172.69.71.111
      1 172.69.71.109
      1 172.69.71.106
      1 172.69.71.10
      1 172.69.69.225
      1 172.69.69.223
      1 172.69.69.221
      1 172.69.69.205
      1 172.69.69.203
      1 172.69.69.180
      1 172.69.69.176
      1 172.69.69.174
      1 172.69.69.172
      1 172.69.69.164
      1 172.69.69.162
      1 172.69.69.160
      1 172.69.69.141
      1 172.69.69.134
      1 172.69.69.128
      1 172.69.69.126
      1 172.69.69.124
      1 172.69.69.122
      1 172.69.69.120
      1 172.69.69.118
      1 172.69.69.114
      1 172.69.69.112
      1 172.69.68.90
      1 172.69.68.72
      1 172.69.68.66
      1 172.69.68.6
      1 172.69.68.54
      1 172.69.68.48
      1 172.69.68.43
      1 172.69.68.4
      1 172.69.68.39
      1 172.69.68.37
      1 172.69.68.35
      1 172.69.68.33
      1 172.69.68.31
      1 172.69.68.29
      1 172.69.68.27
      1 172.69.68.241
      1 172.69.68.239
      1 172.69.68.233
      1 172.69.68.231
      1 172.69.68.227
      1 172.69.68.225
      1 172.69.68.223
      1 172.69.68.221
      1 172.69.68.219
      1 172.69.68.217
      1 172.69.68.215
      1 172.69.68.213
      1 172.69.68.211
      1 172.69.68.208
      1 172.69.68.206
      1 172.69.68.204
      1 172.69.68.202
      1 172.69.68.200
      1 172.69.68.20
      1 172.69.68.198
      1 172.69.68.196
      1 172.69.68.194
      1 172.69.68.192
      1 172.69.68.190
      1 172.69.68.188
      1 172.69.68.186
      1 172.69.68.184
      1 172.69.68.182
      1 172.69.68.174
      1 172.69.68.172
      1 172.69.68.170
      1 172.69.68.17
      1 172.69.68.168
      1 172.69.68.162
      1 172.69.68.158
      1 172.69.68.156
      1 172.69.68.154
      1 172.69.68.150
      1 172.69.68.15
      1 172.69.68.148
      1 172.69.68.138
      1 172.69.68.120
      1 172.69.68.114
      1 172.69.68.108
      1 172.69.55.95
      1 172.69.35.224
      1 172.69.35.184
      1 172.69.35.16
      1 172.69.34.97
      1 172.69.34.77
      1 172.69.34.217
      1 172.69.34.149
      1 172.69.23.78
      1 172.69.23.6
      1 172.69.22.89
      1 172.69.22.47
      1 172.69.22.23
      1 172.69.22.149
      1 172.69.166.211
      1 172.69.156.151
      1 172.69.142.61
      1 172.68.245.4
      1 172.68.244.225
      1 172.68.242.227
      1 172.68.24.61
      1 172.68.24.189
      1 172.68.24.185
      1 172.68.189.44
      1 172.68.189.238
      1 172.68.143.237
      1 172.68.143.141
      1 172.68.143.111
      1 172.68.142.152
      1 172.68.141.217
      1 172.68.141.181
      1 172.68.141.145
      1 172.68.141.103
      1 172.68.133.85
      1 172.68.133.7
      1 172.68.133.5
      1 172.68.133.189
      1 172.68.133.147
      1 172.68.133.107
      1 172.68.132.82
      1 172.68.132.224
      1 172.68.132.22
      1 172.68.132.216
      1 172.68.132.192
      1 172.68.132.182
      1 172.68.132.174
      1 172.68.132.118
      1 172.68.132.10
      1 172.68.118.79
      1 172.68.118.41
      1 172.68.11.90
      1 172.68.11.8
      1 172.68.11.200
      1 172.68.10.243
      1 172.68.10.15
      1 172.68.10.143
      1 172.67.148.71
      1 162.158.90.17
      1 162.158.89.28
      1 162.158.88.95
      1 162.158.88.79
      1 162.158.88.133
      1 162.158.75.43
      1 162.158.75.189
      1 162.158.75.173
      1 162.158.75.163
      1 162.158.74.96
      1 162.158.74.236
      1 162.158.74.218
      1 162.158.74.196
      1 162.158.74.152
      1 162.158.74.140
      1 162.158.74.138
      1 162.158.74.130
      1 162.158.63.7
      1 162.158.63.65
      1 162.158.63.115
      1 162.158.62.94
      1 162.158.62.40
      1 162.158.62.32
      1 162.158.62.248
      1 162.158.62.138
      1 162.158.62.122
      1 162.158.62.118
      1 162.158.62.110
      1 162.158.50.4
      1 162.158.50.26
      1 162.158.255.35
      1 162.158.255.229
      1 162.158.255.215
      1 162.158.2.219
      1 162.158.2.207
      1 162.158.187.94
      1 162.158.183.170
      1 162.158.166.56
      1 162.158.165.27
      1 162.158.165.183
      1 162.158.165.169
      1 162.158.159.123
      1 162.158.158.152
      1 162.158.158.122
      1 162.158.155.65
      1 162.158.155.53
      1 162.158.154.136
      1 162.158.122.83
      1 162.158.119.152
      1 162.158.107.92
      1 162.158.107.50
      1 162.158.107.46
      1 162.158.107.18
      1 162.158.106.83
      1 162.158.106.225
      1 162.158.106.189
      1 162.158.106.173
      1 162.158.106.165
      1 162.158.106.155
      1 141.101.99.234
      1 141.101.99.230
      1 141.101.98.195
      1 141.101.98.149
      1 141.101.97.104
      1 141.101.96.31
      1 141.101.69.248
      1 141.101.107.85
      1 141.101.107.75
      1 141.101.107.57
      1 141.101.107.49
      1 141.101.107.179
      1 141.101.107.143
      1 141.101.105.79
      1 141.101.104.174
      1 141.101.104.114
      1 108.162.250.254
      1 108.162.250.228
      1 108.162.250.132
      1 108.162.250.124
      1 108.162.249.43
      1 108.162.249.37
      1 108.162.246.62
      1 108.162.246.6
      1 108.162.246.4
      1 108.162.246.36
      1 108.162.246.188
      1 108.162.246.160
      1 108.162.246.134
      1 108.162.245.81
      1 108.162.245.55
      1 108.162.245.25
      1 108.162.245.243
      1 108.162.245.225
      1 108.162.245.21
      1 108.162.245.199
      1 108.162.245.167
      1 108.162.245.143
      1 108.162.245.135
      1 108.162.245.117
      1 108.162.245.11
      1 108.162.245.101
      1 108.162.241.63
      1 108.162.241.145
      1 108.162.241.123
      1 108.162.241.111
      1 108.162.229.43
      1 108.162.221.99
      1 108.162.221.97
      1 108.162.221.91
      1 108.162.221.89
      1 108.162.221.87
      1 108.162.221.85
      1 108.162.221.83
      1 108.162.221.81
      1 108.162.221.79
      1 108.162.221.77
      1 108.162.221.75
      1 108.162.221.73
      1 108.162.221.71
      1 108.162.221.69
      1 108.162.221.65
      1 108.162.221.53
      1 108.162.221.249
      1 108.162.221.247
      1 108.162.221.243
      1 108.162.221.241
      1 108.162.221.239
      1 108.162.221.237
      1 108.162.221.235
      1 108.162.221.233
      1 108.162.221.231
      1 108.162.221.229
      1 108.162.221.227
      1 108.162.221.223
      1 108.162.221.221
      1 108.162.221.217
      1 108.162.221.215
      1 108.162.221.213
      1 108.162.221.209
      1 108.162.221.205
      1 108.162.221.201
      1 108.162.221.199
      1 108.162.221.195
      1 108.162.221.193
      1 108.162.221.191
      1 108.162.221.187
      1 108.162.221.185
      1 108.162.221.181
      1 108.162.221.179
      1 108.162.221.177
      1 108.162.221.173
      1 108.162.221.171
      1 108.162.221.167
      1 108.162.221.165
      1 108.162.221.163
      1 108.162.221.161
      1 108.162.221.159
      1 108.162.221.157
      1 108.162.221.155
      1 108.162.221.153
      1 108.162.221.151
      1 108.162.221.149
      1 108.162.221.147
      1 108.162.221.145
      1 108.162.221.143
      1 108.162.221.141
      1 108.162.221.139
      1 108.162.221.129
      1 108.162.221.127
      1 108.162.221.125
      1 108.162.221.123
      1 108.162.221.121
      1 108.162.221.119
      1 108.162.221.113
      1 108.162.221.111
      1 108.162.221.109
      1 108.162.221.107
      1 108.162.221.105
      1 108.162.221.101
      1 108.162.219.97
      1 108.162.219.81
      1 108.162.219.57
      1 108.162.219.45
      1 108.162.219.43
      1 108.162.219.15
      1 108.162.219.145
      1 108.162.216.75
      1 108.162.216.229
      1 108.162.215.87
      1 108.162.215.55
      1 108.162.215.185
      1 108.162.215.159
      1 108.162.212.36
      1 108.162.212.200
      1 108.162.212.198
      1 108.162.210.207
      1 108.162.210.197
      1 108.162.210.175
1 Like

Do you have some theme component that’s doing api calls? Plugins?

1 Like

Those look like they are all CloudFlare IP ranges. Did you setup your cloudflare correctly in app.yml?

3 Likes

add `cloudflare.template.yml` to the end of the templates section in `app.yml` .
1 Like

I think you removed this restriction, because you did not have it configured cloudflare.template.yml
all users had one IP address and these restrictions were applied to them.
you can try:

  1. install cloudflare.template.yml
  2. install web.ratelimited.template.yml
3 Likes

Even if Cloudflare is set up correctly, you will still see CF IP addresses in netstat, so this is not necessarily a sign of the cloudflare.template.yml not being set up correctly.

3 Likes

Is the purpose of this config:

cloudflare.template.yml

to see real IP addresses instead of Cloudflare addresses?

1 Like

Yes, but that works at the point where nginx sends the information to the Discourse Unicorn services and logs the requests. At network level you will still see the real (Cloudflare) IP addresses.

Compare it to when someone sends you a letter. When you look at the envelope you will see it’s from your aunt Betty. netstat doesn’t look at the envelope, it looks who is coming at your door - and you will see the mailman bringing every letter, not your aunt Betty.

10 Likes

@RGJ thanks for your advice.

We are now getting real IPs of the users in our discourse user profile pages from Admin GUI.

We did that by uncommenting below line in our outer NGIXN,

proxy_ssl_server_name       on;
        proxy_ssl_protocols         TLSv1 TLSv1.1 TLSv1.2;
        proxy_pass       http://localhost:2045;
        proxy_redirect  http://localhost:2045/ https://howtodiscuss.com;
       # proxy_set_header X-Real-IP $remote_addr;
       # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       # proxy_set_header Host $http_host;
       # proxy_set_header X-NginX-Proxy true;

        proxy_redirect off;

        # Socket.IO Support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

However, netstat still shows cloudlfare IPs.

Our original problem is why our CPU usage is still 100% even though our real time online traffic is approx 150 users average on google analytics… and most of the unicorn workers are taking CPU usage, and i have already shared my app.yaml configs + and server specs in my first post. you can see above,

Should we resize our server with more CPUs, or what changes/settings/optimization we can do in app.yaml, discourse or serverside to speed up our site and lower our CPU usage?

Should we uncomment the ratelimit template in our app.yaml now? Last time when we uncommented it, our normal users were facing the 429 too many requests because all users were likely showing as CF IPs in our server. but now we think real users IPs are coming and it will be safe to uncomment the ratelimit template in app.yml?

how do we confirm that discourse and our server is infact getting real IPs of the users, since netstats is still showing CF IPs. ?

Is someone doing DDOS attack to down our discourse forum, if yes, how to stop that?

If the 100% cpu usage is genuinely because of real genuine traffic increase, what server specs should we resize to now?

are there any multi-threading or parallel CPU usage optimizations settings we can do on linux/discourse level to fix our 100% CPU usage problem and load our site faster to users? We have disabled most of the plugins which we are not using.

1 Like

As I tried to explain, netstat works on a different level. There is no nginx config in the world that will make netstat show real IP’s if your forum is behind Cloudflare.

Edit: I see you changed “not” into “now” so your post makes much more sense now.

4 Likes

There are other topics about tuning.

It should be fine to turn in the rate limiting template.

Fast cpu and SSD are important.

How big is your database? Have you changed any of the database settings?

Do you have CDN in place for forum and S3 assets? Cloudflare isn’t a cdn.

1 Like

@pfaffman

can you suggest us any topics that can help us?

we have turned on rate limiting template all of our users were encountering 429

Then we turn off Cloudflare template after that we cannot access discourse with local IP. Can you tell us how to whitelist IP in discourse?

16 GB / 8 vCPUs with ssd’s , Intel Dedicated Digital Ocean Droplet

table_name                           | row_estimate | table_size | index_size | total_size
-----------------------------------------------------------------------------------
topic_views                          | 7638895      | 339 MB     | 704 MB     | 1044 MB
post_search_data                     | 95984        | 493 MB     | 117 MB     | 610 MB
posts                                | 115690       | 512 MB     | 32 MB      | 544 MB
post_revisions                       | 18876        | 368 MB     | 1512 kB    | 370 MB
top_topics                           | 74384        | 70 MB      | 209 MB     | 279 MB
incoming_links                       | 1194305      | 72 MB      | 75 MB      | 147 MB
topic_links                          | 248176       | 46 MB      | 54 MB      | 101 MB
stylesheet_cache                     | 1819         | 94 MB      | 352 kB     | 95 MB
user_histories                       | 38564        | 83 MB      | 6376 kB    | 90 MB
topics                               | 88662        | 49 MB      | 37 MB      | 85 MB
notifications                        | 144259       | 39 MB      | 26 MB      | 65 MB
topic_search_data                    | 77107        | 47 MB      | 16 MB      | 63 MB
user_profile_views                   | 298745       | 30 MB      | 27 MB      | 57 MB
scheduler_stats                      | 247499       | 38 MB      | 6736 kB    | 44 MB
topic_users                          | 199664       | 28 MB      | 14 MB      | 42 MB
user_actions                         | 113450       | 15 MB      | 23 MB      | 38 MB
topic_tags                           | 293483       | 20 MB      | 13 MB      | 33 MB
search_logs                          | 172445       | 24 MB      | 8752 kB    | 33 MB
post_timings                         | 179627       | 15 MB      | 8568 kB    | 24 MB
post_stats                           | 105481       | 13 MB      | 5104 kB    | 18 MB
reviewables                          | 4737         | 15 MB      | 1184 kB    | 17 MB
optimized_images                     | 50814        | 12 MB      | 4704 kB    | 16 MB
user_auth_token_logs                 | 42998        | 12 MB      | 1904 kB    | 14 MB
incoming_referers                    | 26867        | 4520 kB    | 4592 kB    | 9112 kB
directory_items                      | 33456        | 2928 kB    | 5488 kB    | 8416 kB
draft_sequences                      | 38948        | 5144 kB    | 2448 kB    | 7592 kB
user_associated_accounts             | 2237         | 6728 kB    | 448 kB     | 7176 kB
user_custom_fields                   | 27921        | 2768 kB    | 3568 kB    | 6336 kB
reviewable_histories                 | 8538         | 5784 kB    | 528 kB     | 6312 kB
user_visits                          | 27147        | 1552 kB    | 2656 kB    | 4208 kB
uploads                              | 5623         | 1560 kB    | 2128 kB    | 3688 kB
users                                | 5633         | 1720 kB    | 1744 kB    | 3464 kB
topic_link_clicks                    | 16494        | 2088 kB    | 768 kB     | 2856 kB
email_logs                           | 5057         | 1128 kB    | 1568 kB    | 2696 kB
web_crawler_requests                 | 5533         | 848 kB     | 1632 kB    | 2480 kB
topic_allowed_users                  | 14487        | 976 kB     | 1128 kB    | 2104 kB
category_tag_stats                   | 11176        | 736 kB     | 1192 kB    | 1928 kB
drafts                               | 219          | 1712 kB    | 32 kB      | 1744 kB
category_users                       | 10579        | 544 kB     | 1120 kB    | 1664 kB
user_badges                          | 7852         | 768 kB     | 888 kB     | 1656 kB
email_tokens                         | 6175         | 800 kB     | 832 kB     | 1632 kB
tag_search_data                      | 6860         | 736 kB     | 808 kB     | 1544 kB
group_users                          | 9277         | 808 kB     | 696 kB     | 1504 kB
user_profiles                        | 5629         | 968 kB     | 472 kB     | 1440 kB
user_search_data                     | 5633         | 712 kB     | 704 kB     | 1416 kB
user_emails                          | 5117         | 504 kB     | 760 kB     | 1264 kB
tags                                 | 7121         | 592 kB     | 640 kB     | 1232 kB
plugin_store_rows                    | 4103         | 864 kB     | 360 kB     | 1224 kB
user_stats                           | 5631         | 1064 kB    | 144 kB     | 1208 kB
unsubscribe_keys                     | 2190         | 528 kB     | 520 kB     | 1048 kB
reviewable_scores                    | 4540         | 704 kB     | 320 kB     | 1024 kB
user_options                         | 5469         | 880 kB     | 144 kB     | 1024 kB
user_avatars                         | 5346         | 392 kB     | 456 kB     | 848 kB
incoming_domains                     | 4985         | 408 kB     | 432 kB     | 840 kB
post_uploads                         | 3194         | 264 kB     | 568 kB     | 832 kB
post_custom_fields                   | 2804         | 376 kB     | 432 kB     | 808 kB
topic_thumbnails                     | 3014         | 248 kB     | 536 kB     | 784 kB
user_uploads                         | 5549         | 328 kB     | 432 kB     | 760 kB
user_auth_tokens                     | 1382         | 416 kB     | 336 kB     | 752 kB
topic_custom_fields                  | 1058         | 352 kB     | 240 kB     | 592 kB
application_requests                 | 5401         | 280 kB     | 272 kB     | 552 kB
topic_allowed_groups                 | 3116         | 176 kB     | 264 kB     | 440 kB
quoted_posts                         | 2142         | 168 kB     | 240 kB     | 408 kB
theme_fields                         | 23           | 328 kB     | 32 kB      | 360 kB
post_actions                         | 1057         | 112 kB     | 224 kB     | 336 kB
watched_words                        | 1946         | 168 kB     | 144 kB     | 312 kB
javascript_caches                    | 11           | 248 kB     | 64 kB      | 312 kB
schema_migration_details             | 1218         | 200 kB     | 104 kB     | 304 kB
dismissed_topic_users                | 463          | 104 kB     | 160 kB     | 264 kB
discourse_voting_topic_vote_count    | 1407         | 120 kB     | 96 kB      | 216 kB
category_featured_topics             | 409          | 64 kB      | 120 kB     | 184 kB
categories                           | 73           | 72 kB      | 112 kB     | 184 kB
post_replies                         | 916          | 88 kB      | 80 kB      | 168 kB
group_histories                      | 498          | 56 kB      | 112 kB     | 168 kB
site_settings                        | 255          | 112 kB     | 48 kB      | 160 kB
bookmarks                            | 179          | 24 kB      | 112 kB     | 136 kB
user_security_keys                   | 3            | 16 kB      | 112 kB     | 128 kB
skipped_email_logs                   | 296          | 48 kB      | 80 kB      | 128 kB
category_custom_fields               | 513          | 56 kB      | 72 kB      | 128 kB
groups                               | 10           | 80 kB      | 48 kB      | 128 kB
schema_migrations                    | 1217         | 64 kB      | 56 kB      | 120 kB
assignments                          | 98           | 56 kB      | 48 kB      | 104 kB
user_archived_messages               | 637          | 40 kB      | 64 kB      | 104 kB
given_daily_likes                    | 569          | 32 kB      | 64 kB      | 96 kB
invites                              | 21           | 16 kB      | 80 kB      | 96 kB
do_not_disturb_timings               | 0            | 8192 bytes | 80 kB      | 88 kB
sitemaps                             | 9            | 48 kB      | 32 kB      | 80 kB
badges                               | 52           | 32 kB      | 48 kB      | 80 kB
user_api_keys                        | 1            | 16 kB      | 64 kB      | 80 kB
incoming_emails                      | 0            | 8192 bytes | 64 kB      | 72 kB
user_second_factors                  | 21           | 16 kB      | 48 kB      | 64 kB
email_change_requests                | 2            | 16 kB      | 48 kB      | 64 kB
screened_ip_addresses                | 5            | 16 kB      | 48 kB      | 64 kB
api_keys                             | 4            | 16 kB      | 48 kB      | 64 kB
backup_metadata                      | 6            | 48 kB      | 16 kB      | 64 kB
child_themes                         | 4            | 8192 bytes | 48 kB      | 56 kB
tag_users                            | 59           | 8192 bytes | 48 kB      | 56 kB
invited_users                        | 4            | 8192 bytes | 48 kB      | 56 kB
group_mentions                       | 6            | 8192 bytes | 48 kB      | 56 kB
tag_group_permissions                | 1            | 8192 bytes | 48 kB      | 56 kB
category_search_data                 | 73           | 16 kB      | 40 kB      | 56 kB
anonymous_users                      | 102          | 8192 bytes | 48 kB      | 56 kB
topic_invites                        | 6            | 8192 bytes | 48 kB      | 56 kB
user_notification_schedules          | 8            | 8192 bytes | 48 kB      | 56 kB
directory_columns                    | 8            | 16 kB      | 32 kB      | 48 kB
themes                               | 6            | 16 kB      | 32 kB      | 48 kB
external_upload_stubs                | 0            | 8192 bytes | 40 kB      | 48 kB
api_key_scopes                       | 6            | 16 kB      | 32 kB      | 48 kB
badge_types                          | 3            | 16 kB      | 32 kB      | 48 kB
color_scheme_colors                  | 20           | 16 kB      | 32 kB      | 48 kB
onceoff_logs                         | 40           | 16 kB      | 32 kB      | 48 kB
permalinks                           | 24           | 16 kB      | 32 kB      | 48 kB
theme_modifier_sets                  | 6            | 16 kB      | 32 kB      | 48 kB
topic_timers                         | 3            | 8192 bytes | 40 kB      | 48 kB
user_api_key_scopes                  | 2            | 16 kB      | 32 kB      | 48 kB
group_category_notification_defaults | 1            | 8192 bytes | 32 kB      | 40 kB
group_requests                       | 0            | 8192 bytes | 32 kB      | 40 kB
category_groups                      | 4            | 8192 bytes | 32 kB      | 40 kB
tag_group_memberships                | 7            | 8192 bytes | 32 kB      | 40 kB
invited_groups                       | 1            | 8192 bytes | 32 kB      | 40 kB
topic_groups                         | 47           | 8192 bytes | 32 kB      | 40 kB
discourse_voting_votes               | 4            | 8192 bytes | 32 kB      | 40 kB
theme_translation_overrides          | 0            | 8192 bytes | 24 kB      | 32 kB
oauth2_user_infos                    | 0            | 8192 bytes | 24 kB      | 32 kB
screened_urls                        | 0            | 8192 bytes | 24 kB      | 32 kB
screened_emails                      | 0            | 8192 bytes | 24 kB      | 32 kB
remote_themes                        | 3            | 16 kB      | 16 kB      | 32 kB
tag_groups                           | 1            | 16 kB      | 16 kB      | 32 kB
push_subscriptions                   | 13           | 16 kB      | 16 kB      | 32 kB
published_pages                      | 0            | 8192 bytes | 24 kB      | 32 kB
theme_settings                       | 9            | 16 kB      | 16 kB      | 32 kB
imap_sync_logs                       | 0            | 8192 bytes | 24 kB      | 32 kB
poll_options                         | 0            | 8192 bytes | 24 kB      | 32 kB
embeddable_hosts                     | 0            | 16 kB      | 16 kB      | 32 kB
color_schemes                        | 2            | 16 kB      | 16 kB      | 32 kB
user_exports                         | 1            | 16 kB      | 16 kB      | 32 kB
user_field_options                   | 4            | 16 kB      | 16 kB      | 32 kB
user_fields                          | 7            | 16 kB      | 16 kB      | 32 kB
badge_groupings                      | 5            | 16 kB      | 16 kB      | 32 kB
backup_draft_posts                   | 0            | 8192 bytes | 24 kB      | 32 kB
ar_internal_metadata                 | 1            | 16 kB      | 16 kB      | 32 kB
polls                                | 0            | 8192 bytes | 24 kB      | 32 kB
poll_votes                           | 0            | 0 bytes    | 32 kB      | 32 kB
web_hook_event_types                 | 13           | 16 kB      | 16 kB      | 32 kB
single_sign_on_records               | 0            | 8192 bytes | 24 kB      | 32 kB
group_custom_fields                  | 0            | 8192 bytes | 16 kB      | 24 kB
allowed_pm_users                     | 0            | 0 bytes    | 24 kB      | 24 kB
user_warnings                        | 0            | 0 bytes    | 24 kB      | 24 kB
user_ip_address_histories            | 0            | 8192 bytes | 16 kB      | 24 kB
user_open_ids                        | 0            | 8192 bytes | 16 kB      | 24 kB
muted_users                          | 0            | 0 bytes    | 24 kB      | 24 kB
topic_embeds                         | 0            | 8192 bytes | 16 kB      | 24 kB
post_action_types                    | 7            | 8192 bytes | 16 kB      | 24 kB
message_bus                          | 0            | 8192 bytes | 16 kB      | 24 kB
linked_topics                        | 0            | 0 bytes    | 24 kB      | 24 kB
post_details                         | 0            | 8192 bytes | 16 kB      | 24 kB
ignored_users                        | 0            | 0 bytes    | 24 kB      | 24 kB
web_hook_events                      | 0            | 8192 bytes | 16 kB      | 24 kB
translation_overrides                | 0            | 8192 bytes | 16 kB      | 24 kB
backup_draft_topics                  | 0            | 0 bytes    | 24 kB      | 24 kB
post_reply_keys                      | 0            | 0 bytes    | 24 kB      | 24 kB
custom_emojis                        | 0            | 8192 bytes | 16 kB      | 24 kB
category_tags                        | 0            | 0 bytes    | 24 kB      | 24 kB
shared_drafts                        | 0            | 0 bytes    | 24 kB      | 24 kB
category_tag_groups                  | 0            | 0 bytes    | 16 kB      | 16 kB
group_archived_messages              | 0            | 0 bytes    | 16 kB      | 16 kB
discourse_voting_category_settings   | 0            | 0 bytes    | 16 kB      | 16 kB
developers                           | 0            | 0 bytes    | 16 kB      | 16 kB
web_hooks                            | 0            | 8192 bytes | 8192 bytes | 16 kB
shelved_notifications                | 0            | 0 bytes    | 16 kB      | 16 kB
group_tag_notification_defaults      | 0            | 0 bytes    | 16 kB      | 16 kB
reviewable_claimed_topics            | 0            | 0 bytes    | 16 kB      | 16 kB
groups_web_hooks                     | 0            | 0 bytes    | 8192 bytes | 8192 bytes
tags_web_hooks                       | 0            | 0 bytes    | 8192 bytes | 8192 bytes
categories_web_hooks                 | 0            | 0 bytes    | 8192 bytes | 8192 bytes
web_hook_event_types_hooks           | 0            | 0 bytes    | 8192 bytes | 8192 bytes
badge_posts                          | 0            | 0 bytes    | 0 bytes    | 0 bytes

we have changed some settings in app.yml
db_shared_buffers: “4096MB”
db_work_mem: “400MB”
UNICORN_WORKERS: 16

## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  # - "templates/sshd.template.yml"
  #- "templates/web.ratelimited.template.yml" #removed this to avoid 429 too many requests errors  https://meta.discourse.org/t/remove-the-ip-limits/185829
  - "templates/cloudflare.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
#- "templates/web.ssl.template.yml"
#- "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
  - "1010:80" # http
#  - "443:443" # https

params:
  db_default_text_search_config: "pg_catalog.english"

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ## will be set automatically by bootstrap based on detected RAM, or you can override
  db_shared_buffers: "4096MB"

  ## can improve sorting performance, but adds memory usage per-connection
  db_work_mem: "400MB"

  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LANG: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  UNICORN_WORKERS: 16

  ## TODO: The domain name this Discourse instance will respond to
  ## Required. Discourse will not work with a bare IP number.
  DISCOURSE_HOSTNAME: "howtodiscuss.com"

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: "smtp_username"

  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  # SMTP ADDRESS, username, and password are required
  # WARNING the char '#' in SMTP password can cause problems!
  DISCOURSE_SMTP_ADDRESS: smtp.example.com
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: admin@example.com
  DISCOURSE_SMTP_PASSWORD: p@$$word
  #DISCOURSE_SMTP_DOMAIN: discourse.example.com    # (required by some providers)
  #DISCOURSE_NOTIFICATION_EMAIL: nobody@discourse.example.com    # (address to send notifications from)

  #DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
  #LETSENCRYPT_ACCOUNT_EMAIL: me@example.com
  #DISCOURSE_MAXMIND_LICENSE_KEY: 1234567890123456

  ## The http or https CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: https://discourse-cdn.example.com

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-sitemap.git
          #- git clone https://github.com/discourse/discourse-math.git # custom plugin for math JAX support
          - git clone https://github.com/discourse/discourse-solved.git # custom plugin for question and answers and choosing best answer
          #- git clone https://github.com/discourse/discourse-voting.git # custom plugin for voting on topics only
          #- git clone https://github.com/discourse/discourse-yearly-review.git # custom plugin for automatically posting a yearly topic of forum activity
          - git clone https://github.com/discourse/discourse-user-notes.git # custom plugin to allow staff to add notes about a user
          - git clone https://github.com/paviliondev/discourse-question-answer.git # custom non official plugin for SO style QnA voting on posts
          - git clone https://github.com/discourse/discourse-adplugin.git # custom official plugin for ads
          #- git clone https://github.com/discourse/discourse-whos-online.git # custom plugin to see who is online
          - git clone https://github.com/paviliondev/discourse-topic-previews.git # custom plugin for showing topics previews nicely
          #- git clone https://github.com/paviliondev/discourse-ratings.git # custom plugin to allow user rating on topics
          #- git clone https://github.com/discourse/discourse-cakeday.git # official plugin for birthdays and join date celebrations
          #- git clone https://github.com/discourse/discourse-saved-searches.git # official plugin for saved search notifications
          #- git clone https://github.com/paviliondev/discourse-follow.git # custom plugin to allow users to follow each other
          - git clone https://github.com/discourse/discourse-assign.git # custom for assigning topics to staff
          - git clone https://github.com/paviliondev/x-discourse-tlp-thumbnail.git # custom plugin for showing thumbnails from CDN images
          - git clone https://github.com/discourse/discourse-prometheus.git # custom plugin for slow speed monitoring
          - git clone https://github.com/discourse/discourse-push-notifications.git # plugin for push notifications
## Any custom commands to run after building
run:
  - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  - exec: rails r "SiteSetting.notification_email='noreply@howtodiscuss.com'"
  - exec: echo "End of custom commands"
  ## to show real IPs https://meta.discourse.org/t/troubleshooting-a-429-rate-limit/81060/19
  # This is the magic to get IP numbers transmitted to Discourse
  # See https://meta.discourse.org/t/last-ip-address-and-action-dispatch-trusted-proxies/50098/3?u=pfaffman
  - replace:
      filename: /etc/nginx/conf.d/discourse.conf
      from: "types {"
      to: |
        set_real_ip_from 172.17.0.0/24;
        real_ip_header X-Forwarded-For;
        real_ip_recursive on;
        types {

It could be that your replace section that adds the real_ip for your reverse proxy(?) is conflicting with the cloudflare template since they both do similar things. You need to make sure that the user IP and not cloudflare’s and not your reverse proxy is what Discourse sees. My best guess is that this is why the rate limiting template won’t work. You will likely need to fix that as right now all traffic is coming from your local IP.

The AMD servers are a little bit faster than the Intels (but I don’t they don’t have dedicated CPU for those? So Intel might be preferred?)

2 Likes

The ip 172.17.0.1 is probably a docker ip. I assume that the logs you posted are from the nginx service in the discourse container. My guess is that the nginx in your host is sending the request to the nginx in discourse, and the nginx in discourse is seeing the ip from the nginx in your host.

You should be able to fix it either by setting the real ip in discourse to be the value of a header sent by the nginx in the host, or by using a socket to connect the nginx services.

For example, in the nginx in the host you do proxy_set_header X-Real-IP $remote_addr;, and in the nginx in the discourse container you would use the real ip according to this header, set_real_ip_from 0.0.0.0/0; (assuming the discourse nginx is not publicly accessed) and real_ip_header X-Real-IP;. You probably will need a template or run some sed command to include it in the nginx configuration file in discourse.

Another option is to use sockets, and include the web.socketed.template.yml according to:

But my recommendation, if affordable, is to use discourse in it’s own server, without having to use nginx as a reverse proxy in front, which would make the setup much more straightforward and easier.

3 Likes

Is this load normal for the amount of traffic? - he has a pretty decent VPS.

Did you check if there are other things that might be dragging your CPU? Plugins maybe?

Your site seems bloated… when I visited I got at least 20 ads when clicking through 3 topics… I’m not sure if this can affect your server but you never know until tested.

Rate limit on + cloudflare template disabled ratelimits everybody as they all use CloudFlare IP’s. You need to turn on cloudflare template to get real IP’s and avoid everyone getting ratelimited.

2 Likes