All 8 vCPUs usage on our site is 100%. Our users get this message often, Due to extreme load, this is temporarily being shown to everyone as a logged out user would see it. Our droplet specs are: CPU-Optimized / 16 GB / 8 vCPUs , Intel Dedicated Digital Ocean Droplet Traffic: Online real time users on Google analytics as of writing are approx 70 to 150. htop snap for CPU and RAM usage is below, [image] [image] Last 14 days of graph usage from digital ocean: [image] [image] App.yaml looks like, db_shared_buffers: “4096MB” db_work_mem: “400MB” UNICORN_WORKERS: 16 #- “templates/web.ratelimited.template.yml” #removed this to avoid 429 too many requests errors Remove the ip limits As you can see, we have commented the ratelimited template because it was giving 429 nginx too many request errors to our normal users despite of very low actual traffic on our site. CF firewall page looks like, [image] Launcher log looks like, ./launcher logs app [image] Prometheus Metrics look like, # HELP discourse_collector_working Is the master process collector able to collect metrics # TYPE discourse_collector_working gauge discourse_collector_working 1 # HELP discourse_collector_rss total memory used by collector process # TYPE discourse_collector_rss gauge discourse_collector_rss 46530560 # HELP discourse_collector_metrics_total Total metrics processed by exporter web. # TYPE discourse_collector_metrics_total counter discourse_collector_metrics_total 215465 # HELP discourse_collector_sessions_total Total send_metric sessions processed by exporter web. # TYPE discourse_collector_sessions_total counter discourse_collector_sessions_total 7499 # HELP discourse_collector_bad_metrics_total Total mis-handled metrics by collector. # TYPE discourse_collector_bad_metrics_total counter discourse_collector_bad_metrics_total 0 # HELP discourse_page_views Page views reported by admin dashboard # TYPE discourse_page_views counter discourse_page_views{db="default",type="crawler",device="crawler"} 128620 discourse_page_views{db="default",type="anon",device="mobile"} 2840 discourse_page_views{db="default",type="anon",device="desktop"} 4094 discourse_page_views{db="default",type="logged_in",device="mobile"} 55 discourse_page_views{db="default",type="logged_in",device="desktop"} 64 # HELP discourse_http_requests Total HTTP requests from web app # TYPE discourse_http_requests counter discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="200"} 166429 discourse_http_requests{db="default",api="web",verb="POST",type="background",status="429"} 19628 discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="404"} 3945 discourse_http_requests{db="default",api="web",verb="HEAD",type="regular",status="200"} 154 discourse_http_requests{db="default",api="web",verb="POST",type="regular",status="200"} 2232 discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="301"} 152 discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="304"} 2295 discourse_http_requests{db="default",api="admin",verb="POST",type="regular",status="200"} 97 discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="409"} 141 discourse_http_requests{db="default",api="web",verb="POST",type="background",status="200"} 7540 discourse_http_requests{db="default",api="web",verb="POST",type="background",status="-1"} 1985 discourse_http_requests{db="default",api="admin",verb="POST",type="regular",status="422"} 3 discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="500"} 1 discourse_http_requests{db="default",api="web",verb="POST",type="regular",status="302"} 3 discourse_http_requests{db="default",api="web",verb="PUT",type="regular",status="200"} 7 discourse_http_requests{db="default",api="web",verb="GET",type="regular",status="302"} 2 # HELP discourse_http_duration_seconds Time spent in HTTP reqs in seconds # TYPE discourse_http_duration_seconds summary discourse_http_duration_seconds{controller="other",action="other",quantile="0.99"} 0.942478418815881 discourse_http_duration_seconds{controller="other",action="other",quantile="0.9"} 0.05924026109278202 discourse_http_duration_seconds{controller="other",action="other",quantile="0.5"} 0.000657188706099987 discourse_http_duration_seconds{controller="other",action="other",quantile="0.1"} 0.0004582637920975685 discourse_http_duration_seconds{controller="other",action="other",quantile="0.01"} 0.0004117507487535477 discourse_http_duration_seconds_sum{controller="other",action="other"} 3942.3923240867443 discourse_http_duration_seconds_count{controller="other",action="other"} 58870 discourse_http_duration_seconds{controller="topics",action="show",quantile="0.99"} 4.58280345518142 discourse_http_duration_seconds{controller="topics",action="show",quantile="0.9"} 2.827586194965988 discourse_http_duration_seconds{controller="topics",action="show",quantile="0.5"} 0.01292009698227048 discourse_http_duration_seconds{controller="topics",action="show",quantile="0.1"} 0.002048521768301725 discourse_http_duration_seconds{controller="topics",action="show",quantile="0.01"} 0.001819557975977659 discourse_http_duration_seconds_sum{controller="topics",action="show"} 128537.17967935232 discourse_http_duration_seconds_count{controller="topics",action="show"} 143926 discourse_http_duration_seconds{controller="list",action="latest",quantile="0.99"} 5.726923605892807 discourse_http_duration_seconds{controller="list",action="latest",quantile="0.9"} 5.168957637157291 discourse_http_duration_seconds{controller="list",action="latest",quantile="0.5"} 0.009740645997226238 discourse_http_duration_seconds{controller="list",action="latest",quantile="0.1"} 0.003163302782922983 discourse_http_duration_seconds{controller="list",action="latest",quantile="0.01"} 0.002744518686085939 discourse_http_duration_seconds_sum{controller="list",action="latest"} 3052.3646773528308 discourse_http_duration_seconds_count{controller="list",action="latest"} 1676 discourse_http_duration_seconds{controller="users",action="show",quantile="0.99"} 0.1768659572117031 discourse_http_duration_seconds{controller="users",action="show",quantile="0.9"} 0.1768659572117031 discourse_http_duration_seconds{controller="users",action="show",quantile="0.5"} 0.1768659572117031 discourse_http_duration_seconds{controller="users",action="show",quantile="0.1"} 0.1768659572117031 discourse_http_duration_seconds{controller="users",action="show",quantile="0.01"} 0.1768659572117031 discourse_http_duration_seconds_sum{controller="users",action="show"} 22.969309298321605 discourse_http_duration_seconds_count{controller="users",action="show"} 142 # HELP discourse_http_redis_duration_seconds Time spent in HTTP reqs in redis seconds # TYPE discourse_http_redis_duration_seconds summary discourse_http_redis_duration_seconds{controller="other",action="other",quantile="0.99"} 0.02405449468642473 discourse_http_redis_duration_seconds{controller="other",action="other",quantile="0.9"} 0.0092271463945508 discourse_http_redis_duration_seconds{controller="other",action="other",quantile="0.5"} 0.0 discourse_http_redis_duration_seconds{controller="other",action="other",quantile="0.1"} 0.0 discourse_http_redis_duration_seconds{controller="other",action="other",quantile="0.01"} 0.0 discourse_http_redis_duration_seconds_sum{controller="other",action="other"} 177.52055060770363 discourse_http_redis_duration_seconds_count{controller="other",action="other"} 58870 discourse_http_redis_duration_seconds{controller="topics",action="show",quantile="0.99"} 0.064909300301224 discourse_http_redis_duration_seconds{controller="topics",action="show",quantile="0.9"} 0.03797017782926559 discourse_http_redis_duration_seconds{controller="topics",action="show",quantile="0.5"} 0.007125054951757193 discourse_http_redis_duration_seconds{controller="topics",action="show",quantile="0.1"} 0.0005201729945838451 discourse_http_redis_duration_seconds{controller="topics",action="show",quantile="0.01"} 0.0004380182363092899 discourse_http_redis_duration_seconds_sum{controller="topics",action="show"} 1814.5181179852225 discourse_http_redis_duration_seconds_count{controller="topics",action="show"} 143924 discourse_http_redis_duration_seconds{controller="list",action="latest",quantile="0.99"} 0.03446856886148453 discourse_http_redis_duration_seconds{controller="list",action="latest",quantile="0.9"} 0.02759678149595857 discourse_http_redis_duration_seconds{controller="list",action="latest",quantile="0.5"} 0.004360581748187542 discourse_http_redis_duration_seconds{controller="list",action="latest",quantile="0.1"} 0.001454399898648262 discourse_http_redis_duration_seconds{controller="list",action="latest",quantile="0.01"} 0.00108026061207056 discourse_http_redis_duration_seconds_sum{controller="list",action="latest"} 20.32011368824169 discourse_http_redis_duration_seconds_count{controller="list",action="latest"} 1676 discourse_http_redis_duration_seconds{controller="users",action="show",quantile="0.99"} 0.01075387001037598 discourse_http_redis_duration_seconds{controller="users",action="show",quantile="0.9"} 0.01075387001037598 discourse_http_redis_duration_seconds{controller="users",action="show",quantile="0.5"} 0.01075387001037598 discourse_http_redis_duration_seconds{controller="users",action="show",quantile="0.1"} 0.01075387001037598 discourse_http_redis_duration_seconds{controller="users",action="show",quantile="0.01"} 0.01075387001037598 discourse_http_redis_duration_seconds_sum{controller="users",action="show"} 1.3520850692875683 discourse_http_redis_duration_seconds_count{controller="users",action="show"} 142 # HELP discourse_http_sql_duration_seconds Time spent in HTTP reqs in SQL in seconds # TYPE discourse_http_sql_duration_seconds summary discourse_http_sql_duration_seconds{controller="other",action="other",quantile="0.99"} 0.1874205535277724 discourse_http_sql_duration_seconds{controller="other",action="other",quantile="0.9"} 0.009177460800856352 discourse_http_sql_duration_seconds{controller="other",action="other",quantile="0.5"} 0.0 discourse_http_sql_duration_seconds{controller="other",action="other",quantile="0.1"} 0.0 discourse_http_sql_duration_seconds{controller="other",action="other",quantile="0.01"} 0.0 discourse_http_sql_duration_seconds_sum{controller="other",action="other"} 375.05954352673143 discourse_http_sql_duration_seconds_count{controller="other",action="other"} 58870 discourse_http_sql_duration_seconds{controller="topics",action="show",quantile="0.99"} 0.4393073697574437 discourse_http_sql_duration_seconds{controller="topics",action="show",quantile="0.9"} 0.2883010846562684 discourse_http_sql_duration_seconds{controller="topics",action="show",quantile="0.5"} 0.0 discourse_http_sql_duration_seconds{controller="topics",action="show",quantile="0.1"} 0.0 discourse_http_sql_duration_seconds{controller="topics",action="show",quantile="0.01"} 0.0 discourse_http_sql_duration_seconds_sum{controller="topics",action="show"} 13847.843351929914 discourse_http_sql_duration_seconds_count{controller="topics",action="show"} 143926 discourse_http_sql_duration_seconds{controller="list",action="latest",quantile="0.99"} 0.6733640311285853 discourse_http_sql_duration_seconds{controller="list",action="latest",quantile="0.9"} 0.5781291825696826 discourse_http_sql_duration_seconds{controller="list",action="latest",quantile="0.5"} 0.0 discourse_http_sql_duration_seconds{controller="list",action="latest",quantile="0.1"} 0.0 discourse_http_sql_duration_seconds{controller="list",action="latest",quantile="0.01"} 0.0 discourse_http_sql_duration_seconds_sum{controller="list",action="latest"} 324.1082574711181 discourse_http_sql_duration_seconds_count{controller="list",action="latest"} 1676 discourse_http_sql_duration_seconds{controller="users",action="show",quantile="0.99"} 0.07832226948812604 discourse_http_sql_duration_seconds{controller="users",action="show",quantile="0.9"} 0.07832226948812604 discourse_http_sql_duration_seconds{controller="users",action="show",quantile="0.5"} 0.07832226948812604 discourse_http_sql_duration_seconds{controller="users",action="show",quantile="0.1"} 0.07832226948812604 discourse_http_sql_duration_seconds{controller="users",action="show",quantile="0.01"} 0.07832226948812604 discourse_http_sql_duration_seconds_sum{controller="users",action="show"} 9.40852727741003 discourse_http_sql_duration_seconds_count{controller="users",action="show"} 142 # HELP discourse_http_net_duration_seconds Time spent in external network requests # TYPE discourse_http_net_duration_seconds summary discourse_http_net_duration_seconds{controller="other",action="other",quantile="0.99"} 0.0 discourse_http_net_duration_seconds{controller="other",action="other",quantile="0.9"} 0.0 discourse_http_net_duration_seconds{controller="other",action="other",quantile="0.5"} 0.0 discourse_http_net_duration_seconds{controller="other",action="other",quantile="0.1"} 0.0 discourse_http_net_duration_seconds{controller="other",action="other",quantile="0.01"} 0.0 discourse_http_net_duration_seconds_sum{controller="other",action="other"} 349.2784005794674 discourse_http_net_duration_seconds_count{controller="other",action="other"} 58870 discourse_http_net_duration_seconds{controller="topics",action="show",quantile="0.99"} 0.0 discourse_http_net_duration_seconds{controller="topics",action="show",quantile="0.9"} 0.0 discourse_http_net_duration_seconds{controller="topics",action="show",quantile="0.5"} 0.0 discourse_http_net_duration_seconds{controller="topics",action="show",quantile="0.1"} 0.0 discourse_http_net_duration_seconds{controller="topics",action="show",quantile="0.01"} 0.0 discourse_http_net_duration_seconds_sum{controller="topics",action="show"} 0.0 discourse_http_net_duration_seconds_count{controller="topics",action="show"} 143926 discourse_http_net_duration_seconds{controller="list",action="latest",quantile="0.99"} 0.0 discourse_http_net_duration_seconds{controller="list",action="latest",quantile="0.9"} 0.0 discourse_http_net_duration_seconds{controller="list",action="latest",quantile="0.5"} 0.0 discourse_http_net_duration_seconds{controller="list",action="latest",quantile="0.1"} 0.0 discourse_http_net_duration_seconds{controller="list",action="latest",quantile="0.01"} 0.0 discourse_http_net_duration_seconds_sum{controller="list",action="latest"} 0.0 discourse_http_net_duration_seconds_count{controller="list",action="latest"} 1676 discourse_http_net_duration_seconds{controller="users",action="show",quantile="0.99"} 0.0 discourse_http_net_duration_seconds{controller="users",action="show",quantile="0.9"} 0.0 discourse_http_net_duration_seconds{controller="users",action="show",quantile="0.5"} 0.0 discourse_http_net_duration_seconds{controller="users",action="show",quantile="0.1"} 0.0 discourse_http_net_duration_seconds{controller="users",action="show",quantile="0.01"} 0.0 discourse_http_net_duration_seconds_sum{controller="users",action="show"} 0.0 discourse_http_net_duration_seconds_count{controller="users",action="show"} 142 # HELP discourse_http_queue_duration_seconds Time spent queueing requests between NGINX and Ruby # TYPE discourse_http_queue_duration_seconds summary discourse_http_queue_duration_seconds{controller="other",action="other",quantile="0.99"} 7.055693626403809 discourse_http_queue_duration_seconds{controller="other",action="other",quantile="0.9"} 5.77228569984436 discourse_http_queue_duration_seconds{controller="other",action="other",quantile="0.5"} 3.628639221191406 discourse_http_queue_duration_seconds{controller="other",action="other",quantile="0.1"} 1.691039323806763 discourse_http_queue_duration_seconds{controller="other",action="other",quantile="0.01"} 0.1517226696014404 discourse_http_queue_duration_seconds_sum{controller="other",action="other"} 128979.19509196281 discourse_http_queue_duration_seconds_count{controller="other",action="other"} 58870 discourse_http_queue_duration_seconds{controller="topics",action="show",quantile="0.99"} 6.773485660552979 discourse_http_queue_duration_seconds{controller="topics",action="show",quantile="0.9"} 5.540528535842896 discourse_http_queue_duration_seconds{controller="topics",action="show",quantile="0.5"} 3.386407375335693 discourse_http_queue_duration_seconds{controller="topics",action="show",quantile="0.1"} 1.529008388519287 discourse_http_queue_duration_seconds{controller="topics",action="show",quantile="0.01"} 0.3203737735748291 discourse_http_queue_duration_seconds_sum{controller="topics",action="show"} 358491.9590744972 discourse_http_queue_duration_seconds_count{controller="topics",action="show"} 143926 discourse_http_queue_duration_seconds{controller="list",action="latest",quantile="0.99"} 7.470051765441895 discourse_http_queue_duration_seconds{controller="list",action="latest",quantile="0.9"} 5.970950841903687 discourse_http_queue_duration_seconds{controller="list",action="latest",quantile="0.5"} 3.233286619186401 discourse_http_queue_duration_seconds{controller="list",action="latest",quantile="0.1"} 1.421695709228516 discourse_http_queue_duration_seconds{controller="list",action="latest",quantile="0.01"} 0.4568729400634766 discourse_http_queue_duration_seconds_sum{controller="list",action="latest"} 5290.475468635559 discourse_http_queue_duration_seconds_count{controller="list",action="latest"} 1676 discourse_http_queue_duration_seconds{controller="users",action="show",quantile="0.99"} 5.078022718429565 discourse_http_queue_duration_seconds{controller="users",action="show",quantile="0.9"} 5.078022718429565 discourse_http_queue_duration_seconds{controller="users",action="show",quantile="0.5"} 5.078022718429565 discourse_http_queue_duration_seconds{controller="users",action="show",quantile="0.1"} 5.078022718429565 discourse_http_queue_duration_seconds{controller="users",action="show",quantile="0.01"} 5.078022718429565 discourse_http_queue_duration_seconds_sum{controller="users",action="show"} 381.56648421287537 discourse_http_queue_duration_seconds_count{controller="users",action="show"} 142 # HELP discourse_http_forced_anon_count Total count of logged in requests forced into anonymous mode # TYPE discourse_http_forced_anon_count counter discourse_http_forced_anon_count{db="default",api="web",verb="GET",type="regular",status="200"} 38907 discourse_http_forced_anon_count{db="default",api="web",verb="GET",type="regular",status="404"} 1236 # HELP discourse_http_sql_calls_per_request How many SQL statements ran per request # TYPE discourse_http_sql_calls_per_request gauge discourse_http_sql_calls_per_request{controller="users",action="show",logged_in="false"} 30 discourse_http_sql_calls_per_request{controller="list",action="latest",logged_in="true"} 271 discourse_http_sql_calls_per_request{controller="topics",action="show",logged_in="true"} 164 discourse_http_sql_calls_per_request{controller="users",action="show",logged_in="true"} 98 discourse_http_sql_calls_per_request{controller="list",action="latest",logged_in="false"} 223 discourse_http_sql_calls_per_request{controller="other",action="other",logged_in="false"} 6 discourse_http_sql_calls_per_request{controller="topics",action="show",logged_in="false"} 136 # HELP discourse_http_anon_cache_store How many a payload is stored in redis for anonymous cache # TYPE discourse_http_anon_cache_store counter discourse_http_anon_cache_store{controller="topics",action="show"} 21358 discourse_http_anon_cache_store{controller="other",action="other"} 260 discourse_http_anon_cache_store{controller="list",action="latest"} 286 # HELP discourse_http_anon_cache_hit How many a payload from redis is used for anonymous cache # TYPE discourse_http_anon_cache_hit counter discourse_http_anon_cache_hit{controller="other",action="other"} 7507 discourse_http_anon_cache_hit{controller="topics",action="show"} 82761 discourse_http_anon_cache_hit{controller="list",action="latest"} 929 # HELP discourse_heap_free_slots Free ruby heap slots # TYPE discourse_heap_free_slots gauge discourse_heap_free_slots{type="sidekiq",pid="131"} 12166 discourse_heap_free_slots{type="web",pid="163"} 119712 discourse_heap_free_slots{type="web",pid="183"} 69322 discourse_heap_free_slots{type="web",pid="454"} 31247 discourse_heap_free_slots{type="web",pid="201"} 162260 discourse_heap_free_slots{type="web",pid="503"} 83322 discourse_heap_free_slots{type="web",pid="308"} 72679 discourse_heap_free_slots{type="web",pid="328"} 79991 discourse_heap_free_slots{type="web",pid="396"} 204322 discourse_heap_free_slots{type="web",pid="219"} 221194 discourse_heap_free_slots{type="web",pid="374"} 122163 discourse_heap_free_slots{type="web",pid="270"} 43722 discourse_heap_free_slots{type="web",pid="418"} 155036 discourse_heap_free_slots{type="web",pid="434"} 158153 discourse_heap_free_slots{type="web",pid="479"} 116781 discourse_heap_free_slots{type="web",pid="350"} 167763 discourse_heap_free_slots{type="web",pid="287"} 95786 # HELP discourse_heap_live_slots Used ruby heap slots # TYPE discourse_heap_live_slots gauge discourse_heap_live_slots{type="sidekiq",pid="131"} 883728 discourse_heap_live_slots{type="web",pid="163"} 816121 discourse_heap_live_slots{type="web",pid="183"} 866514 discourse_heap_live_slots{type="web",pid="454"} 904588 discourse_heap_live_slots{type="web",pid="201"} 773576 discourse_heap_live_slots{type="web",pid="503"} 852518 discourse_heap_live_slots{type="web",pid="308"} 903104 discourse_heap_live_slots{type="web",pid="328"} 855035 discourse_heap_live_slots{type="web",pid="396"} 731505 discourse_heap_live_slots{type="web",pid="219"} 714638 discourse_heap_live_slots{type="web",pid="374"} 810818 discourse_heap_live_slots{type="web",pid="270"} 892104 discourse_heap_live_slots{type="web",pid="418"} 780793 discourse_heap_live_slots{type="web",pid="434"} 777678 discourse_heap_live_slots{type="web",pid="479"} 819045 discourse_heap_live_slots{type="web",pid="350"} 768077 discourse_heap_live_slots{type="web",pid="287"} 840051 # HELP discourse_v8_heap_size Total JavaScript V8 heap size (bytes) # TYPE discourse_v8_heap_size gauge discourse_v8_heap_size{type="sidekiq",pid="131"} 16392192 discourse_v8_heap_size{type="web",pid="163"} 15900672 discourse_v8_heap_size{type="web",pid="183"} 16424960 discourse_v8_heap_size{type="web",pid="454"} 17702912 discourse_v8_heap_size{type="web",pid="201"} 16433152 discourse_v8_heap_size{type="web",pid="503"} 17440768 discourse_v8_heap_size{type="web",pid="308"} 17178624 discourse_v8_heap_size{type="web",pid="328"} 16424960 discourse_v8_heap_size{type="web",pid="396"} 16162816 discourse_v8_heap_size{type="web",pid="219"} 16171008 discourse_v8_heap_size{type="web",pid="374"} 16171008 discourse_v8_heap_size{type="web",pid="270"} 17219584 discourse_v8_heap_size{type="web",pid="418"} 16957440 discourse_v8_heap_size{type="web",pid="434"} 15908864 discourse_v8_heap_size{type="web",pid="479"} 16654336 discourse_v8_heap_size{type="web",pid="350"} 16949248 discourse_v8_heap_size{type="web",pid="287"} 16654336 # HELP discourse_v8_used_heap_size Total used JavaScript V8 heap size (bytes) # TYPE discourse_v8_used_heap_size gauge discourse_v8_used_heap_size{type="sidekiq",pid="131"} 12932148 discourse_v8_used_heap_size{type="web",pid="163"} 12252448 discourse_v8_used_heap_size{type="web",pid="183"} 12526820 discourse_v8_used_heap_size{type="web",pid="454"} 12993364 discourse_v8_used_heap_size{type="web",pid="201"} 12462852 discourse_v8_used_heap_size{type="web",pid="503"} 12626676 discourse_v8_used_heap_size{type="web",pid="308"} 12702604 discourse_v8_used_heap_size{type="web",pid="328"} 12126824 discourse_v8_used_heap_size{type="web",pid="396"} 12269764 discourse_v8_used_heap_size{type="web",pid="219"} 12293912 discourse_v8_used_heap_size{type="web",pid="374"} 12210560 discourse_v8_used_heap_size{type="web",pid="270"} 12628080 discourse_v8_used_heap_size{type="web",pid="418"} 12538324 discourse_v8_used_heap_size{type="web",pid="434"} 12179620 discourse_v8_used_heap_size{type="web",pid="479"} 12715960 discourse_v8_used_heap_size{type="web",pid="350"} 12362024 discourse_v8_used_heap_size{type="web",pid="287"} 12654196 # HELP discourse_v8_physical_size Physical size consumed by V8 heaps # TYPE discourse_v8_physical_size gauge discourse_v8_physical_size{type="sidekiq",pid="131"} 14118508 discourse_v8_physical_size{type="web",pid="163"} 13315992 discourse_v8_physical_size{type="web",pid="183"} 13639620 discourse_v8_physical_size{type="web",pid="454"} 14201860 discourse_v8_physical_size{type="web",pid="201"} 13543736 discourse_v8_physical_size{type="web",pid="503"} 13779544 discourse_v8_physical_size{type="web",pid="308"} 13836520 discourse_v8_physical_size{type="web",pid="328"} 13250748 discourse_v8_physical_size{type="web",pid="396"} 13379408 discourse_v8_physical_size{type="web",pid="219"} 13401800 discourse_v8_physical_size{type="web",pid="374"} 13285780 discourse_v8_physical_size{type="web",pid="270"} 13776112 discourse_v8_physical_size{type="web",pid="418"} 13634224 discourse_v8_physical_size{type="web",pid="434"} 13221264 discourse_v8_physical_size{type="web",pid="479"} 13876884 discourse_v8_physical_size{type="web",pid="350"} 13471300 discourse_v8_physical_size{type="web",pid="287"} 13738052 # HELP discourse_v8_heap_count Number of V8 contexts running # TYPE discourse_v8_heap_count gauge discourse_v8_heap_count{type="sidekiq",pid="131"} 2 discourse_v8_heap_count{type="web",pid="163"} 2 discourse_v8_heap_count{type="web",pid="183"} 2 discourse_v8_heap_count{type="web",pid="454"} 2 discourse_v8_heap_count{type="web",pid="201"} 2 discourse_v8_heap_count{type="web",pid="503"} 2 discourse_v8_heap_count{type="web",pid="308"} 2 discourse_v8_heap_count{type="web",pid="328"} 2 discourse_v8_heap_count{type="web",pid="396"} 2 discourse_v8_heap_count{type="web",pid="219"} 2 discourse_v8_heap_count{type="web",pid="374"} 2 discourse_v8_heap_count{type="web",pid="270"} 2 discourse_v8_heap_count{type="web",pid="418"} 2 discourse_v8_heap_count{type="web",pid="434"} 2 discourse_v8_heap_count{type="web",pid="479"} 2 discourse_v8_heap_count{type="web",pid="350"} 2 discourse_v8_heap_count{type="web",pid="287"} 2 # HELP discourse_rss Total RSS used by process # TYPE discourse_rss gauge discourse_rss{type="sidekiq",pid="131"} 412172288 discourse_rss{type="web",pid="163"} 481796096 discourse_rss{type="web",pid="183"} 485736448 discourse_rss{type="web",pid="454"} 432590848 discourse_rss{type="web",pid="201"} 408793088 discourse_rss{type="web",pid="503"} 400375808 discourse_rss{type="web",pid="308"} 444911616 discourse_rss{type="web",pid="328"} 418308096 discourse_rss{type="web",pid="396"} 413487104 discourse_rss{type="web",pid="219"} 400482304 discourse_rss{type="web",pid="374"} 440221696 discourse_rss{type="web",pid="270"} 501215232 discourse_rss{type="web",pid="418"} 479678464 discourse_rss{type="web",pid="434"} 431116288 discourse_rss{type="web",pid="479"} 428220416 discourse_rss{type="web",pid="350"} 430444544 discourse_rss{type="web",pid="287"} 428658688 # HELP discourse_thread_count Total number of active threads per process # TYPE discourse_thread_count gauge discourse_thread_count{type="sidekiq",pid="131"} 22 discourse_thread_count{type="web",pid="163"} 8 discourse_thread_count{type="web",pid="183"} 8 discourse_thread_count{type="web",pid="454"} 8 discourse_thread_count{type="web",pid="201"} 8 discourse_thread_count{type="web",pid="503"} 8 discourse_thread_count{type="web",pid="308"} 8 discourse_thread_count{type="web",pid="328"} 8 discourse_thread_count{type="web",pid="396"} 8 discourse_thread_count{type="web",pid="219"} 8 discourse_thread_count{type="web",pid="374"} 8 discourse_thread_count{type="web",pid="270"} 8 discourse_thread_count{type="web",pid="418"} 8 discourse_thread_count{type="web",pid="434"} 8 discourse_thread_count{type="web",pid="479"} 8 discourse_thread_count{type="web",pid="350"} 8 discourse_thread_count{type="web",pid="287"} 8 # HELP discourse_deferred_jobs_queued Number of jobs queued in the deferred job queue # TYPE discourse_deferred_jobs_queued gauge discourse_deferred_jobs_queued{type="sidekiq",pid="131"} 0 discourse_deferred_jobs_queued{type="web",pid="163"} 0 discourse_deferred_jobs_queued{type="web",pid="183"} 0 discourse_deferred_jobs_queued{type="web",pid="454"} 0 discourse_deferred_jobs_queued{type="web",pid="201"} 0 discourse_deferred_jobs_queued{type="web",pid="503"} 0 discourse_deferred_jobs_queued{type="web",pid="308"} 0 discourse_deferred_jobs_queued{type="web",pid="328"} 0 discourse_deferred_jobs_queued{type="web",pid="396"} 0 discourse_deferred_jobs_queued{type="web",pid="219"} 0 discourse_deferred_jobs_queued{type="web",pid="374"} 0 discourse_deferred_jobs_queued{type="web",pid="270"} 0 discourse_deferred_jobs_queued{type="web",pid="418"} 0 discourse_deferred_jobs_queued{type="web",pid="434"} 0 discourse_deferred_jobs_queued{type="web",pid="479"} 0 discourse_deferred_jobs_queued{type="web",pid="350"} 0 discourse_deferred_jobs_queued{type="web",pid="287"} 0 # HELP discourse_active_record_connections_count Total number of connections in ActiveRecord's connection pools # TYPE discourse_active_record_connections_count gauge discourse_active_record_connections_count{type="sidekiq",pid="131",status="busy"} 2 discourse_active_record_connections_count{type="sidekiq",pid="131",status="dead"} 0 discourse_active_record_connections_count{type="sidekiq",pid="131",status="idle"} 4 discourse_active_record_connections_count{type="sidekiq",pid="131",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="163",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="163",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="163",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="163",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="183",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="183",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="183",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="183",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="454",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="454",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="454",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="454",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="201",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="201",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="201",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="201",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="503",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="503",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="503",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="503",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="308",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="308",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="308",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="308",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="328",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="328",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="328",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="328",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="396",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="396",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="396",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="396",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="219",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="219",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="219",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="219",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="374",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="374",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="374",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="374",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="270",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="270",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="270",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="270",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="418",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="418",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="418",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="418",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="434",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="434",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="434",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="434",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="479",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="479",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="479",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="479",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="350",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="350",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="350",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="350",status="waiting"} 0 discourse_active_record_connections_count{type="web",pid="287",status="busy"} 1 discourse_active_record_connections_count{type="web",pid="287",status="dead"} 0 discourse_active_record_connections_count{type="web",pid="287",status="idle"} 1 discourse_active_record_connections_count{type="web",pid="287",status="waiting"} 0 # HELP discourse_active_record_failover_count Count of ActiveRecord databases in a failover state # TYPE discourse_active_record_failover_count gauge discourse_active_record_failover_count{type="sidekiq",pid="131"} 0 discourse_active_record_failover_count{type="web",pid="163"} 0 discourse_active_record_failover_count{type="web",pid="183"} 0 discourse_active_record_failover_count{type="web",pid="454"} 0 discourse_active_record_failover_count{type="web",pid="201"} 0 discourse_active_record_failover_count{type="web",pid="503"} 0 discourse_active_record_failover_count{type="web",pid="308"} 0 discourse_active_record_failover_count{type="web",pid="328"} 0 discourse_active_record_failover_count{type="web",pid="396"} 0 discourse_active_record_failover_count{type="web",pid="219"} 0 discourse_active_record_failover_count{type="web",pid="374"} 0 discourse_active_record_failover_count{type="web",pid="270"} 0 discourse_active_record_failover_count{type="web",pid="418"} 0 discourse_active_record_failover_count{type="web",pid="434"} 0 discourse_active_record_failover_count{type="web",pid="479"} 0 discourse_active_record_failover_count{type="web",pid="350"} 0 discourse_active_record_failover_count{type="web",pid="287"} 0 # HELP discourse_redis_failover_count Count of Redis servers in a failover state # TYPE discourse_redis_failover_count gauge discourse_redis_failover_count{type="sidekiq",pid="131"} 0 discourse_redis_failover_count{type="web",pid="163"} 0 discourse_redis_failover_count{type="web",pid="183"} 0 discourse_redis_failover_count{type="web",pid="454"} 0 discourse_redis_failover_count{type="web",pid="201"} 0 discourse_redis_failover_count{type="web",pid="503"} 0 discourse_redis_failover_count{type="web",pid="308"} 0 discourse_redis_failover_count{type="web",pid="328"} 0 discourse_redis_failover_count{type="web",pid="396"} 0 discourse_redis_failover_count{type="web",pid="219"} 0 discourse_redis_failover_count{type="web",pid="374"} 0 discourse_redis_failover_count{type="web",pid="270"} 0 discourse_redis_failover_count{type="web",pid="418"} 0 discourse_redis_failover_count{type="web",pid="434"} 0 discourse_redis_failover_count{type="web",pid="479"} 0 discourse_redis_failover_count{type="web",pid="350"} 0 discourse_redis_failover_count{type="web",pid="287"} 0 # HELP discourse_major_gc_count Major GC operations by process # TYPE discourse_major_gc_count counter discourse_major_gc_count{type="sidekiq",pid="131"} 39 discourse_major_gc_count{type="web",pid="163"} 902 discourse_major_gc_count{type="web",pid="183"} 848 discourse_major_gc_count{type="web",pid="454"} 881 discourse_major_gc_count{type="web",pid="201"} 896 discourse_major_gc_count{type="web",pid="503"} 849 discourse_major_gc_count{type="web",pid="308"} 878 discourse_major_gc_count{type="web",pid="328"} 888 discourse_major_gc_count{type="web",pid="396"} 882 discourse_major_gc_count{type="web",pid="219"} 897 discourse_major_gc_count{type="web",pid="374"} 919 discourse_major_gc_count{type="web",pid="270"} 883 discourse_major_gc_count{type="web",pid="418"} 921 discourse_major_gc_count{type="web",pid="434"} 853 discourse_major_gc_count{type="web",pid="479"} 923 discourse_major_gc_count{type="web",pid="350"} 909 discourse_major_gc_count{type="web",pid="287"} 823 # HELP discourse_minor_gc_count Minor GC operations by process # TYPE discourse_minor_gc_count counter discourse_minor_gc_count{type="sidekiq",pid="131"} 251 discourse_minor_gc_count{type="web",pid="163"} 8220 discourse_minor_gc_count{type="web",pid="183"} 8171 discourse_minor_gc_count{type="web",pid="454"} 8435 discourse_minor_gc_count{type="web",pid="201"} 8249 discourse_minor_gc_count{type="web",pid="503"} 8252 discourse_minor_gc_count{type="web",pid="308"} 8356 discourse_minor_gc_count{type="web",pid="328"} 8276 discourse_minor_gc_count{type="web",pid="396"} 8377 discourse_minor_gc_count{type="web",pid="219"} 8311 discourse_minor_gc_count{type="web",pid="374"} 8521 discourse_minor_gc_count{type="web",pid="270"} 8256 discourse_minor_gc_count{type="web",pid="418"} 8503 discourse_minor_gc_count{type="web",pid="434"} 8332 discourse_minor_gc_count{type="web",pid="479"} 8577 discourse_minor_gc_count{type="web",pid="350"} 8480 discourse_minor_gc_count{type="web",pid="287"} 8241 # HELP discourse_total_allocated_objects Total number of allocateds objects by process # TYPE discourse_total_allocated_objects counter discourse_total_allocated_objects{type="sidekiq",pid="131"} 30060560 discourse_total_allocated_objects{type="web",pid="163"} 985232557 discourse_total_allocated_objects{type="web",pid="183"} 983805640 discourse_total_allocated_objects{type="web",pid="454"} 1008867616 discourse_total_allocated_objects{type="web",pid="201"} 988967755 discourse_total_allocated_objects{type="web",pid="503"} 998885290 discourse_total_allocated_objects{type="web",pid="308"} 1004069224 discourse_total_allocated_objects{type="web",pid="328"} 996554004 discourse_total_allocated_objects{type="web",pid="396"} 1013631939 discourse_total_allocated_objects{type="web",pid="219"} 999330059 discourse_total_allocated_objects{type="web",pid="374"} 1016924963 discourse_total_allocated_objects{type="web",pid="270"} 989042711 discourse_total_allocated_objects{type="web",pid="418"} 1024175277 discourse_total_allocated_objects{type="web",pid="434"} 1004407864 discourse_total_allocated_objects{type="web",pid="479"} 1025782126 discourse_total_allocated_objects{type="web",pid="350"} 1013371770 discourse_total_allocated_objects{type="web",pid="287"} 1003799360 # HELP discourse_scheduled_job_duration_seconds Total time spent in scheduled jobs # TYPE discourse_scheduled_job_duration_seconds counter discourse_scheduled_job_duration_seconds{job_name="Jobs::PendingUsersReminder"} 0.644000000000000011e0 discourse_scheduled_job_duration_seconds{job_name="Jobs::WhosOnlineGoingOffline"} 0.5267999999999998071e1 discourse_scheduled_job_duration_seconds{job_name="Jobs::PresenceChannelAutoLeave"} 0.6388000000000000119e1 discourse_scheduled_job_duration_seconds{job_name="Jobs::ProcessBadgeBacklog"} 0.25297000000000000432e2 discourse_scheduled_job_duration_seconds{job_name="Jobs::TopicTimerEnqueuer"} 0.5947000000000000053e1 discourse_scheduled_job_duration_seconds{job_name="Jobs::Heartbeat"} 0.1635000000000000038e1 discourse_scheduled_job_duration_seconds{job_name="Jobs::ProcessShelvedNotifications"} 0.1221000000000000029e1 discourse_scheduled_job_duration_seconds{job_name="Jobs::EnqueueOnceoffs"} 0.4557000000000000043e1 discourse_scheduled_job_duration_seconds{job_name="Jobs::BookmarkReminderNotifications"} 0.1019000000000000007e1 discourse_scheduled_job_duration_seconds{job_name="Jobs::CleanUpUploads"} 0.138000000000000002e0 discourse_scheduled_job_duration_seconds{job_name="Jobs::PollMailbox"} 0.851000000000000004e0 discourse_scheduled_job_duration_seconds{job_name="Jobs::TopRefreshToday"} 13.684000000000001 discourse_scheduled_job_duration_seconds{job_name="Jobs::DestroyOldDeletionStubs"} 0.272400000000000003e1 discourse_scheduled_job_duration_seconds{job_name="Jobs::ActivationReminderEmails"} 0.096 discourse_scheduled_job_duration_seconds{job_name="Jobs::UnsilenceUsers"} 0.8790000000000002 discourse_scheduled_job_duration_seconds{job_name="Jobs::PeriodicalUpdates"} 0.1145550000000000045e3 discourse_scheduled_job_duration_seconds{job_name="Jobs::MigrateUploadScheme"} 0.388000000000000004e0 discourse_scheduled_job_duration_seconds{job_name="Jobs::AboutStats"} 2.139 discourse_scheduled_job_duration_seconds{job_name="Jobs::CheckOutOfDateThemes"} 1.68 discourse_scheduled_job_duration_seconds{job_name="Jobs::EnqueueDigestEmails"} 0.156000000000000008e0 discourse_scheduled_job_duration_seconds{job_name="Jobs::DirectoryRefreshDaily"} 0.43910000000000002e1 discourse_scheduled_job_duration_seconds{job_name="Jobs::DashboardStats"} 0.153000000000000009e0 discourse_scheduled_job_duration_seconds{job_name="Jobs::PendingReviewablesReminder"} 0.060000000000000005 discourse_scheduled_job_duration_seconds{job_name="Jobs::CreateMissingAvatars"} 0.200000000000000005e0 discourse_scheduled_job_duration_seconds{job_name="Jobs::UpdateAnimatedUploads"} 0.133 discourse_scheduled_job_duration_seconds{job_name="Jobs::EnqueueSuspectUsers"} 0.12100000000000001 discourse_scheduled_job_duration_seconds{job_name="Jobs::UpdateSitemaps"} 3.831 discourse_scheduled_job_duration_seconds{job_name="Jobs::PendingQueuedPostsReminder"} 2.649 discourse_scheduled_job_duration_seconds{job_name="Jobs::ReindexSearch"} 1.431 discourse_scheduled_job_duration_seconds{job_name="Jobs::DisableBootstrapMode"} 0.038 discourse_scheduled_job_duration_seconds{job_name="Jobs::CleanUpUnusedStagedUsers"} 0.065 # HELP discourse_scheduled_job_count Total number of scheduled jobs executued # TYPE discourse_scheduled_job_count counter discourse_scheduled_job_count{job_name="Jobs::PendingUsersReminder"} 38 discourse_scheduled_job_count{job_name="Jobs::WhosOnlineGoingOffline"} 187 discourse_scheduled_job_count{job_name="Jobs::PresenceChannelAutoLeave"} 187 discourse_scheduled_job_count{job_name="Jobs::ProcessBadgeBacklog"} 188 discourse_scheduled_job_count{job_name="Jobs::TopicTimerEnqueuer"} 187 discourse_scheduled_job_count{job_name="Jobs::Heartbeat"} 63 discourse_scheduled_job_count{job_name="Jobs::ProcessShelvedNotifications"} 38 discourse_scheduled_job_count{job_name="Jobs::EnqueueOnceoffs"} 19 discourse_scheduled_job_count{job_name="Jobs::BookmarkReminderNotifications"} 38 discourse_scheduled_job_count{job_name="Jobs::CleanUpUploads"} 4 discourse_scheduled_job_count{job_name="Jobs::PollMailbox"} 38 discourse_scheduled_job_count{job_name="Jobs::TopRefreshToday"} 4 discourse_scheduled_job_count{job_name="Jobs::DestroyOldDeletionStubs"} 7 discourse_scheduled_job_count{job_name="Jobs::ActivationReminderEmails"} 2 discourse_scheduled_job_count{job_name="Jobs::UnsilenceUsers"} 13 discourse_scheduled_job_count{job_name="Jobs::PeriodicalUpdates"} 13 discourse_scheduled_job_count{job_name="Jobs::MigrateUploadScheme"} 19 discourse_scheduled_job_count{job_name="Jobs::AboutStats"} 6 discourse_scheduled_job_count{job_name="Jobs::CheckOutOfDateThemes"} 1 discourse_scheduled_job_count{job_name="Jobs::EnqueueDigestEmails"} 7 discourse_scheduled_job_count{job_name="Jobs::DirectoryRefreshDaily"} 3 discourse_scheduled_job_count{job_name="Jobs::DashboardStats"} 6 discourse_scheduled_job_count{job_name="Jobs::PendingReviewablesReminder"} 3 discourse_scheduled_job_count{job_name="Jobs::CreateMissingAvatars"} 3 discourse_scheduled_job_count{job_name="Jobs::UpdateAnimatedUploads"} 3 discourse_scheduled_job_count{job_name="Jobs::EnqueueSuspectUsers"} 2 discourse_scheduled_job_count{job_name="Jobs::UpdateSitemaps"} 3 discourse_scheduled_job_count{job_name="Jobs::PendingQueuedPostsReminder"} 3 discourse_scheduled_job_count{job_name="Jobs::ReindexSearch"} 1 discourse_scheduled_job_count{job_name="Jobs::DisableBootstrapMode"} 1 discourse_scheduled_job_count{job_name="Jobs::CleanUpUnusedStagedUsers"} 1 # HELP discourse_sidekiq_job_duration_seconds Total time spent in sidekiq jobs # TYPE discourse_sidekiq_job_duration_seconds counter discourse_sidekiq_job_duration_seconds{job_name="Jobs::RunHeartbeat"} 0.226169511675834654e0 discourse_sidekiq_job_duration_seconds{job_name="Jobs::NotifyMailingListSubscribers"} 0.8942126949783414593e1 discourse_sidekiq_job_duration_seconds{job_name="Jobs::PullHotlinkedImages"} 0.8282361692283302553e1 discourse_sidekiq_job_duration_seconds{job_name="Jobs::FeatureTopicUsers"} 0.1046757272724062204e2 discourse_sidekiq_job_duration_seconds{job_name="Jobs::BotInput"} 0.114331910074222833095e3 discourse_sidekiq_job_duration_seconds{job_name="Jobs::PostUpdateTopicTrackingState"} 0.17196641702670603947e2 discourse_sidekiq_job_duration_seconds{job_name="Jobs::CrawlTopicLink"} 0.1373126078015193336375e3 discourse_sidekiq_job_duration_seconds{job_name="Jobs::PostAlert"} 0.22092239051265641677e3 discourse_sidekiq_job_duration_seconds{job_name="Jobs::ProcessPost"} 0.30518020779779181136e3 # HELP discourse_sidekiq_job_count Total number of sidekiq jobs executed # TYPE discourse_sidekiq_job_count counter discourse_sidekiq_job_count{job_name="Jobs::RunHeartbeat"} 63 discourse_sidekiq_job_count{job_name="Jobs::NotifyMailingListSubscribers"} 94 discourse_sidekiq_job_count{job_name="Jobs::PullHotlinkedImages"} 100 discourse_sidekiq_job_count{job_name="Jobs::FeatureTopicUsers"} 101 discourse_sidekiq_job_count{job_name="Jobs::BotInput"} 217 discourse_sidekiq_job_count{job_name="Jobs::PostUpdateTopicTrackingState"} 101 discourse_sidekiq_job_count{job_name="Jobs::CrawlTopicLink"} 414 discourse_sidekiq_job_count{job_name="Jobs::PostAlert"} 106 discourse_sidekiq_job_count{job_name="Jobs::ProcessPost"} 106 # HELP discourse_postgres_readonly_mode Indicates whether site is in readonly mode due to PostgreSQL failover # TYPE discourse_postgres_readonly_mode gauge discourse_postgres_readonly_mode 0 # HELP discourse_redis_master_available Whether or not we have an active connection to the master Redis # TYPE discourse_redis_master_available gauge discourse_redis_master_available 1 # HELP discourse_redis_slave_available Whether or not we have an active connection a Redis slave # TYPE discourse_redis_slave_available gauge discourse_redis_slave_available 0 # HELP discourse_postgres_master_available Whether or not we have an active connection to the master PostgreSQL # TYPE discourse_postgres_master_available gauge discourse_postgres_master_available 1 # HELP discourse_postgres_replica_available Whether or not we have an active connection to the replica PostgreSQL # TYPE discourse_postgres_replica_available gauge # HELP discourse_active_app_reqs Number of active web requests in progress # TYPE discourse_active_app_reqs gauge discourse_active_app_reqs 16 # HELP discourse_queued_app_reqs Number of queued web requests # TYPE discourse_queued_app_reqs gauge discourse_queued_app_reqs 95 # HELP discourse_sidekiq_jobs_enqueued Number of jobs queued in the Sidekiq worker processes # TYPE discourse_sidekiq_jobs_enqueued gauge discourse_sidekiq_jobs_enqueued{queue="default"} 0 discourse_sidekiq_jobs_enqueued{queue="low"} 0 discourse_sidekiq_jobs_enqueued{queue="critical"} 0 discourse_sidekiq_jobs_enqueued{queue="ultra_low"} 0 # HELP discourse_sidekiq_processes Number of Sidekiq job processes # TYPE discourse_sidekiq_processes gauge discourse_sidekiq_processes 1 # HELP discourse_sidekiq_paused Whether or not Sidekiq is paused # TYPE discourse_sidekiq_paused gauge # HELP discourse_sidekiq_workers Total number of active sidekiq workers # TYPE discourse_sidekiq_workers gauge discourse_sidekiq_workers 5 # HELP discourse_sidekiq_jobs_stuck Number of sidekiq jobs which have been running for more than 120 minutes # TYPE discourse_sidekiq_jobs_stuck gauge # HELP discourse_scheduled_jobs_stuck Number of scheduled jobs which have been running for more than their expected duration # TYPE discourse_scheduled_jobs_stuck gauge # HELP discourse_missing_s3_uploads Number of missing uploads in S3 # TYPE discourse_missing_s3_uploads gauge # HELP discourse_version_info Labelled with `revision` (current core commit hash), and `version` (Discourse::VERSION::STRING) # TYPE discourse_version_info gauge discourse_version_info{revision="e3164671698aeef94cf2e142903a30e4688e9dfe",version="2.8.0.beta6"} 1 # HELP discourse_readonly_sites Count of sites currently in readonly mode, grouped by the relevant key from Discourse::READONLY_KEYS # TYPE discourse_readonly_sites gauge discourse_readonly_sites{key="readonly_mode"} 0 discourse_readonly_sites{key="readonly_mode:postgres"} 0 discourse_readonly_sites{key="readonly_mode:user"} 0 discourse_readonly_sites{key="readonly_mode:postgres_force"} 0 Questions: Is it DDOS or bruteforce attack on the forum or some bots traffic? OR its genuine traffic and we need to upgrade our droplet specs, if so, what should we upgrade, by how much? What should we do ? right now, our site is loading very slow, and navigation from 1 topic to another, the loading transition is like 5 seconds+ How to find out which traffic or resource is using this much traffic? Whether if its real human traffic or bots trying to hit down our site? What changes should we do in our app.yaml if we decide to upgrade our droplet specs? On users profile admin page, we see unique IP of each user, means CF is getting real IPs.? but they why we get 429 too many requests error when we uncomment the ratelimited template from app.yml How should we configure our CF DDOS settings? [image] Is there any way to configure our server/discourse to work in multi threaded or parallel way to support our high traffic using our same current specs droplet? If yes, how to configure multithreaded/parallel scaling in discourse to use multi core/CPUs droplets effectively and in best optimized way? We are using cloudflare and when we turn on CF I am under attack mode, then CPU usage goes down, but we cannot annoy our new users to solve captcha each time they use our site. Right now CF bot fight mode is ON and CF security level is high. We have tried rebuilding but after rebuild is done, it goes to normal but after few minutes CPU usage goes back to 100% Please someone help.

[image] Faizan_Zahid: We are using cloudflare and when we turn on CF I am under attack mode, then CPU usage goes down, but we cannot annoy our new users to solve captcha each time they use our site. Well, you need to decide whether the convenience or stable operation of your site is important to you. During DDoS attacks, there will always be some compromises in the operation of your site. You can try https://ddos-guard.net

[image] valsha: Well, you need to decide whether the convenience or stable operation of your site is important to you. During DDoS attacks, there will always be some compromises in the operation of your site. You can try https://ddos-guard.net How do i make sure that it is indeed a DDOS attack on my site? or its just real genuine traffic growth? First i need to find the real problem and then go towards the right solution.

[image] valsha: netstat -ntu|awk '{print $5}'|cut -d: -f1 -s|sort|uniq -c|sort -nk1 -r @valsha I just ran the above command, and my google analytics users were 73 and out put of above command is below and it shows 596 lines, now what does this means? 3178 127.0.0.1 3105 172.17.0.2 434 172.64.200.21 27 172.64.201.21 8 162.158.89.144 7 162.158.94.247 7 162.158.92.123 7 162.158.90.37 7 162.158.90.253 7 162.158.89.140 7 162.158.88.163 6 162.158.94.163 6 162.158.90.239 6 162.158.89.34 6 162.158.89.246 6 162.158.89.106 6 162.158.89.100 6 162.158.88.61 6 162.158.88.119 5 162.158.88.7 4 172.69.71.157 4 172.68.142.200 3 172.70.142.205 3 172.70.126.91 3 172.70.126.7 3 172.70.110.233 3 172.70.110.197 3 172.69.71.67 3 172.69.35.204 3 172.69.22.107 3 172.69.22.101 3 172.68.189.142 3 172.68.143.57 3 172.68.143.231 3 172.68.133.21 3 172.68.133.165 3 172.68.132.34 3 172.68.132.120 3 162.158.63.241 3 162.158.255.183 3 108.162.245.79 3 108.162.221.137 3 108.162.219.187 2 173.245.52.212 2 172.70.83.8 2 172.70.82.187 2 172.70.82.115 2 172.70.61.179 2 172.70.61.135 2 172.70.54.63 2 172.70.178.199 2 172.70.178.103 2 172.70.130.59 2 172.70.130.165 2 172.70.126.9 2 172.70.126.81 2 172.70.114.73 2 172.70.114.123 2 172.70.110.125 2 172.69.71.89 2 172.69.71.87 2 172.69.71.204 2 172.69.69.182 2 172.69.69.170 2 172.69.68.229 2 172.69.68.144 2 172.69.55.161 2 172.69.34.119 2 172.69.22.173 2 172.69.22.125 2 172.68.143.21 2 172.68.141.187 2 172.68.141.175 2 172.68.133.13 2 162.158.74.198 2 108.162.250.192 2 108.162.246.22 2 108.162.221.67 2 108.162.221.207 2 108.162.219.19 2 108.162.219.165 2 108.162.219.115 2 108.162.215.107 2 108.162.212.20 2 108.162.210.199 1 198.41.238.97 1 198.41.231.66 1 188.141.50.15 1 173.245.52.222 1 173.245.52.210 1 173.245.52.204 1 173.245.52.168 1 173.245.52.166 1 172.70.98.9 1 172.70.98.141 1 172.70.98.131 1 172.70.82.99 1 172.70.82.95 1 172.70.82.85 1 172.70.82.57 1 172.70.82.29 1 172.70.82.27 1 172.70.82.249 1 172.70.82.233 1 172.70.82.229 1 172.70.82.221 1 172.70.82.213 1 172.70.82.211 1 172.70.82.203 1 172.70.82.173 1 172.70.82.159 1 172.70.82.153 1 172.70.82.151 1 172.70.82.137 1 172.70.82.135 1 172.70.82.107 1 172.70.61.193 1 172.70.61.189 1 172.70.61.185 1 172.70.61.177 1 172.70.61.173 1 172.70.61.171 1 172.70.61.169 1 172.70.61.161 1 172.70.61.157 1 172.70.61.153 1 172.70.61.151 1 172.70.61.149 1 172.70.61.139 1 172.70.61.133 1 172.70.57.187 1 172.70.57.181 1 172.70.54.97 1 172.70.54.71 1 172.70.54.55 1 172.70.54.53 1 172.70.54.45 1 172.70.54.43 1 172.70.54.139 1 172.70.54.109 1 172.70.54.105 1 172.70.188.53 1 172.70.188.159 1 172.70.178.51 1 172.70.178.237 1 172.70.178.141 1 172.70.143.58 1 172.70.143.36 1 172.70.142.55 1 172.70.142.239 1 172.70.142.235 1 172.70.142.215 1 172.70.142.129 1 172.70.142.119 1 172.70.134.17 1 172.70.130.95 1 172.70.130.19 1 172.70.127.38 1 172.70.127.34 1 172.70.126.79 1 172.70.126.3 1 172.70.126.251 1 172.70.126.25 1 172.70.126.243 1 172.70.126.23 1 172.70.126.227 1 172.70.126.225 1 172.70.126.203 1 172.70.126.201 1 172.70.126.197 1 172.70.126.179 1 172.70.126.153 1 172.70.126.143 1 172.70.126.139 1 172.70.126.131 1 172.70.126.111 1 172.70.122.217 1 172.70.114.99 1 172.70.114.175 1 172.70.114.119 1 172.70.110.85 1 172.70.110.77 1 172.70.110.49 1 172.70.110.249 1 172.70.110.221 1 172.70.110.215 1 172.70.110.211 1 172.70.110.201 1 172.70.110.185 1 172.70.110.173 1 172.70.110.167 1 172.70.110.129 1 172.69.71.97 1 172.69.71.95 1 172.69.71.93 1 172.69.71.91 1 172.69.71.85 1 172.69.71.83 1 172.69.71.81 1 172.69.71.75 1 172.69.71.73 1 172.69.71.71 1 172.69.71.69 1 172.69.71.65 1 172.69.71.63 1 172.69.71.6 1 172.69.71.57 1 172.69.71.55 1 172.69.71.53 1 172.69.71.51 1 172.69.71.49 1 172.69.71.47 1 172.69.71.45 1 172.69.71.42 1 172.69.71.32 1 172.69.71.30 1 172.69.71.202 1 172.69.71.200 1 172.69.71.199 1 172.69.71.197 1 172.69.71.195 1 172.69.71.193 1 172.69.71.191 1 172.69.71.189 1 172.69.71.187 1 172.69.71.185 1 172.69.71.183 1 172.69.71.181 1 172.69.71.179 1 172.69.71.177 1 172.69.71.175 1 172.69.71.173 1 172.69.71.171 1 172.69.71.167 1 172.69.71.165 1 172.69.71.163 1 172.69.71.161 1 172.69.71.155 1 172.69.71.153 1 172.69.71.149 1 172.69.71.147 1 172.69.71.143 1 172.69.71.141 1 172.69.71.139 1 172.69.71.137 1 172.69.71.135 1 172.69.71.133 1 172.69.71.131 1 172.69.71.129 1 172.69.71.127 1 172.69.71.125 1 172.69.71.123 1 172.69.71.121 1 172.69.71.117 1 172.69.71.115 1 172.69.71.113 1 172.69.71.111 1 172.69.71.109 1 172.69.71.106 1 172.69.71.10 1 172.69.69.225 1 172.69.69.223 1 172.69.69.221 1 172.69.69.205 1 172.69.69.203 1 172.69.69.180 1 172.69.69.176 1 172.69.69.174 1 172.69.69.172 1 172.69.69.164 1 172.69.69.162 1 172.69.69.160 1 172.69.69.141 1 172.69.69.134 1 172.69.69.128 1 172.69.69.126 1 172.69.69.124 1 172.69.69.122 1 172.69.69.120 1 172.69.69.118 1 172.69.69.114 1 172.69.69.112 1 172.69.68.90 1 172.69.68.72 1 172.69.68.66 1 172.69.68.6 1 172.69.68.54 1 172.69.68.48 1 172.69.68.43 1 172.69.68.4 1 172.69.68.39 1 172.69.68.37 1 172.69.68.35 1 172.69.68.33 1 172.69.68.31 1 172.69.68.29 1 172.69.68.27 1 172.69.68.241 1 172.69.68.239 1 172.69.68.233 1 172.69.68.231 1 172.69.68.227 1 172.69.68.225 1 172.69.68.223 1 172.69.68.221 1 172.69.68.219 1 172.69.68.217 1 172.69.68.215 1 172.69.68.213 1 172.69.68.211 1 172.69.68.208 1 172.69.68.206 1 172.69.68.204 1 172.69.68.202 1 172.69.68.200 1 172.69.68.20 1 172.69.68.198 1 172.69.68.196 1 172.69.68.194 1 172.69.68.192 1 172.69.68.190 1 172.69.68.188 1 172.69.68.186 1 172.69.68.184 1 172.69.68.182 1 172.69.68.174 1 172.69.68.172 1 172.69.68.170 1 172.69.68.17 1 172.69.68.168 1 172.69.68.162 1 172.69.68.158 1 172.69.68.156 1 172.69.68.154 1 172.69.68.150 1 172.69.68.15 1 172.69.68.148 1 172.69.68.138 1 172.69.68.120 1 172.69.68.114 1 172.69.68.108 1 172.69.55.95 1 172.69.35.224 1 172.69.35.184 1 172.69.35.16 1 172.69.34.97 1 172.69.34.77 1 172.69.34.217 1 172.69.34.149 1 172.69.23.78 1 172.69.23.6 1 172.69.22.89 1 172.69.22.47 1 172.69.22.23 1 172.69.22.149 1 172.69.166.211 1 172.69.156.151 1 172.69.142.61 1 172.68.245.4 1 172.68.244.225 1 172.68.242.227 1 172.68.24.61 1 172.68.24.189 1 172.68.24.185 1 172.68.189.44 1 172.68.189.238 1 172.68.143.237 1 172.68.143.141 1 172.68.143.111 1 172.68.142.152 1 172.68.141.217 1 172.68.141.181 1 172.68.141.145 1 172.68.141.103 1 172.68.133.85 1 172.68.133.7 1 172.68.133.5 1 172.68.133.189 1 172.68.133.147 1 172.68.133.107 1 172.68.132.82 1 172.68.132.224 1 172.68.132.22 1 172.68.132.216 1 172.68.132.192 1 172.68.132.182 1 172.68.132.174 1 172.68.132.118 1 172.68.132.10 1 172.68.118.79 1 172.68.118.41 1 172.68.11.90 1 172.68.11.8 1 172.68.11.200 1 172.68.10.243 1 172.68.10.15 1 172.68.10.143 1 172.67.148.71 1 162.158.90.17 1 162.158.89.28 1 162.158.88.95 1 162.158.88.79 1 162.158.88.133 1 162.158.75.43 1 162.158.75.189 1 162.158.75.173 1 162.158.75.163 1 162.158.74.96 1 162.158.74.236 1 162.158.74.218 1 162.158.74.196 1 162.158.74.152 1 162.158.74.140 1 162.158.74.138 1 162.158.74.130 1 162.158.63.7 1 162.158.63.65 1 162.158.63.115 1 162.158.62.94 1 162.158.62.40 1 162.158.62.32 1 162.158.62.248 1 162.158.62.138 1 162.158.62.122 1 162.158.62.118 1 162.158.62.110 1 162.158.50.4 1 162.158.50.26 1 162.158.255.35 1 162.158.255.229 1 162.158.255.215 1 162.158.2.219 1 162.158.2.207 1 162.158.187.94 1 162.158.183.170 1 162.158.166.56 1 162.158.165.27 1 162.158.165.183 1 162.158.165.169 1 162.158.159.123 1 162.158.158.152 1 162.158.158.122 1 162.158.155.65 1 162.158.155.53 1 162.158.154.136 1 162.158.122.83 1 162.158.119.152 1 162.158.107.92 1 162.158.107.50 1 162.158.107.46 1 162.158.107.18 1 162.158.106.83 1 162.158.106.225 1 162.158.106.189 1 162.158.106.173 1 162.158.106.165 1 162.158.106.155 1 141.101.99.234 1 141.101.99.230 1 141.101.98.195 1 141.101.98.149 1 141.101.97.104 1 141.101.96.31 1 141.101.69.248 1 141.101.107.85 1 141.101.107.75 1 141.101.107.57 1 141.101.107.49 1 141.101.107.179 1 141.101.107.143 1 141.101.105.79 1 141.101.104.174 1 141.101.104.114 1 108.162.250.254 1 108.162.250.228 1 108.162.250.132 1 108.162.250.124 1 108.162.249.43 1 108.162.249.37 1 108.162.246.62 1 108.162.246.6 1 108.162.246.4 1 108.162.246.36 1 108.162.246.188 1 108.162.246.160 1 108.162.246.134 1 108.162.245.81 1 108.162.245.55 1 108.162.245.25 1 108.162.245.243 1 108.162.245.225 1 108.162.245.21 1 108.162.245.199 1 108.162.245.167 1 108.162.245.143 1 108.162.245.135 1 108.162.245.117 1 108.162.245.11 1 108.162.245.101 1 108.162.241.63 1 108.162.241.145 1 108.162.241.123 1 108.162.241.111 1 108.162.229.43 1 108.162.221.99 1 108.162.221.97 1 108.162.221.91 1 108.162.221.89 1 108.162.221.87 1 108.162.221.85 1 108.162.221.83 1 108.162.221.81 1 108.162.221.79 1 108.162.221.77 1 108.162.221.75 1 108.162.221.73 1 108.162.221.71 1 108.162.221.69 1 108.162.221.65 1 108.162.221.53 1 108.162.221.249 1 108.162.221.247 1 108.162.221.243 1 108.162.221.241 1 108.162.221.239 1 108.162.221.237 1 108.162.221.235 1 108.162.221.233 1 108.162.221.231 1 108.162.221.229 1 108.162.221.227 1 108.162.221.223 1 108.162.221.221 1 108.162.221.217 1 108.162.221.215 1 108.162.221.213 1 108.162.221.209 1 108.162.221.205 1 108.162.221.201 1 108.162.221.199 1 108.162.221.195 1 108.162.221.193 1 108.162.221.191 1 108.162.221.187 1 108.162.221.185 1 108.162.221.181 1 108.162.221.179 1 108.162.221.177 1 108.162.221.173 1 108.162.221.171 1 108.162.221.167 1 108.162.221.165 1 108.162.221.163 1 108.162.221.161 1 108.162.221.159 1 108.162.221.157 1 108.162.221.155 1 108.162.221.153 1 108.162.221.151 1 108.162.221.149 1 108.162.221.147 1 108.162.221.145 1 108.162.221.143 1 108.162.221.141 1 108.162.221.139 1 108.162.221.129 1 108.162.221.127 1 108.162.221.125 1 108.162.221.123 1 108.162.221.121 1 108.162.221.119 1 108.162.221.113 1 108.162.221.111 1 108.162.221.109 1 108.162.221.107 1 108.162.221.105 1 108.162.221.101 1 108.162.219.97 1 108.162.219.81 1 108.162.219.57 1 108.162.219.45 1 108.162.219.43 1 108.162.219.15 1 108.162.219.145 1 108.162.216.75 1 108.162.216.229 1 108.162.215.87 1 108.162.215.55 1 108.162.215.185 1 108.162.215.159 1 108.162.212.36 1 108.162.212.200 1 108.162.212.198 1 108.162.210.207 1 108.162.210.197 1 108.162.210.175

Do you have some theme component that’s doing api calls? Plugins?

Those look like they are all CloudFlare IP ranges. Did you setup your cloudflare correctly in app.yml?

[image] Faizan_Zahid: now what does this means? add `cloudflare.template.yml` to the end of the templates section in `app.yml` .

[image] Faizan_Zahid: As you can see, we have commented the ratelimited template because it was giving 429 nginx too many request errors to our normal users despite of very low actual traffic on our site. I think you removed this restriction, because you did not have it configured cloudflare.template.yml all users had one IP address and these restrictions were applied to them. you can try: install cloudflare.template.yml install web.ratelimited.template.yml

[image] MarcP: Those look like they are all CloudFlare IP ranges. Did you setup your cloudflare correctly in app.yml? Even if Cloudflare is set up correctly, you will still see CF IP addresses in netstat, so this is not necessarily a sign of the cloudflare.template.yml not being set up correctly.

[image] RGJ: Even if Cloudflare is set up correctly, you will still see CF IP addresses in netstat , so this is not necessarily a sign of the cloudflare.template.yml not being set up correctly. Is the purpose of this config: cloudflare.template.yml to see real IP addresses instead of Cloudflare addresses?

Discourse overloaded real traffic or DDOS? 100% CPU usage despite of decent traffic and high specs server

Product Support Self-hosting

valsha (KingPin) September 19, 2021, 10:14pm 4

try
netstat -ntu|awk '{print $5}'|cut -d: -f1 -s|sort|uniq -c|sort -nk1 -r
and compare with your online users

Topic		Replies	Views
My discourse speed is very slow Self-hosting	24	5488	March 4, 2021
Due to extreme load, this is temporarily being shown to everyone... when it's not really the case Self-hosting server-resources	19	2060	July 21, 2023
I just hit my CPU cap on the Digital Ocean 2GB/2xCPU plan Self-hosting hosting	35	18132	April 30, 2018
Discourse installation has been getting slower and slower and slower Self-hosting server-resources	36	2348	April 15, 2023
freeCodeCamp.org Discourse is Collapsing from Spammer Scripts Self-hosting	15	5334	July 3, 2020

Discourse overloaded real traffic or DDOS? 100% CPU usage despite of decent traffic and high specs server

Related topics