Analyze Discourse performance using NGINX logs

Top 30 not found urls (404s)

Url                                                                              Count
---                                                                              -----
GET /uploads/favicon/favicon.ico HTTP/1.1                                          388 
GET /uploads/favicon/manifest.json HTTP/1.1

Background: I created a folder for favicons under uploads. Then I use a favicon generator to generate a bunch of favicons and a complete declaration HTML:

The favicon seems to work, but it puzzles me why the .ico & .json appear as 404? The .PNG files work.

Discourse 1.5 beta branch and Docker setup.

Question about something Iā€™m seeing in reports:

Over the past week my instance has been having some sporadic performance issues, which appears to coincide with significantly larger than normal Anonymous entries in the NGINX reports.

e.g., normal day:

Top 30 users by Server Load

Username    Duration Reqs Routes                                                                          
--------    -------- ---- ------                                                                          

[Anonymous]   646.15 2862 topics/show(252.48) categories/index(204.90) -(60.69) user_avatars/show(34.32)   
                          topics/feed(12.88)  
Top 100 routes by Server Load

Route                                   Duration Reqs  Mobile      
-----                                   -------- ----  ------      
topics/show                               408.92  1464  136 (9.29)% 
categories/index                          351.09   637   45 (7.06)% 
user_avatars/show                         164.52  2470 426 (17.25)% 
topics/timings (POST)                     128.37  2581 655 (25.38)% 
list/latest                                73.90   358  52 (14.53)% 
-                                          60.69   635  68 (10.71)% 
...
- (POST)                                    0.27    27   3 (11.11)% 

But yesterday, when we were having some issues:

Top 30 users by Server Load

Username    Duration Reqs Routes                                                                          
--------    -------- ---- ------                                                                          

[Anonymous] 11747.63 9557 - (POST)(5430.92) -(5104.66) topics/show(609.38) categories/index(256.95)        
                          user_avatars/show(148.13)                                                        
Top 100 routes by Server Load

Route                                Duration Reqs  Mobile      
-----                                -------- ----  ------      
- (POST)                              5430.92  6576    3 (0.05)% 
-                                     5104.66   609   45 (7.39)% 
user_avatars/show                      764.95  2279 353 (15.49)% 
topics/timings (POST)                  761.55  1791  153 (8.54)% 
topics/show                            750.75  1386 175 (12.63)% 
draft/update (POST)                    441.09   508    5 (0.98)% 
categories/index                       426.02   341   26 (7.62)% 
list/latest                            206.17   464  53 (11.42)% 

Am I making a bad assumption when I think this is probably not valuable traffic? Some sort of bot or whatnot? Havenā€™t had an increase any new users or anything, a little more (maybe ~10%) more usage from actual members over the past couple weeks.

Itā€™s just really annoying, our setup is fine for our normal and even our high activity levels, but it hasnā€™t been holding up well over the past week.

Hey, first of all I really love this feature, well done! :heart:

But this log leaks api_keys to moderators, if an api_key request gets to the ā€œTop 30 urls by Server Loadā€, since the topic sits at the staff category.

2 Likes

This is implemented in

https://github.com/discourse/discourse/commit/85d7526219655ff9d9a28cf618ab24555cacc5d0

4 Likes

Mmh, is it just me or has the nginx-log changed? My site sends the request correctly, it wouldnā€™t work otherwise:

GET /latest.json?api_key=xxxAPI_KEYxxx&api_username=anonymous

But since some older 1.7 beta (3?) the api_key is leaked again because the order is different (according to the performance report):

GET /latest.json?xxxAPI_KEYxxx=[FILTERED]&api_username=anonymous
3 Likes

Hmm you should look into this @tgxworld

2 Likes

Thanks for reporting.

One of the commits I pushed broke it :frowning:

https://github.com/discourse/discourse/commit/4895723f7b812d1db4b91ec3c12dc3ef6cf655fa

3 Likes

I accidentally deleted Staff category. Where the bot will post logs?

You canā€™t if you are not be able to create a new category_staff_id via rails C.
Unfortunately my systeam is not familiar with ruby ā€‹ā€‹on rails commands as he knows docker, but if you find a solution please, share it.

For references:

1 Like

I tried to use nginx_analyze.rb script to analize strange traffic to my v1.8.0.beta11 forum but didnā€™t find it in /var/www/discourse/script after ./launcher enter app.
It was moved somewhere or removed?

The nginx analyzer was moved into a plugin. Iā€™ve updated the OP so you can find the updated path there.

5 Likes

Ok, it works. Thanks :slight_smile:

1 Like

Iā€™m running the container behind an outer NGINX reverse proxy and I cannot seem to get the above type listing of server load by IP. When I link the container via unix socket, I get this:

Top 30 IPs by Server Load

IP Address Duration Reqs
---------- -------- ----
unix:        127.97 1239 
----------------------------------------------------------------------------------------------------

And when I do it via TCP/IP, I get this:

Top 30 IPs by Server Load

IP Address Duration Reqs
---------- -------- ----
 172.17.0.1  166.27 1712 
----------------------------------------------------------------------------------------------------

Since that IP is a private IP, the latter probably has something to do with my VPS provider has set things up, but is it by design that the unix socket eats the client IPs?

1 Like

Hi there. Iā€™m trying to enable performance reports for Discourse SaaS installed from Amazon Marketplace. First of all, I donā€™t see launcher anywhere on the server, thus I can perform:
cd /var/discourse
./launcher rebuild app

/opt/bitnami/ctlscript.sh seems to be closest to launcher, but it is missing rebuild, enter commands.

Can someone provide alternative instructions for SaaS offering? Thanks.

We only support the Docker based installs here, sorry!

1 Like

Bitnami is great for many things, but they donā€™t update their images often enough to be able to keep up with Discourse (and docker) updates. As a result youā€™re really on your own as far as support and maintenance is concerned. I learned this the hard way myself.

While I know this is not ideal, Iā€™d recommend you move your installation to one thatā€™s supported by the Discourse community. You can find suitable hosting here, here or here, build a clean image, install docker, install discourse and import the existing community via the discourse backup/restore function.

Itā€™ll be worth it in the long run.

4 Likes

Has the nginx log analysis plugin gone away?

Oh this is so out of date.

I think @tgxworld moved it to a pluginā€¦ and then later we removed the plugin. I wonder where the code went.

Putting stale on this and a big warning at the top.

Not to rehash ancient history, but was there a reason it was pulled, like a lack of support?

It happened so long ago I donā€™t recall 100% but I think quite a few people complained about having a topic with 300 performance reports in it after a year, it turned out that nobody really looked at it.

I think it was a lot more like lack of demand vs support.

1 Like