Slow forum responses due to internal IP routing issues


(Sam Nazarko) #1

Hi Discourse,

We’re currently experiencing some slow repsonses on our forum at https://discourse.osmc.tv. We have the rate limiting template commented. Are there any other rate limiting policies enabled by default? We have just released a new update for our OSMC software and we are currently experiencing an increase in traffic.

I’d appreciate it if you could give any pointers as to what the cause may be or how we can debug the issue. While we are seeing more traffic, it seems that we should be able to handle this. CPU usage is very low, our network traffic is acceptable and memory use is not erroneously high either.

Someone recently pointed out that when I edited a thread that it updated infront of their eyes (cool stuff!) Could Discourse’s AJAX features be causing a large number of keep alive connections to our box?

Best

Sam


(Michael Downey) #2

What exactly do you mean by this? What does the user experience, or alternatively, what are you seeing particularly that’s “slow”?


(Sam Saffron) #3
  • Are you using our Docker install?

  • What size droplet is this?

  • Enable performance reports in site settings, go to “/sidekiq/scheduler” … run it, paste an anonymised day of stats here.


(Sam Nazarko) #4

Hi

By slow:

  • We see the spinner icon for 10-15 seconds when clicking a post. Slack takes a long time to resolve new posts, and clicking the notifications icon takes a long time to resolve posts.

We are using Docker indeed. Not on a Droplet, but our server is:

Xeon X3440
1Gbit port
8GB RAM

2GB of RAM is in use, iftop shows low TX/RX, htop low CPU and I can wget speed test files in a timely manner without issue. netstat not showing any SYN_RECV nastiness and the box seems stable.

I see some Jobs::*Stats but none called AnonymisedStats.

This is what I see

Thanks for the help

Sam


(Sam Saffron) #5

The job is called: DailyPerformanceReport

but be sure to enable the site setting first. What is your container config, with that amount of power / ram you would increase unicorn workers. 10 seconds is unreasonable

EDIT:

I just clicked around the forum and it seems quite snappy.


(Sam Nazarko) #6

Hi

Unfortunately this issue seems to be sporadic. It is performant for me at the moment as well.

  • We have 4 unicorn workers. Should this be higher?
  • db_shared_buffers is 512M
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
#  - "templates/web.ssl.template.yml"
  - "templates/sshd.template.yml"
#  - "templates/web.ratelimited.template.yml"

I have the daily performance report on now. I’ll send you a PM as well

Sam


(Kane York) #7

So this means that the real IP is not making it all the way to Discourse.

You need to add a set_real_ip_from directive to the container’s nginx.

The directive will look like this: set_real_ip_from 10.0.1.200;

Add this to the run block in your app.yml:

 - sed -i "/sendfile on;/a set_real_ip_from 10.0.1.200;" /etc/nginx/conf.d/discourse.conf

If it’s behind cloudflare, you would use cloudflare.template.yml instead.


(Sam Nazarko) #8

Hi

We’re not behind CloudFlare, we’re behind a Pound -> Varnish configuration which uses 192.168.x.x IP ranges. We’re passing the real IP as HTTP_X_FORWARDED_FOR and this seems accurate as I can see users IP addresses when I click on their profile (even new users who have joined in the last few hours).

It seems the 172.x range is used by Docker itself… I assume Discourse being unable to identify the real IP is artificially rate limiting us here, as shown in Rate Limiting under Settings?

Any idea where the 172.x IP is coming from? Is it Docker? What should I substitute set_real_ip_from 10.0.1.200; to? As I say, we seem to be passing the user’s real IP OK and other applications are picking this up, albeit they are not behind Docker

Sam


(Kane York) #9

Do set_real_ip_from 192.168.0.0/16; though that MIGHT already be trusted… Do you have anything else in front of the container? If so, make sure it’s added there too.


(Sam Nazarko) #10

Varnish, Pound and Discourse are all running on the one box here, so requests would be coming from 127.0.0.1 from the load balancer and that’s the IP I’d expect to see (if any) if the real IPs weren’t showing. Nonetheless, I’ll add the set_real_ip_from values and see what happens

Sam


Varnish and Discourse