Full site CDN acceleration for Discourse

문서 자체 호스팅
,
1

Fastly , CloudFlare and a few other CDNs offer a mode where they accelerate dynamic content.

In a nutshell you point your domain IP address at the CDN and the CDN will intelligently decide how to deal with the request.

  • Static content can be easily served from cache
  • Dynamic content can be routed to the site.

This provides some advantages over only shipping static assets which is covered in the CDN howto.

  • You can elect for “shielding” that protects your site from traffic spikes.
  • Dynamic content can be accelerated using techniques like delta compression. (note: in general our payload fits in 1 RTT so this has less of an impact)
  • SSL negotiation can happen at the edge cutting on expensive round trips for negotiation.

If you enable full site acceleration with a CDN it is critical you follow 3 rules

  1. The “message bus” must be served from the origin.

  2. You need to set up X-Forwarded-For trust. For Cloudflare, add cloudflare.template.yml to your app.yml file.

  3. Be extra careful with techniques that apply optimisation to the site, stuff like Rocket Loader can stop Discourse from working. Discourse is already heavily optimised, this is not needed.

To serve “long polling” requests from a different domain, set the hidden site setting long_polling_base_url to the origin server. This is best configured by adding the DISCOURSE_LONG_POLLING_BASE_URL environment variable in your app.yml, or via the Rails console.

For example, if your site is on “http://forum.example.com” you should set up http://forum-direct.example.com/ to plug into the site setting. If you don’t your site will be broken.

If you are fronting Discourse using Varnish you probably want to follow the same trick here and bypass Varnish for the message bus requests.

Boring technical notes:

Achieving a working message bus on a completely different domain is quite challenging. Our message bus is aware of which user is polling, the other domain may have no cookie set up so untouched there are two issues. Firstly, you can’t even make standard ajax requests cross domain without a huge CORS dance.

Secondly, we needed a mechanism to inform the other domain who the user is so we can poll for the correct information.

When long polling base url is changed, Discourse ships an extra meta tag that shares a “cross domain” auth token. This token is passed using a custom header back to the message bus. The token expires after 7 days or as soon as the user logs off.

You can see most of the implementation here: FEATURE: allow long polling to go to a different url · discourse/discourse@aa9b3bb · GitHub

Last edited by @JammyDodger 2024-05-25T11:15:40Z

Check documentPerform check on document:
24개의 좋아요
How do you setup Cloudflare?
Request header field Dont-Chunk is not allowed by Access-Control-Allow-Headers in preflight response
SSL on Discourse / DO sub-domain of Heroku hosted domain
How do you setup Cloudflare?
Discourse and cloudflare
My discourse speed is very slow
MKJ's Opinionated Discourse Deployment Configuration
Varnish config for discourse
Security tab shows me a different location due to Cloudflare
Optimizing the number of Unicorns and buffer size
429 too many requests good or bad ? (please help)
Add a new site (android) 'was not found'
Query regarding adding Cloudflare
How to use Cloudflare to protect a Discourse site
Please add "spammer" as reason for user suspension
Cloudflare or plugins breaking my Discourse instance?
Update from 2.4.0.beta9 to 2.4.0.beta10 yields page with no content - Cloudflare Rocket Loader
Large amount of errors on /user-badges.json
MessageBus short polling is not working
After upgrade the content security policy script src for GTM
Migrate from AWS to Digital Ocean with 2 containers, spaces and 2 CDNs
2

I don’t know what it means… fits in 1 RTT?

3

1 Round trip, read up abount TCP congestion control, initial windows and so on.

http://samsaffron.com/archive/2012/03/01/why-upgrading-your-linux-kernel-will-make-your-customers-much-happier

4

UPDATE 2015-04-02: I just made the full example more complete.

I am confused with this definition.

But I think this should clarify it. Please, don’t hesitate to tell me if i’m doing something wrong.

The confusing part is that if “http://some-origin.com/”. If you are behind Fastly, you have to use a CNAME entry and then you have to have a sub domain name and not the top level.

Background: In DNS, a top level domain name (i.e. “some-origin.com”) can only have A records. Since Fastly requires we use a CNAME entry, we have no choice but to use a sub domain name.

Let’s say that we will then use “http://discourse.some-origin.com/” to serve our Discourse forum so we can use Fastly.

Now there’s this thing called “long polling” which is basically an OPTION HTTP request with a long time before returning anything. If we use the Fastly or Varnish address, as Discourse would by default, Varnish will time out and “long polling” won’t work.

More background: Varnish has this option to bypass in known contexts through vcl_pipe which is roughly a raw TCP socket. But Fastly doesn’t offer it because of the size of their setup.

Proposed setup

Let’s enable long polling and expose our site under Fastly. We’ll need two names, one pointing to Fastly’s and the other to the IP addresses we give within the service dashboard.

  1. discourse.some-origin.com that’s our desired Discourse site domain name
  2. discoursepolling.some-origin.com (pick any name) that we’ll configure in Discourse to access directly to our public facing frontend web server

In my case, I generally have many web apps running that are only accessible from my internal network. I refer to them as “upstream”; the same term NGINX uses in their config. Since this number of web apps you would host on a site can fluctuate, you might still want the number public IP address to remain stable. That’s why I setup a NGINX server in front that proxies to internal web app server. I refer to them as “frontends”.

Let’s say you have two public facing frontends running NGINX.

Those would be the ones you setup in Fastly like this.

Fastly service configuration, at Hosts tab
Fastly service configuration, at Hosts tab1268×436 5.48 KB

Here we see two Backends in Fastly pannel at Configure -> Hosts.

Notice that in this example i’m using 443 port because my backends are configured to communicate between Fastly and my frontends through TLS. But you don’t need to.

Quoting again @sam;

[quote=“sam, post:1, topic:21467”]
To server “long polling” requests from a different domain, set the Site Setting long polling base url to the origin server.[/quote]

Really means here is that we would have to put one of those IP addresses in Discourse settings.

What I’d recommend is to create a list of A entries for all your frontends.

In the end we need three things:

  1. What’s the public name that Fastly will serve
  2. Which IPs are the frontends
  3. Which hostname we want to use for long polling and we’ll add it to our VirtualHost

The zone file would look like this;

# The public facing URL
discourse.some-origin.com.  IN CNAME global.prod.fastly.net.

# The list of IP addresses you’d give to Fastly as origins/backends
frontends.some-origin.com.  IN A 8.8.8.113
frontends.some-origin.com.  IN A 8.8.8.115

# The long polling URL entry
discoursepolling.some-origin.com.  IN CNAME frontends.some-origin.com.

That way you can setup the “long polling base url” correctly without setting a single point of failure.

Then, we can go in Discourse admin zone and adjust the “long polling base url” to our other domain name.

# /etc/nginx/sites-enabled/10-discourse

# Let’s redirect to SSL, in case somebody tries to access the direct IP with
# host header.
server {
    listen      80;
    server_name discoursepolling.some-origin.com discourse.some-origin.com;
    include     common_params;
    return      301 https://$server_name$request_uri;
}

server {
    listen      443 ssl;
    server_name discoursepolling.some-origin.com discourse.some-origin.com;
    # Rest of NGINX server block
    # Also, I would make a condition if we are in discoursepolling but not
    # under using anything specific to polling.
    # #TODO; find paths specific to polling
}

To see if it works; look at your web browser developer tool “Network inspector” for /poll calls on discoursepolling.some-origin.com, and see if you have 200 OK status code.

3개의 좋아요
5

To clarify something here, in a multisite configuration, all sites should use the same long polling url? It looks to me like the this line is making that a requirement:

thanks!

Edit: No wait, that doesn’t work.

base site: example.com
long polling url: origin.example.com

multisite 1: mysite.com

If mysite uses origin.example.com as the long polling address I get:

XMLHttpRequest cannot load https://origin.example.com/message-bus/634dd18187094c6c950c0bf14f74c239/poll. Response to preflight request doesn't pass access control check: The 'Access-Control-Allow-Origin' header has a value 'https://example.com' that is not equal to the supplied origin. Origin 'https://mysite.com' is therefore not allowed access.

If mysite uses it’s own long polling origin as the domain I get this:

XMLHttpRequest cannot load https://origin.mysite.com/message-bus/b35c9c8e958f44f78d0d4773dc6d75f3/poll. Response to preflight request doesn't pass access control check: The 'Access-Control-Allow-Origin' header has a value 'https://example.com' that is not equal to the supplied origin. Origin 'https://mysite.com' is therefore not allowed access.

Is this because of "Access-Control-Allow-Origin" => Discourse.base_url_no_prefix ?

6

I have noticed there is no “cloudfront.template.yml” in discourse_docker/templates/. So I am wondering:
Can CloudFront work using the same techniques ?

7

Also, can we use http2 ? Is the long polling stuff still needed when using http2 ?

8

If you’re using a the supported Docker-based install, HTTP2 should be working automatically! :sunny:

Long polling is still needed for notifications to appear live.

4개의 좋아요
9

I think if you have cloudfront setup, it’s only delivering specific objects (images), rather than the site/application in it’s entirety with js and so on.

So the only thing you need is to have the correct cloudfront url for those images.

10

Some additional notes for anyone who decides to use HTTPS together with Full site CDN acceleration

  1. Discourse internally uses the value of SiteSetting.force_https to decide if your access-control-allow-origin: is the HTTP or HTTPS version of your site. If while polling you see an error in the browser console along the lines of preflight request doesn't pass access control check: The 'Access-Control-Allow-Origin' header has a value http doesn't match https, double check your force_https setting. Also note the protocol in your DISCOURSE_CORS_ORIGIN in your container definition (http|https) will be overridden by force_https.
    Don’t forget to add DISCORSE_ENABLE_CORS: true in your container definition.

  2. If you were planning to only do HTTPS from your end users to your CDN, and then HTTP from your CDN to your actual Discourse web_only containers, lots of custom configuration will be required.

  3. If your CDN is serving your site on HTTPS, then whatever Long Polling URL you setup must also be on HTTPS, so even if the CDN is handling your HTTPS, you must still setup HTTPS on your Discourse servers. If you run into an error about Same-origin policy, double check that you’re not trying to connect to HTTP instead of HTTPS

    • If you use letsencrypt to generate your certificates, note that fullchain.pem => /shared/ssl/ssl.crt (ssl_certificate)
    • privkey.pem => /shared/ssl/ssl.key (ssl_certificate_key)

  4. You might use the following templates in your container definition:

  - "templates/web.template.yml"
  - "templates/web.ssl.template.yml"
  - "templates/fastly.template.yml"
  • Towards the end of the hook:ssl inside templates/web.ssl.template.yml you’ll see this block being added to your /etc/nginx/conf.d/discourse.conf.
if ($http_host != $$ENV_DISCOURSE_HOSTNAME) {
    rewrite (.*) https://$$ENV_DISCOURSE_HOSTNAME$1 permanent;
}
  • You’ll need to need to comment these lines out, otherwise you’re long polling attempts always serve up 301 redirects back to your origin, instead of respecting whatever you set in SiteSetting.long_polling_base_url
  1. The easiest way I’ve found to do this is to copy templates/web.ssl.template.yml to local.web.ssl.template.yml and just remove those extra lines, and update your container reference to use your local template. If you go that route, you should periodically diff your local version with the origional version, because there are some security improvements that are regularly incorporated into this template.

Some of the error messages you’ll run into until things are configured correctly.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://polling.example.com/message-bus/37c91c51e6cd4b0c95288b8fc29a0480/poll. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

Reason: CORS header ‘Access-Control-Allow-Origin’ missing

Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource

Failed to load https://polling.example.com/message-bus/8caefcec2cf94de3ae684c4b953a1084/poll: Response for preflight is invalid (redirect)

4개의 좋아요
External nginx pagespeed module causes Security Policy violation
11

You can do this with a - replace in your app.yml similar to how it is described on Setting up Let’s Encrypt with Multiple Domains.

  after_ssl:
    - replace:
        filename: "/etc/nginx/conf.d/discourse.conf"
        from: /return 301 https.+/
        to: |
          return 301 https://$host$request_uri;

    - replace:
        filename: "/etc/nginx/conf.d/discourse.conf"
        from: /gzip on;[^\}]+\}/m
        to: |
          gzip on;
          add_header Strict-Transport-Security 'max-age=31536000'; # remember the certificate for a year and automatically connect to HTTPS for this domain
1개의 좋아요
12

@brahn, I was looking at using a pups replace line, but I couldn’t figure out how to do a multi-line match in pups…

Note that templates/web.ssl.template.yml is inside of the port 443 block, not the port 80 block.

Is there a way to use the pups, replace command to match the entire mult-line string?

if ($http_host != www.example.com) {
   rewrite (.*) https://www.example.com$1 permanent;
}

The most direct way I thought of inside the container definition is an exec line running perl, awk or sed to do the multiline replace… but then you’ve got shell escaping along with your target language to disentangle before it will work…

13

The first replace takes care of the redirect from http to https for multisite. Perhaps that one is not relevant for you.

The second replace is multi-line. It replaces everything from line 33 to line 39 that was added by the web.ssl template.

It just removes that whole rewrite block. I could not figure out what purpose it serves and it breaks mutlisite so…

You could do this in your app.yml:

after_ssl:
    - replace:
        filename: "/etc/nginx/conf.d/discourse.conf"
        from: /gzip on;[^\}]+\}/m
        to: |
          gzip on;
          add_header Strict-Transport-Security 'max-age=31536000'; # remember the certificate for a year and automatically connect to HTTPS for this domain
          if ($http_host != www.example.com) {
            rewrite (.*) https://www.example.com$1 permanent;
          }
1개의 좋아요
14

@sam

I’m curious to know one thing:
Eg. My discourse is hosted on forum.example.com
Can I set long polling base to poll.example.org which points to the same server IP?
Will it have any impact considering CSP?

3개의 좋아요
15

안녕하세요. 왜 Discourse를 사용하면서 Cloudflare를 사용하지 않는 것을 권장하시나요?

새로운 VPS(Hetzner의 Debian)에 설치하려고 하고 있으며, 제 작은 서버에는 Cloudflare를 켜 두는 것이 유용할 것 같다고 생각했습니다.

시간 내어 답변해 주셔서 감사합니다.

16

CloudFlare는 일반적인 CDN이 아니라 네트워크 프록시입니다. 그들 성능 기능 중 일부는 클라이언트와 서버 사이의 코드를 변경합니다.

이러한 기능을 켜두면 Discourse가 새로운 방식으로 깨질 수 있습니다. 기능을 끄면 브라우저의 Discourse 앱과 서버 사이에 추가적인 네트워크 홉만 추가됩니다. 홉이 많을수록 인터페이스 반응성이 떨어집니다.

7개의 좋아요
17

음, 저는 Hetzner VPS를 사용 중인데, 서버를 안전하게 보호하기 위해(예: 공격 발생 시) Cloudflare을 사용하는 것이 좋다는 내용을 읽었습니다. 또한 저는 독일이 아닌 미국에 있으므로 CDN 사용도 괜찮을 것 같습니다.

어떻게 생각하시나요?

18

공격에 대해 걱정해야 하는지에 대해서는 언급하지 않겠습니다. 해당 판단은 직접 내려야 하지만, FUD(공포, 불확실성, 의심)에 현혹되지 마세요.

성능 기능이 활성화된 상태로 두시면 여기서 지원이 불가능합니다. 위에서 언급했듯이, 이러한 기능들은 좋은 영향을 주지 않는 방식으로 JavaScript에 간섭을 일으킵니다.

다른 모든 성능 기능을 비활성화하면 기본 자산 캐싱은 작동할 수 있습니다.

그러나 설치 및 설정 과정에서 Cloudflare가 활성화되어 있으면 인증서 등록이 실패합니다. 초기 등록 시 Cloudflare 프록시 뒤에서는 Let’s Encrypt가 지원되지 않습니다.

4개의 좋아요
19

답변해 주셔서 감사합니다, 스티븐. 디스코urs를 설치하려고 하는데 문제가 생겨서 클라우드플레어 때문일 수도 있다고 생각했어요.

그럼 DNS 관리에도 사용할 수 없다는 건가요? 클라우드플레어 없이 서버를 보호하면서 디스코urs를 유지하려면 어떻게 해야 하나요?

20

Cloudflare 제어판에서 호스트명 옆의 주황색 구름을 눌러 구름이 회색으로 변하게 하세요.
그 후 Discourse를 설치하세요. 서버를 보호하려면 회색 구름을 눌러 주황색으로 변경하세요. 단, 먼저 모든 성능 기능을 비활성화했는지 확인하세요.

8개의 좋아요