Cloudflare 뒤에서 끝없는 로딩

[MichaIng]

We moved our server from one VPS provider to another and I upgraded the instance via launcher rebuild to the latest version as well, from 3.5.0.beta3 to 3.5.0.beta4.

The instance was always running fine behind Cloudflare, but now trying to access it leads to endless 5 dots loading animation.

I have a hosts file entry on my local system to bypass Cloudflare, since my ISP (Deutsche Telekom AG) has shitty peering policies so that access is very low through Cloudflare by times. So at first I did not recognize the issue, as access without Cloudflare works fine. So I upgraded the instance, and hence am now not sure whether the changed VPS or the Discourse upgrade was the relevant change. I assured via VPN and mobile network, that the issue really is Cloudflare itself now, not the bad peering of my ISP, and also other users face the same issue. Old and new VPS have IPv6 available, and the whole system is exactly the same, transferred as raw image file.

There are zero error message, neither in browser (console), nor by the host system’s proxy, nor by Nginx within the container, nor by Rails or anywhere else. The HTML documents and several scripts load fine, and comparing them with those served when bypassing Cloudflare show that everything (I checked) is identical. Also response headers look mostly the same, aside of a few Cloudflare-specific ones, of course. The last things I see being loaded is the mini profiler:

Of course clearing browser cache, using private windows etc all did not change something. Also clearing/disabling the Cloudflare cache does not help, so the cache is not the issue. I temporarily disabled the CF cache completely for the whole forum.

Notable to say that the forum runs on a sub path behind an Apache proxy on the host, following these instructions: Serve Discourse from a subfolder (path prefix) instead of a subdomain
Previously, we created just a ln -s . forum symlink instead of the uploads/backups symlinks and doubled rewrites of the instructions, which worked well for years (and also now without Cloudflare), but as part of my debugging efforts I switched to those instructions to assure the internal proxy applies all rules as intended. Trusted header is CF-Connecting-IP, though I enabled cloudflare.template.yml as well, even that it doubles things somewhat. And I also tried to change back and forth various parts of these templates and above instructions, also in attempt to check whether the proxy IP headers do any difference, as missing CF-Connecting-IP is one thing when bypassing Cloudflare.

At this point I am completely out of ideas, have not a single trace where the issue might be coming from, not a single related log/output anywhere. Through Cloudflare, Discourse just hangs in loading animation without further trace.

I hope someone has an idea how to debug this, or whether there was a change between 3.5.0.beta3 and 3.5.0.beta4 which could be related. I guess a downgrade is problematic?

This is the instance: https://dietpi.com/forum/
EDIT: I disabled Cloudflare for now. But there is a CNAME which is still passed through Cloudflare, so those two can be compared: https://www.dietpi.com/forum/

[RGJ (Richard - Communiteq)]

Interesting problem.

It’s simply https://www.dietpi.com/forum/ that is hanging forever.

$ wget https://www.dietpi.com/forum/
--2025-05-03 10:52:18--  https://www.dietpi.com/forum/
Resolving www.dietpi.com (www.dietpi.com)... 104.21.12.65, 172.67.193.183, 2606:4700:3035::6815:c41, ...
Connecting to www.dietpi.com (www.dietpi.com)|104.21.12.65|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: âindex.html.1â

    [<=>                            

The interesting thing is that calls like https://www.dietpi.com/forum/site.json do succeed.

https://www.dietpi.com/forum/t/why-there-are-two-kernals-in-my-raspberry-pi4/23355 does not work and hangs forever, but
https://www.dietpi.com/forum/t/why-there-are-two-kernals-in-my-raspberry-pi4/23355.json does.

[MichaIng]

Interesting indeed. I recognize just now that the HTML documents are not loading completely but hang at some point. I compared /forum/ in both cases and thought they were identical, but probably I was focusing too much on the head, while parts of the body at the bottom are missing.

That last line when loading via Cloudflare:

      <discourse-assets-json>
        <div class="hidden" id="data-preloaded" data-preloaded="{&quot;topic_list&quot;:&quot;{\&quot;users\&quot;...\&quot;:false,\&quot;allowed_iframes\&quot;:\&quot;https://dietpi.com/forum/discobot/certificate.svg\

Needed to truncate it as it vastly exceeds the character limit for a post. The document usually continues like this:

      <discourse-assets-json>
        <div class="hidden" id="data-preloaded" data-preloaded="{&quot;topic_list&quot;:&quot;{\&quot;users\&quot;...\&quot;:false,\&quot;allowed_iframes\&quot;:\&quot;https://dietpi.com/forum/discobot/certificate.svg\&quot;,\&quot;can_permanently_delete...

The other pages hang at the very same point. I think we are one something here.

EDIT: Ah wait, I checked wrong, other pages hang elsewhere. So it is not this particular HTML element/attribute.

Yeah each page/HTML document hangs at the very same character when loaded via browser again and again, in private window etc. But a different page hangs at a different point. And when loading those via curl, they as well hang always at the very same point, but a different one, and wget again always hangs at a same point, but a slightly different one. Very weird.

[pfaffman (Jay Pfaffman)]

Do you have some optimization enabled?

[MichaIng]

Nope, no (content) optimizations. I did have the 103 Early Hints feature enabled, but disabled it already as attempt to solve things. Tried the same with the protocol settings, but that did not change anything eiher:

Btw, there is no content-length response header, could that cause an issue? I mean it does not exist when bypassing Cloudflare either, but probably Cloudflare then has some issue? EDIT: No seems to be normal for dynamic pages, same with our Wordpress and Matomo pages which however do not cause any issues.

And another find when playing with curl. Printing to STDOUT results in full HTML document being displayed, but it still hangs and at the end:

  <p class='powered-by-link'>Powered by <a href="https://www.discourse.org">Discourse</a>, best viewed with JavaScript enabled</p>
</footer>



  </body>

</html>

But when trying to save it via -o or simple redirect, or even just piping into grep, it hangs at a different place:

            <div class="link-bottom-line">
                <a href='/forum/c/general-discussion/7' class='badge-wrapper bullet'>
                  <span class='badge-category-bg' style='background-color: #F7941D'></span>
                  <span class='badge-category clear-badge'>
                    <span class='category-name'>General Discussion</span>
                  </span>
                </a>

And I can 100% replicate this very same 73728 bytes when accessing https://www.dietpi.com/forum/ with curl without just printing it to console right away. This is so weird .

---

So:

• All clients hang at loading any HTML document from our Discourse instance.
• Each client hangs at the very same byte when loading the same page.
• Different clients hang at different points, but at the very same byte when repeating with the same client.
• Each page hangs at a different point in the document and at different downloaded size.
• The same tool curl hangs at different points when just printing to STDOUT vs piping or storing the document somewhere.
• wget is able to download the full document (at least https://www.dietpi.com/forum/) to a file, but still hangs at the end, same when curl https://www.dietpi.com/forum/ prints the full document to console, but hangs at the end.

[RGJ (Richard - Communiteq)]

@MichaIng

Yeah each page/HTML document hangs at the very same character when loaded via browser again and again, in private window etc.

I think that might be buffering. But when investigating, I noticed something else.

wget -O - https://www.dietpi.com/forum/latest

Ends with

  </body>
</html>

But the connection is never closed.

Theory: there is a configuration issue somewhere where there is a mismatch in HTTP versions or headers (like keep alive connection) and this only becomes a problem when a document is larger than X (I suspect 64KB).

[MichaIng]

Yes, wget always downloads the whole document, and curl does so when printing directly to console, but connection is not closed. Same happens with much smaller documents, like I tested a 14k one of a topic with only 2 posts. But even the smaller ones usually are not fully downloaded by curl when piping or storing to file, neither in browser.

Both tools always show HTTP/2, and in Cloudflare I have HTTP/2 origin requests enabled. But worth to test using other HTTP versions explicitly. Yesterday I disabled all protocol settings in Cloudflare seen on the screenshot above, and it did not help. But I’ll try again. Also I can enable access logs on the server to see the actually incoming request from Cloudflare.

[MichaIng]

I tried all combinations of supported HTTP (1.1-3) and TLS (1.2-1.3) versions, but that does not make a difference. I also disabled HTTP3 support, HTTP2 origin requests this 0-RTT connection resumption again. No difference, curl keeps hanging at exactly the same 73,728 bytes of https://www.dietpi.com/forum/.

Regarding the theory of too large document sizes, https://www.dietpi.com/dietpi-software.html has 199,475 bytes and loads perfectly fine. I should mention that the server (same webserver) hosts a static website, MkDocs instance, Wordpress, Matomo, which all work perfectly fine. Also there is a Grafana instance where the front webserver acts as proxy via UNIX socket.

But I agree it seems to be related to buffers or chunk sizes or something like that. It is just weird that the downloaded size until hang varies so much between clients and pages, while it remains exactly the same despite changing protocol versions, and that the connection is not even closed when the document has been fully downloaded. Like if the stop signal is missing, though I am missing insights into HTTP at this point. Hence I thought about content-length header, but that one is obviously not mandatory.

The webserver also acts as proxy for the Discourse container via UNIX socket. I could enable the TCP listener to make the Discourse instance additionally available without the proxy (leaving the Nginx within the container, of course).

[RGJ (Richard - Communiteq)]

Could you try KeepAlive Off in Apache?

@MichaIng

I could enable the TCP listener to make the Discourse instance additionally available without the proxy (leaving the Nginx within the container, of course).

I guess that would at least possibly rule out the webserver, so that would be worth a try.

[MichaIng]

@RGJ

Could you try KeepAlive Off in Apache?

No change. Also from Apache docs:

In addition, a Keep-Alive connection with an HTTP/1.0 client can only be used when the length of the content is known in advance.

Hence as of missing content-length probably it makes sense that it is not used anyway for this request.

@RGJ

I guess that would at least possibly rule out the webserver, so that would be worth a try.

Since it requires a rebuild, I’ll do a little later when our common website activity is at a minimum. Um, I am just thinking about HTTPS … looks like I need to do some custom adjustments to the internal Nginx config to keep the UNIX socket functional as well as plain HTTP connections, while listening on an additional port for HTTPS with the TLS certificates from the host, but without HTTPS redirect/enforcement. … and additional plain HTTP TCP port would be also interesting, for clients which can ignore HSTS.

[tdperry]

Are you by chance using RocketLoader in CloudFlare? I know with some other scripts it causes issues.
Also, did you clear the CF cache?
Are you using inbound rules on CF that may have been tied to your old VPS IP address and not updated to the new one?

[MichaIng]

No RocketLoader: Note that as of above tests with curl and wget, which do not interpret any syntax, hence do not load any JavaScript or styles or anything else, the problem is that the download of the raw HTML document always hangs.

Cloudflare cache is not active for the forum, the raw HTML documents were never cached anyway.

No VPS-specific rules. Generally not rules for the forum, aside of to bypass cache. The issue appears in both cases, so cache isn’t the issue either.

[MichaIng]

Discourse 컨테이너 호스트에서 Apache2 프록시를 우회하는 테스트를 수행하고, Cloudflare에서 강제 HTTPS 리디렉트를 비활성화하여 curl을 통해 일반 HTTP 연결도 테스트하던 중, 마침내 Cloudflare에서 원인을 찾았습니다:

Automatic HTTPS Rewrites

VPS 전환 시 무엇이 변경되었는지, 그리고/또는 Discourse 3.5.0.beta3에서 3.5.0.beta4로 업그레이드할 때 무엇이 변경되었는지, 혹은 우연히 같은 시점에 Discourse에서 다른 문제가 발생했는지는 정확히 모르겠습니다. 하지만 Discourse의 HTML, CSS 또는 JavaScript 문서 중 무언가가 Cloudflare의 내장 URL HTTPS 재작성 기능을 막히는 것처럼 보입니다. 부분적이고 중단된 curl 요청이 실제로는 관련이 없었던 것 같기도 하고, 아니면 관련이 있을 수도 있습니다. 브라우저의 네트워크 탭에서 HTML 문서의 일부 내용을 볼 수 있다는 점이 이상합니다. 이는 HTTPS 재작성 기능이 문서를 스트리밍하는 동안 처리하고 있기 때문인 것처럼 보입니다.

혹시 다른 분도 인스턴스와 Cloudflare 계정을 가지고 계신다면, 이것이 일반적인 문제인지 아니면 우리 인스턴스/설정과 관련된 것인지 테스트해 주실 수 있을까요?

참고로, 프록시 우회 및 HTTP를 테스트하면서 프록시를 통한 연결은 활성화된 상태로 유지하기 위해, 컨테이너 내부의 Nginx 구성을 다음과 같이 수동으로 조정하면 완벽하게 잘 동작합니다:

root@dietpi-discourse:/var/www/discourse# cat /etc/nginx/conf.d/outlets/server/10-http.conf
listen unix:/shared/nginx.http.sock;
set_real_ip_from unix:;
listen 8080;
listen [::]:8080;
listen 8443 ssl;
listen [::]:8443 ssl;
http2 on;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;

ssl_certificate /shared/fullchain.cer;
ssl_certificate_key /shared/dietpi.com.key;

ssl_session_tickets off;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:1m;

물론 HTTPS 리디렉트와 HSTS 헤더를 제거하고, 추가된 포트들을 노출시키는 것이 중요합니다.

[MichaIng]

또 다른 발견: 우리는 text/html 응답의 </head> 종료 태그 바로 앞에 Matomo 트래커 코드를 추가하기 위해 mod_sed를 사용하고 있습니다. Cloudflare의 자동 HTTPS 리라이트가 활성화되어 있음에도 불구하고, Discourse에 대해 이 기능을 비활성화하거나(또는 Apache2 프록시를 우회하면) 문제가 해결됩니다. 둘 중 하나를 비활성화하는 것만으로도 문제가 해결됩니다. 다른 모든 페이지에서는 이 조합이 잘 작동하며, 실패하는 포럼 페이지보다 훨씬 큰 매우 큰 페이지에서도 마찬가지입니다. 따라서 프록시에서 먼저 적용되는 mod_set과 Cloudflare에 의해 임베드되는 URL 리라이트라는 두 가지 필터가 문서 크기나 청크 크기 등과 관련하여 무언가를 깨뜨리는 것일 수 있습니다.

현재 우리는 Discourse 테마 편집을 통해 트래커를 임베드하고 있으며, 추가로 Cloudflare의 자동 HTTPS 리라이트도 비활성화했습니다. 웹사이트 전체에 혼합 콘텐츠(mixed content)가 없습니다. 만약 존재한다면, Cloudflare가 이를 영원히 가려 두는 것보다 확인하고 수정하는 것이 좋습니다.

[pfaffman (Jay Pfaffman)]

@MichaIng

curl을 통해 일반 HTTP 연결도 테스트하려면,

그것이 작동할 수 없다고 확신합니다.

어떤 문제를 해결하려는지 정확히 모르지만, app.yml에서 force_https를 활성화해야 할 가능성이 큽니다.

[MichaIng]

이름만 보고 "Cloudflare Automatic HTTPS Rewrites"가 무엇을 하는지 오해할 수 있다고 생각합니다. Cloudflare에는 두 가지 기능이 있습니다:

• "Always Use HTTPS"는 모든 일반 HTTP 요청을 HTTPS로 리디렉션하며, 이는 Discourse의 force_https와 동일한 동작을 합니다. 이전에는 이 두 기능이 모두 활성화되어 있었으며, HTTPS가 문제와 무관한지, 또는 Discourse 페이지의 끝없는 로딩과 멈추는 curl 요청과 관련이 있는지 테스트하기 위해 이 두 기능을 모두 비활성화했습니다. 이 조치로 문제가 완벽하게 해결되었지만, 이는 HTTPS 요청에 대해서도 마찬가지였으며, 단순히 같은 시점에 "Cloudflare Automatic HTTPS Rewrites"를 비활성화했기 때문이었습니다.
• "Cloudflare Automatic HTTPS Rewrites"는 HTML, CSS, JavaScript 문서 내의 모든 내장된 일반 HTTP URL을 HTTPS 버전으로 교체합니다. 이는 Cloudflare가 해당 호스트가 HTTPS를 통해 접근 가능하다고 판단하는 경우(HSTS 사전 로드 목록 등 기준)에만 적용됩니다. 이 기능은 혼합 콘텐츠(mixed content) 경고 발생을 방지하기 위한 것입니다.

Cloudflare, 호스트 프록시 또는 Discourse에서 HTTPS를 강제하거나 강제하지 않는 것은 중요하지 않습니다. 실제 문제를 일으킨 것은 호스트 프록시의 mod_sed 필터와 Cloudflare의 내장된 일반 HTTP 수정이 결합된 것이었습니다. 즉, 문서 내용이 두 단계의 필터를 통과한 것입니다. 문제는 실제 콘텐츠가 변경된 것이 아니라(우리 사이트에는 혼합 콘텐츠가 없으므로 "Cloudflare Automatic HTTPS Rewrites"는 문서 본문을 실제로 변경하지 않습니다) 아마도 청크(chunk), 버퍼 또는 타이밍과 관련이 있을 것입니다.