When editing and saving extremely long posts (around 100,000 characters), we encountered a backend timeout issue. The server becomes unresponsive during the save operation, resulting in 502/504 errors. The frontend console shows the following error stack:
ajax-error.js:36:15
l ajax-error.js:36
u ajax-error.js:75
d ajax-error.js:84
Ember 41
update rest.js:72
update rest.js:72
save rest.js:115
editPost composer.js:1147
Ember 6
I performed some comparative tests which suggest the bottleneck is in the diff calculation between old and new versions:
Directly editing a long post A into long post B and saving consistently triggers a 502/504 timeout.
Clearing the long post first, saving a short placeholder (e.g. 5 characters), then pasting the full new content B and saving completes quickly.
It seems the current Diff engine struggles with extreme cases involving very long text combined with a high percentage of changes. Would it be possible to add a performance fallback mechanism? For example, when the text is extremely long and the modification ratio is high, the system could treat it as a “Complete Rewrite” instead of performing a detailed line-by-line diff.
Does the team have any plans to optimize the Diff handling for large posts, or introduce some kind of graceful degradation / protection for such scenarios? Another idea is to allow the save to succeed first, and compute the diff asynchronously afterwards.
Meanwhile, the server-side Unicorn logs captured the exact point of the timeout, confirming that the worker was killed while processing the markdown diff:
Unicorn worker received USR2 signal indicating it is about to timeout, dumping backtrace for main thread
config/unicorn.conf.rb:204:in `backtrace'
config/unicorn.conf.rb:204:in `block (2 levels) in reload'
/var/www/discourse/lib/discourse_diff.rb:172:in `[]'
/var/www/discourse/lib/discourse_diff.rb:172:in `tokenize_markdown'
/var/www/discourse/lib/discourse_diff.rb:115:in `side_by_side_markdown'
/var/www/discourse/app/serializers/post_revision_serializer.rb:128:in `body_changes'
...
In Feb I added `diff_too_complex: “The diff is too complex to display. Please try viewing individual smaller edits.”` Our diff algorithm will just stop if stuff gets too complicated.
Are you running latest here? It should handle these cases, unless you have a very very under-powered server.
I was on the early version from this year earlier, but I just upgraded to the absolute latest version!
After the upgrade, I successfully saw the “The diff is too complex to display” message, and honestly, it’s awesome. At least it doesn’t hard-crash and still allows me to go ahead and delete edit histories.
It also feels like the freezing/lagging issue when editing huge posts might be fixed now. I’m not entirely sure yet, but during my last test, it lagged for just a bit and then thankfully successfully processed the edited version.
I believe the “diff too complex” threshold can be tweaked, so if you feel there is such a need, we can figure it out (there’s a lot of factors to take into account, so there might not be a one-size-fit-all solution)
I have tested this multiple times now. Even with edits complex enough to trigger the “The diff is too complex to display” notice, the post has never failed to publish again.
I believe this issue has been completely resolved in the latest version. Thanks again to the team for the great work!
