The underlying problem which caused this bug is that query parameters, at least, are not being tagged with any particular encoding. Based on some reading, it’s my understanding that modern Rails versions do correctly indicate the encoding of parameters provided in request bodies (by looking at the Content-Type request header), and experimentation indicates that explicit query parameters (such as ?foo=bar) do get marked as being UTF-8, but as far as I can tell params which come from parsing the URL through route patterns, and format markers, are being marked as ASCII-8BIT. I’ve played a bit of whack-a-mole with the problems reported in the bug topic above, but given that we found three separate instances of potentially needing to force an encoding in one place, I’m thinking the issue is fairly likely to be extremely widespread throughout the codebase. So, the questions which come to mind are: Does this constitute a bug in Rails itself which should be fixed? How should this be worked around in Discourse, either until Rails itself gets fixed, or forever? How has this not been a problem before? It’s not like Discourse suddenly got its first non-seven-bit-ASCII users in the last week or two…

Unfortunately this looks like an upstream bug in Ruby / Rails so it is not easy to fix.

We should at least escalate it to the appropriate Ruby/Rails tracker.

Looks like they rejected it: (I submitted it to Rails, and it was rejected because this is fixed in another way in Rails 5. But Discourse is still on Rails 4.)

Specifically here: ASCII comes in two flavors that need to be UTF-8ified. by begrif · Pull Request #30119 · rails/rails · GitHub (link goes to a particular comment). It could be that’s me just going about submitting fixes in the wrong way.

It looks like this is fixed in Rails 5 but it is unclear when we can move Discourse to that (we are currently on Rails 4). There are typically a lot of performance regressions in each new release of Rails that we have to help fix, so it takes some time for us to move.

Now that Discourse is on Rails 5, is it safe to reactivate the slug generation to encoded?

I guess so, based on this reply: [image] When slug generation method is encoded, It redirected many times when we open new tab opens in topic Bug @Alavi1412 Can you upgrade to latest? This is fixed after we upgraded to Rails 5.1 but is there any benefit to change the slug generation method? for example for SEO or …?

It’s for SEO and for human consumption. Our community heavily uses Japanese.

Yes, apologies for the delay but it should be 100% working now! Let us know if you run into any issues.

Automatic encoding of parsed URL params

Development

mpalmer (Matt Palmer) August 11, 2017, 1:37am 28

Well, time for a custom before_filter, I guess.

Topic		Replies	Views
Topic with Japanese in URL doesn't redirect if URL doesn't perfectly match Bug	71	5627	October 21, 2019
When slug generation method is encoded, It redirected many times when we open new tab opens in topic Bug	8	5517	September 29, 2017
Mods/admins to change topic URL slug (just like in WordPress) Feature	2	1402	December 2, 2014
Removing the concept of "slugs" for some languages Feature	24	10655	December 13, 2017
Force Lowercase slug URLs when set to "encoded" Product Support	22	4800	June 20, 2016

Automatic encoding of parsed URL params

Related topics