The underlying problem which caused this bug is that query parameters, at least, are not being tagged with any particular encoding. Based on some reading, it’s my understanding that modern Rails versions do correctly indicate the encoding of parameters provided in request bodies (by looking at the Content-Type request header), and experimentation indicates that explicit query parameters (such as ?foo=bar) do get marked as being UTF-8, but as far as I can tell params which come from parsing the URL through route patterns, and format markers, are being marked as ASCII-8BIT. I’ve played a bit of whack-a-mole with the problems reported in the bug topic above, but given that we found three separate instances of potentially needing to force an encoding in one place, I’m thinking the issue is fairly likely to be extremely widespread throughout the codebase. So, the questions which come to mind are: Does this constitute a bug in Rails itself which should be fixed? How should this be worked around in Discourse, either until Rails itself gets fixed, or forever? How has this not been a problem before? It’s not like Discourse suddenly got its first non-seven-bit-ASCII users in the last week or two…

Looks like they rejected it: (I submitted it to Rails, and it was rejected because this is fixed in another way in Rails 5. But Discourse is still on Rails 4.)

Specifically here: ASCII comes in two flavors that need to be UTF-8ified. by begrif · Pull Request #30119 · rails/rails · GitHub (link goes to a particular comment). It could be that’s me just going about submitting fixes in the wrong way.

It looks like this is fixed in Rails 5 but it is unclear when we can move Discourse to that (we are currently on Rails 4). There are typically a lot of performance regressions in each new release of Rails that we have to help fix, so it takes some time for us to move.

Well, time for a custom before_filter, I guess.

Now that Discourse is on Rails 5, is it safe to reactivate the slug generation to encoded?

I guess so, based on this reply: [image] When slug generation method is encoded, It redirected many times when we open new tab opens in topic Bug @Alavi1412 Can you upgrade to latest? This is fixed after we upgraded to Rails 5.1 but is there any benefit to change the slug generation method? for example for SEO or …?

It’s for SEO and for human consumption. Our community heavily uses Japanese.

Yes, apologies for the delay but it should be 100% working now! Let us know if you run into any issues.

when the slug generation method is changed to “encoding”, the spaces between each two words in the topic titles changes to - and it makes the topic list page very unclean. [k] is it something related to rails or is it solvable by changes to the forum?

It’s a bug that’s been fixed a few hours ago. Update to latest.

Automatic encoding of parsed URL params

Development

tgxworld (Alan Tan) October 27, 2017, 2:27pm 41

Can you pass me a link to your site? I’ll have a look at the topic titles to see how we can fix it.

Topic		Replies	Views
Topic with Japanese in URL doesn't redirect if URL doesn't perfectly match Bug	71	5627	October 21, 2019
When slug generation method is encoded, It redirected many times when we open new tab opens in topic Bug	8	5517	September 29, 2017
Mods/admins to change topic URL slug (just like in WordPress) Feature	2	1402	December 2, 2014
Removing the concept of "slugs" for some languages Feature	24	10655	December 13, 2017
Force Lowercase slug URLs when set to "encoded" Product Support	22	4800	June 20, 2016

Automatic encoding of parsed URL params

Related topics