# Tags does not work with Cyrillic

**URL:** https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217
**Category:** Bug
**Created:** [August 1, 2017, 5:29am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217 "2017-08-01T05:29:41Z")
**Posts on this page:** 11
**Page:** 1

<div class="post-metadata">

### Author: ![Stranik](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/stranik/32/85638_2.png) [@Stranik](https://meta.discourse.org/u/Stranik)
#### Post date: [August 1, 2017, 5:29am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217/1 "2017-08-01T05:29:41Z")

</div>

http://\*\*\*.ru/tags/земля  
500 Internal Server Error

 ![1](https://global.discourse-cdn.com/meta/original/3X/d/8/d8c91c7c0c879f6923119bb6ad404709a056dfd4.jpg)

`Encoding::UndefinedConversionError ("\xD1" from ASCII-8BIT to UTF-8) /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.9/lib/active_support/core_ext/object/json.rb:34:in `encode’`

With safe\_mode, similarly does not work

---

<div class="post-metadata">

### Author: ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [August 1, 2017, 6:29am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217/2 "2017-08-01T06:29:10Z")

</div>

Ooooooh, this is going to get messy.

In theory, URLs can only contain 7-bit ASCII characters. In practice, everyone’s pretty much decided that percent-encoded UTF-8 is 👌, which is why, if you look in the “env” for the log message, you’ll see something like

```
REQUEST_URI /tags/%D0%B7%D0%B5%D0%BC%D0%BB%D1%8F/notifications

```

(Which is the UTF-8 encoding for “земля”)

So, the problem is occuring because the contents of `params` is being encoded as `ASCII-8BIT`, and while _most_ everything manages to figure out what’s going on and roll with it, the JSON encoding of the result, containing as it does an ASCII-8BIT string whose individual bytes don’t translate into valid UTF-8 codepoints, explodes.

Luckily, the fix is relatively straightforward:

[https://github.com/discourse/discourse/commit/7ee861f4571f6e7259e631d0404e0d958501dcf0](https://github.com/discourse/discourse/commit/7ee861f4571f6e7259e631d0404e0d958501dcf0)

---

<div class="post-metadata">

### Author: ![Stranik](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/stranik/32/85638_2.png) [@Stranik](https://meta.discourse.org/u/Stranik)
#### Post date: [August 1, 2017, 6:37am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217/3 "2017-08-01T06:37:17Z")

</div>

Thank you, greatly appreciated.

---

<div class="post-metadata">

### Author: ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [August 1, 2017, 6:58am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217/4 "2017-08-01T06:58:44Z")

</div>



---

<div class="post-metadata">

### Author: ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [August 1, 2017, 7:10am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217/5 "2017-08-01T07:10:24Z")

</div>



---

<div class="post-metadata">

### Author: ![Stranik](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/stranik/32/85638_2.png) [@Stranik](https://meta.discourse.org/u/Stranik)
#### Post date: [August 1, 2017, 7:13am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217/6 "2017-08-01T07:13:51Z")

</div>

Apparently, something else is wrong. The form of the error has changed.

 ![2](https://global.discourse-cdn.com/meta/original/3X/b/b/bb1fdbbd157a1ee7eb2a7d6ad5164fc89034acb6.jpg)

---

<div class="post-metadata">

### Author: ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [August 1, 2017, 7:41am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217/7 "2017-08-01T07:41:09Z")

</div>

How did you cause that to happen? I can’t make it happen on a local instance with that tag created. The JSON always renders for me.

---

<div class="post-metadata">

### Author: ![Stranik](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/stranik/32/85638_2.png) [@Stranik](https://meta.discourse.org/u/Stranik)
#### Post date: [August 1, 2017, 7:45am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217/8 "2017-08-01T07:45:01Z")

</div>

I’m just trying to click on any tag: [http://toxu.ru](http://toxu.ru)

`http://toxu.ru/tags/земля`  
`http://toxu.ru/tags/faq` - works

`http://toxu.ru/tags/земля/l/latest.json?order=default&ascending=false&filter=tags/земля/l/latest`

`http://toxu.ru/tags/faq/l/latest.json?order=default&ascending=false&filter=tags/faq/l/latest` - works

---

<div class="post-metadata">

### Author: ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [August 1, 2017, 8:06am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217/9 "2017-08-01T08:06:47Z")

</div>

It must be the additional complexity of the data you’ve got that’s causing the bug to appear, whilst my trivial example DB doesn’t trip it up.

Since I can’t reproduce it myself, I’ve had to take a bit of a speculative bug fix attempt in [Hunt-and-kill a few more mis-encoded params · discourse/discourse@67882ec · GitHub](https://github.com/discourse/discourse/commit/67882ec37da6dac2ec0ce69e110014a6fe11882c); please let me know if it does/doesn’t work for you. I think we’re going to have to do a larger and more comprehensive fix for this, along the lines of [this initializer](https://jasoncodes.com/posts/ruby19-rails2-encodings); playing whack-a-mole with parameter encodings one-by-one seems like a good way to go prematurely bald.

---

<div class="post-metadata">

### Author: ![Stranik](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/stranik/32/85638_2.png) [@Stranik](https://meta.discourse.org/u/Stranik)
#### Post date: [August 1, 2017, 8:44am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217/10 "2017-08-01T08:44:39Z")

</div>

Now it works! Thank you!

---

<div class="post-metadata">

### Author: ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [August 1, 2017, 9:26am UTC](https://meta.discourse.org/t/tags-does-not-work-with-cyrillic/67217/11 "2017-08-01T09:26:02Z")

</div>

Generic discussion of parsed parameter encoding is [over here](https://meta.discourse.org/t/automatic-encoding-of-parsed-url-params/67232).
