# Onebox breaks if there's chinese text in URL

**URL:** <https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364>\
**Category:** Bug\
**Created:** [2017年八月3日 07:48 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364 "2017-08-03T07:48:56Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![kiki](https://avatars.discourse-cdn.com/v4/letter/k/dc4da7/32.png) [@kiki](https://meta.discourse.org/u/kiki)\
**Post date:** [2017年八月3日 07:48 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/1 "2017-08-03T07:48:56Z")

</div>

discourse/lib/final\_destination.rb

```
  def initialize(url, opts = nil)
    @url = url
    Rails.logger.info("=============================1 #{@url}")
    @uri =
      begin
        URI(escape_url) if @url
      rescue URI::InvalidURIError
      end
    Rails.logger.info("=============================2 #{@uri.to_s}")

```

Output logs:

```
I, [2017-08-03T15:45:31.697669 #112637] INFO -- : =============================1 https://domain/%E6%B5%8B%E8%AF%95
I, [2017-08-03T15:45:31.757180 #112637] INFO -- : =============================2 https://domain/%25E6%25B5%258B%25E8%25AF%2595

```

---

<div class="post-metadata">

**Author:** ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)\
**Post date:** [2017年八月3日 07:51 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/2 "2017-08-03T07:51:38Z")

</div>

What exactly breaks? Do you have an exception and backtrace?

---

<div class="post-metadata">

**Author:** ![kiki](https://avatars.discourse-cdn.com/v4/letter/k/dc4da7/32.png) [@kiki](https://meta.discourse.org/u/kiki)\
**Post date:** [2017年八月3日 07:52 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/3 "2017-08-03T07:52:59Z")

</div>

the input url is `https://domain/%E6%B5%8B%E8%AF%95`

the url i got in onebox turns to be: `https://domain/%25E6%25B5%258B%25E8%25AF%2595`

I’ve added logs in discourse/lib/final\_destination.rb (as shown in the above post) and find it escape to the wrong url.

they are different which expected the same.

---

<div class="post-metadata">

**Author:** ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)\
**Post date:** [2017年八月3日 07:57 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/4 "2017-08-03T07:57:34Z")

</div>

Well, yes, of _course_ they’re going to be different; if you’re escaping a URL, any `%` characters are always going to be converted to `%25`, that’s what URI escaping _does_.

---

<div class="post-metadata">

**Author:** ![kiki](https://avatars.discourse-cdn.com/v4/letter/k/dc4da7/32.png) [@kiki](https://meta.discourse.org/u/kiki)\
**Post date:** [2017年八月3日 08:19 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/5 "2017-08-03T08:19:02Z")

</div>

It’s no escaped by me. It’s escaped by discourse/lib/final\_destination.rb.

It did not work this way before.

The scenario is:

1. i wrote a onebox plugin, to handle url like `https://domain/%E6%B5%8B%E8%AF%95`

2. user input url : `https://domain/%E6%B5%8B%E8%AF%95`  

3. and the onbebox got the `@url`=`https://domain/%E6%B5%8B%E8%AF%95` and do the work  

in the old days, the @url is the exact the user input url `https://domain/%E6%B5%8B%E8%AF%95`.

but now it turns to be `https://domain/%25E6%25B5%258B%25E8%25AF%2595`

---

<div class="post-metadata">

**Author:** ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)\
**Post date:** [2017年八月3日 08:23 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/6 "2017-08-03T08:23:11Z")

</div>

The change was made in [FIX: Escape URL before attempting to resolve it. · discourse/discourse@b534778 · GitHub](https://github.com/discourse/discourse/commit/b534778f46ac310d9b59afa6f5390fced267f2f0). There’s no bug reference on the commit. @tgxworld, do you recall the purpose of that commit?

---

<div class="post-metadata">

**Author:** ![Stranik](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/stranik/32/85638_2.png) [@Stranik](https://meta.discourse.org/u/Stranik)\
**Post date:** [2017年八月3日 08:24 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/7 "2017-08-03T08:24:52Z")

</div>

I noticed this with other languages.  
Or is this another scenario?

`https://en.wikipedia.org/wiki/Free_software`

> **[Free software](https://en.wikipedia.org/wiki/Free_software)**
>
> Free software, libre software, libreware sometimes known as freedom-respecting software is computer software distributed under terms that allow users to run the software for any purpose as well as to study, change, and distribute it and any adapted versions. Free software is a matter of liberty, not price; all users are legally free to do what they want with their copies of free software (including profiting from them) regardless of how much is paid to obtain the program.\[ The right to study and ...

**Russian version**

`https://ru.wikipedia.org/wiki/%D0%A1%D0%B2%D0%BE%D0%B1%D0%BE%D0%B4%D0%BD%D0%BE%D0%B5_%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%BD%D0%BE%D0%B5_%D0%BE%D0%B1%D0%B5%D1%81%D0%BF%D0%B5%D1%87%D0%B5%D0%BD%D0%B8%D0%B5`

> **[Свободное программное обеспечение](https://ru.wikipedia.org/wiki/%D0%A1%D0%B2%D0%BE%D0%B1%D0%BE%D0%B4%D0%BD%D0%BE%D0%B5_%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%BD%D0%BE%D0%B5_%D0%BE%D0%B1%D0%B5%D1%81%D0%BF%D0%B5%D1%87%D0%B5%D0%BD%D0%B8%D0%B5)**
>
> Свободное программное обеспе́чение (СПО, англ. free software, также software libre или libre software), свободный софт или свободные программы — программное обеспечение, пользователи которого имеют права («свободы») на его неограниченную установку, запуск, свободное использование, изучение, распространение и изменение (совершенствование), а также распространение копий и результатов изменения. Если на программное обеспечение есть исключительные права, то свободы объявляются при помощи свобод Как и...

---

<div class="post-metadata">

**Author:** ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)\
**Post date:** [2017年八月3日 08:29 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/8 "2017-08-03T08:29:57Z")

</div>

There’s nothing language-specific here. Any URL data that can’t be represented in the constrained set of characters permitted in URLs gets percent-escaped. In both cases here, the data is UTF-8 coded character data, that’s possibly undergoing a double round of encoding. Unfortunately, because no backtrace has been provided, it’s impossible to see where the data’s coming from, merely that _something_ is going on.

---

<div class="post-metadata">

**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\
**Post date:** [2017年八月11日 17:16 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/9 "2017-08-11T17:16:11Z")

</div>

@tgxworld looks like there is a regression here:

> **[Свободное программное обеспечение](https://ru.wikipedia.org/wiki/%D0%A1%D0%B2%D0%BE%D0%B1%D0%BE%D0%B4%D0%BD%D0%BE%D0%B5_%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%BD%D0%BE%D0%B5_%D0%BE%D0%B1%D0%B5%D1%81%D0%BF%D0%B5%D1%87%D0%B5%D0%BD%D0%B8%D0%B5)**
>
> Свободное программное обеспе́чение (СПО, англ. free software, также software libre или libre software), свободный софт или свободные программы — программное обеспечение, пользователи которого имеют права («свободы») на его неограниченную установку, запуск, свободное использование, изучение, распространение и изменение (совершенствование), а также распространение копий и результатов изменения. Если на программное обеспечение есть исключительные права, то свободы объявляются при помощи свобод Как и...

Used to work as a Onebox in 1.8 and now no longer works.

---

<div class="post-metadata">

**Author:** ![tgxworld](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tgxworld/32/106117_2.png) [@tgxworld](https://meta.discourse.org/u/tgxworld)\
**Post date:** [2017年八月22日 08:06 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/11 "2017-08-22T08:06:27Z")

</div>

> [@mpalmer](#):
>
> There’s no bug reference on the commit. @tgxworld, do you recall the purpose of that commit?

Oops not sure how I missed the reply but I was fixing an error flooding our logs.

```plaintext
[1] pry(main)> URI("https://eviltrout.com?s=180&#038;d=mm&#038;r=g")
URI::InvalidURIError: bad URI(is not URI?): https://eviltrout.com?s=180&#038;d=mm&#038;r=g
from /home/tgxworld/.rbenv/versions/2.4.1/lib/ruby/2.4.0/uri/rfc3986_parser.rb:67:in `split'

```

---

<div class="post-metadata">

**Author:** ![tgxworld](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tgxworld/32/106117_2.png) [@tgxworld](https://meta.discourse.org/u/tgxworld)\
**Post date:** [2017年八月22日 08:15 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/12 "2017-08-22T08:15:30Z")

</div>

Hmm I’m not sure if it actually worked in 1.8 because `FinalDestination` was introduced before the 1.8 release and the URL wouldn’t have been resolved at al.

```plaintext
[2] pry(main)> URI("https://ru.wikipedia.org/wiki/Свободное_программное_обеспечение2")
URI::InvalidURIError: URI must be ascii only "https://ru.wikipedia.org/wiki/\u0421\u0432\u043E\u0431\u043E\u0434\u043D\u043E\u0435_\u043F\u0440\u043E\u0433\u0440\u0430\u043C\u043C\u043D\u043E\u0435_\u043E\u0431\u0435\u0441\u043F\u0435\u0447\u0435\u043D\u0438\u04352"
from /home/tgxworld/.rbenv/versions/2.4.1/lib/ruby/2.4.0/uri/rfc3986_parser.rb:21:in `split'

```

---

<div class="post-metadata">

**Author:** ![tgxworld](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tgxworld/32/106117_2.png) [@tgxworld](https://meta.discourse.org/u/tgxworld)\
**Post date:** [2017年八月22日 08:34 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/13 "2017-08-22T08:34:27Z")

</div>

Hmm OK I see what is happening here. `FinalDestination` is given

`https://ru.wikipedia.org/wiki/%D0%A1%D0%B2%D0%BE%D0%B1%D0%BE%D0%B4%D0%BD%D0%BE%D0%B5_%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%BD%D0%BE%D0%B5_%D0%BE%D0%B1%D0%B5%D1%81%D0%BF%D0%B5%D1%87%D0%B5%D0%BD%D0%B8%D0%B5`

instead of

`https://ru.wikipedia.org/wiki/Свободное_программное_обеспечение`

For the first case we end up escaping the `%`sign.. Hmm will need to figure out the format given to `FinalDestination` because the `url` passed to it is sometimes not escaped.

---

<div class="post-metadata">

**Author:** ![tgxworld](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tgxworld/32/106117_2.png) [@tgxworld](https://meta.discourse.org/u/tgxworld)\
**Post date:** [2017年九月26日 10:36 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/15 "2017-09-26T10:36:44Z")

</div>

Fixed in

> <https://github.com/discourse/discourse/commit/367fb1c524cff06a33c7a4144cd13a270a9f3489>
>
> https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364

---

<div class="post-metadata">

**Author:** ![tgxworld](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tgxworld/32/106117_2.png) [@tgxworld](https://meta.discourse.org/u/tgxworld)\
**Post date:** [2017年九月27日 02:58 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/16 "2017-09-27T02:58:19Z")

</div>

Yay encoded URLs are being properly oneboxed again.

> **[Свободное программное обеспечение](https://ru.wikipedia.org/wiki/%D0%A1%D0%B2%D0%BE%D0%B1%D0%BE%D0%B4%D0%BD%D0%BE%D0%B5_%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%BD%D0%BE%D0%B5_%D0%BE%D0%B1%D0%B5%D1%81%D0%BF%D0%B5%D1%87%D0%B5%D0%BD%D0%B8%D0%B5)**
>
> Свободное программное обеспе́чение (СПО, англ. free software, также software libre или libre software), свободный софт или свободные программы — программное обеспечение, пользователи которого имеют права («свободы») на его неограниченную установку, запуск, свободное использование, изучение, распространение и изменение (совершенствование), а также распространение копий и результатов изменения. Если на программное обеспечение есть исключительные права, то свободы объявляются при помощи свобод Как и...

> **[Свободное программное обеспечение](https://ru.wikipedia.org/wiki/%D0%A1%D0%B2%D0%BE%D0%B1%D0%BE%D0%B4%D0%BD%D0%BE%D0%B5_%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%BD%D0%BE%D0%B5_%D0%BE%D0%B1%D0%B5%D1%81%D0%BF%D0%B5%D1%87%D0%B5%D0%BD%D0%B8%D0%B5)**
>
> Свободное программное обеспе́чение (СПО, англ. free software, также software libre или libre software), свободный софт или свободные программы — программное обеспечение, пользователи которого имеют права («свободы») на его неограниченную установку, запуск, свободное использование, изучение, распространение и изменение (совершенствование), а также распространение копий и результатов изменения. Если на программное обеспечение есть исключительные права, то свободы объявляются при помощи свобод Как и...

---

<div class="post-metadata">

**Author:** ![tgxworld](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tgxworld/32/106117_2.png) [@tgxworld](https://meta.discourse.org/u/tgxworld)\
**Post date:** [2017年九月27日 02:58 UTC](https://meta.discourse.org/t/onebox-breaks-if-theres-chinese-text-in-url/67364/17 "2017-09-27T02:58:29Z")

</div>