# What is the problem with PrettyText.markdown call?

**URL:** https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674
**Category:** Development
**Created:** [4월 5, 2018, 9:11오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674 "2018-04-05T21:11:49Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![ballistic](https://avatars.discourse-cdn.com/v4/letter/b/dfb087/32.png) [@ballistic](https://meta.discourse.org/u/ballistic)
#### Post date: [4월 5, 2018, 9:11오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/1 "2018-04-05T21:11:49Z")

</div>

```
PrettyText.markdown("❤️❤️❤️", {}) 

```

Will not generate the same code as:

```
PrettyText.markdown(":heart::heart::heart:", {})

```

It generates:

```
=> "<p><img src=\"/images/emoji/apple/heart.png?v=5\" title=\":heart:\" class=\"emoji\" alt=\":heart:\">️:heart:️:heart:️</p>" 
( there are ':️' & ':', copy paste to https://www.soscisurvey.de/tools/view-chars.php to see it)

```

I think it has something to do with

```
replacement = "\u200b" + replacement;

```

In lib/pretty\_text/shims.js

( on 1.9.4 )

---

<div class="post-metadata">

### Author: ![supermathie](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/supermathie/32/507518_2.png) [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [4월 5, 2018, 9:25오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/2 "2018-04-05T21:25:08Z")

</div>

The control code insertion isn’t happening on latest:

```plaintext
[1] pry(main)> puts PrettyText.markdown("❤️", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:">️</p>
[2] pry(main)> puts PrettyText.markdown(":heart:", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"></p>

```

although I do note it behaves differently:

```plaintext
[1] pry(main)> puts PrettyText.markdown("❤️❤️❤️", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:">️:heart:️:heart:️</p>
[2] pry(main)> puts PrettyText.markdown(":heart::heart::heart:", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"></p>

```

```plaintext
❤️❤️❤️

:heart::heart::heart:

```

generates:

❤❤❤

❤❤❤

---

<div class="post-metadata">

### Author: ![riking](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/riking/32/170938_2.png) [@riking](https://meta.discourse.org/u/riking)
#### Post date: [4월 5, 2018, 9:30오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/3 "2018-04-05T21:30:14Z")

</div>

FFEF is not an assigned Unicode character, so I wonder what’s putting it in. (note: it’s not the BOM, that’s FEFF.)

---

<div class="post-metadata">

### Author: ![Mittineague](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mittineague/32/114259_2.png) [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [4월 5, 2018, 9:32오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/4 "2018-04-05T21:32:22Z")

</div>

I think it’s locale character encoding involved.

I don’t know what it means, but `\u+ffb8` is “Halfwidth Hangul Letter Cieuc”

> **[Find all Unicode Characters from Hieroglyphs to Dingbats – Unicode Compart](https://www.compart.com/en/unicode/U+FFB8)**
>
> U+FFB8 is the unicode hex value of the character Halfwidth Hangul Letter Cieuc. Char U+FFB8, Encodings, HTML Entitys:ﾸ,ﾸ, UTF-8 (hex), UTF-16 (hex), UTF-32 (hex)

---

<div class="post-metadata">

### Author: ![Mittineague](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mittineague/32/114259_2.png) [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [4월 6, 2018, 6:00오전 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/7 "2018-04-06T06:00:50Z")

</div>

> [@riking](#):
>
> it’s not the BOM

That was my initial hunch, a big endian - little endian paste thing. (been there, done that)

If the leading colon isn’t the “beginning of a ‘word’” Discourse adds a “hair space” - the `\u200b`.

> <https://github.com/discourse/discourse/blob/main/lib/pretty_text/shims.js#L12-L25>

Looking at bytes, if it is endian related, I don’t see how.

```plaintext
\u200b
01011100 01110101 00110010 00110000 00110000 01100010

\uffef
01011100 01110101 01100110 01100110 01100101 01100110

\uffb8
01011100 01110101 01100110 01100110 01100010 00111000

\uffe2
01011100 01110101 01100110 01100110 01100101 00110010

```

So I think it must be the regex’s interpretation of what it considers to be a “word”, i.e. locale related.

---

<div class="post-metadata">

### Author: ![ballistic](https://avatars.discourse-cdn.com/v4/letter/b/dfb087/32.png) [@ballistic](https://meta.discourse.org/u/ballistic)
#### Post date: [4월 6, 2018, 3:57오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/8 "2018-04-06T15:57:01Z")

</div>

> [@supermathie](#):
>
> ❤❤

This is what I see on the console:

```
$ echo ":heart:️:heart:️" | hexdump
0000000 3a 68 65 61 72 74 3a ef b8 8f 3a 68 65 61 72 74
0000010 3a ef b8 8f 0a                                 
0000015

```

well, if I type it, it will be:  
`$ echo ":heart::heart:" | hexdump \r\n 0000000 3a 68 65 61 72 74 3a 3a 68 65 61 72 74 3a 0a \r\n 000000f`

( I have problem to format this post )

The first one with invisible characters is 5 char longer and it disabled the following emoji to url escape.

Maybe OS’s clipboard or terminal changed something, but the problem we want to fix is why unicode hearts cannot be escaped to urls.

---

<div class="post-metadata">

### Author: ![ballistic](https://avatars.discourse-cdn.com/v4/letter/b/dfb087/32.png) [@ballistic](https://meta.discourse.org/u/ballistic)
#### Post date: [4월 6, 2018, 4:02오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/9 "2018-04-06T16:02:05Z")

</div>

What is the best way to debug it? console.log doesn’t work on the server side js.

---

<div class="post-metadata">

### Author: ![Mittineague](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mittineague/32/114259_2.png) [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [4월 6, 2018, 7:42오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/10 "2018-04-06T19:42:55Z")

</div>

Please satisfy my curiosity and let me know what the locale is.

I’m guessing it’s something where the typical western concept of what constitutes a “word” doesn’t apply. But if I’m completely off-base it’s the wrong rabbit hole to go down into.

---

<div class="post-metadata">

### Author: ![ballistic](https://avatars.discourse-cdn.com/v4/letter/b/dfb087/32.png) [@ballistic](https://meta.discourse.org/u/ballistic)
#### Post date: [4월 6, 2018, 8:10오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/11 "2018-04-06T20:10:34Z")

</div>

it is from [https://docs.bitnami.com/virtual-machine/apps/discourse/](https://docs.bitnami.com/virtual-machine/apps/discourse/)

So en\_US.

```
$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

```

---

<div class="post-metadata">

### Author: ![Mittineague](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mittineague/32/114259_2.png) [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [4월 6, 2018, 8:28오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/12 "2018-04-06T20:28:04Z")

</div>

Thanks. “Bitnami install” is a completely different rabbit hole 🐇

Not that the cause is because of the Bitnami install, but I have seen many posts here dealing with problems it had. So I’m leaning towards thinking it is more a Bitnami thing rather than Discourse itself. Any way to reach out to others with Bitnami installs to see if they have the same issue?

---

<div class="post-metadata">

### Author: ![ballistic](https://avatars.discourse-cdn.com/v4/letter/b/dfb087/32.png) [@ballistic](https://meta.discourse.org/u/ballistic)
#### Post date: [4월 6, 2018, 9:57오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/13 "2018-04-06T21:57:31Z")

</div>

@Mittineague Can you reproduce the problem?

Because @supermathie did reproduce it. I feel he is not on bitnami.

---

<div class="post-metadata">

### Author: ![Mittineague](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mittineague/32/114259_2.png) [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [4월 6, 2018, 10:29오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/14 "2018-04-06T22:29:39Z")

</div>

> [@ballistic](#):
>
> Can you reproduce the problem?

Yes, 1-2 are supermathie’s results. 5,6,7,8 are mine

```plaintext
[1] pry(main)> puts PrettyText.markdown("❤️", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:">️</p>
[2] pry(main)> puts PrettyText.markdown(":heart:", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"></p>

[5] pry(main)> puts PrettyText.markdown("❤️", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:">️</p>
[6] pry(main)> puts PrettyText.markdown(":heart:", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"></p>

[1] pry(main)> puts PrettyText.markdown("❤️❤️❤️", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:">️:heart:️:heart:️</p>
[2] pry(main)> puts PrettyText.markdown(":heart::heart::heart:", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"></p>

[7] pry(main)> puts PrettyText.markdown("❤️❤️❤️", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:">️:heart:️:heart:️</p>
[8] pry(main)> puts PrettyText.markdown(":heart::heart::heart:", {});
<p><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"><img src="/images/emoji/twitter/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"></p>

```

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [4월 6, 2018, 10:32오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/15 "2018-04-06T22:32:12Z")

</div>

@j.jaffeux is there some internal bug in the emoji remapper?

---

<div class="post-metadata">

### Author: ![riking](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/riking/32/170938_2.png) [@riking](https://meta.discourse.org/u/riking)
#### Post date: [4월 7, 2018, 12:48오전 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/16 "2018-04-07T00:48:58Z")

</div>

> [@Mittineague](#):
>
> `:️:`

Uhhh what’s with the two different colons here?

![30%20PM](https://global.discourse-cdn.com/meta/original/3X/e/2/e29e31bcc6e08dfb896f30a8007d76a17891c0e9.png)

```plaintext
% echo -n :<fe0f>: | xxd
00000000: 3aef b88f 3a :...:

```

There’s a U+FEOF VARIATION SELECTOR 16, also known as “force emoji display” that’s not being picked up by the remapper.

---

<div class="post-metadata">

### Author: ![Mittineague](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mittineague/32/114259_2.png) [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [4월 7, 2018, 2:45오전 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/17 "2018-04-07T02:45:39Z")

</div>

> [@riking](#):
>
> Uhhh what’s with the two different colons here?

Your acuity is much better than mine. I still can’t discern a difference even when I look for it.

As to where they came from, I simply copied supermathie’s example puts into my console to see if I could reproduce the results.

---

<div class="post-metadata">

### Author: ![ballistic](https://avatars.discourse-cdn.com/v4/letter/b/dfb087/32.png) [@ballistic](https://meta.discourse.org/u/ballistic)
#### Post date: [4월 13, 2018, 5:41오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/18 "2018-04-13T17:41:18Z")

</div>

Should we move this to the bug category? Or I should create a new topic in the bug category?

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [4월 14, 2018, 1:58오전 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/19 "2018-04-14T01:58:47Z")

</div>

There is probably a minor issue with the emoji → image convertor but I am struggling here with the real world impact of this.

Can you provide an example in a post of where this is an actual issue short of hexedit showing something off?

---

<div class="post-metadata">

### Author: ![ballistic](https://avatars.discourse-cdn.com/v4/letter/b/dfb087/32.png) [@ballistic](https://meta.discourse.org/u/ballistic)
#### Post date: [4월 14, 2018, 4:24오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/20 "2018-04-14T16:24:45Z")

</div>

This is from an iPad. But I think it applies to many mobile users.

If I just ❤ Discourse, it is ok.

But if we ❤❤❤❤❤❤❤❤❤❤❤❤ Discourse it is a problem, is it?

---

<div class="post-metadata">

### Author: ![ballistic](https://avatars.discourse-cdn.com/v4/letter/b/dfb087/32.png) [@ballistic](https://meta.discourse.org/u/ballistic)
#### Post date: [4월 14, 2018, 4:27오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/21 "2018-04-14T16:27:40Z")

</div>

![image](https://global.discourse-cdn.com/meta/original/3X/3/e/3e7e0950fd3bb194fdaf5ec0bd81e8a276a89521.png)

Mobile view, this is another problem.

---

<div class="post-metadata">

### Author: ![ballistic](https://avatars.discourse-cdn.com/v4/letter/b/dfb087/32.png) [@ballistic](https://meta.discourse.org/u/ballistic)
#### Post date: [4월 14, 2018, 4:30오후 UTC](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674/22 "2018-04-14T16:30:09Z")

</div>

The real world problem is when people are excited, they use a lot of emojis, and this issue breaks their hearts.

[Next page](https://meta.discourse.org/t/what-is-the-problem-with-prettytext-markdown-call/84674.md?page=2)
