Search does not seem to work for tags with emoji

arnaud_sahuguet · February 13, 2020, 1:36am

I have tags that contain emojis, e.g. new-york-city-🇺🇸.
When using the search and advanced search, I don’t get the topics.

Here is an example:
Query:
https://urbantech-forum.cornelltech.io/search?expanded=true&q=tags%3Anew-york-city-🇺🇸subway

And here is a topic that should be returned:

codinghorror · February 13, 2020, 2:46am

Not sure if this is resolvable, what is your take @neil?

j.jaffeux · February 13, 2020, 9:40am

AFAIK it’s only because the regex we have here:

github.com/discourse/discourse

lib/search.rb

main


      
          end
          
          advanced_filter(/\Awith:images\z/i) { |posts| posts.where.not(posts: { image_upload_id: nil }) }
          
          advanced_filter(/\Acategor(?:y|ies):(.+)\z/i) do |posts, terms|
            category_ids = []
          
            matches =
              terms
                .split(",")
                .map do |term|
                  if term[0] == "="
                    [term[1..-1], true]
                  else
                    [term, false]
                  end
                end
                .to_h
          
            if matches.present?
              sql = <<~SQL

is not allowing emojis. If this was matching on emoji, it would work after.

We might want to consider using something like: GitHub - ticky/ruby-emoji-regex: 💎 A set of Ruby regular expressions for matching Unicode Emoji symbols. to have a good emoji regex.

what do you think @sam ?

sam · February 13, 2020, 10:22am

Oh my, this feels like one big monster to support for a major edge case, do we already have a validating regex on the Tag class?

j.jaffeux · February 13, 2020, 10:35am

I don’t think we do, might be wrong though @neil probably knows better
AFAIK we just have this clean_tag function:

github.com/discourse/discourse

lib/discourse_tagging.rb

main


      
          TAG_GROUP_RESTRICTIONS_SQL = <<~SQL
            tag_group_restrictions AS (
              SELECT t.id as tag_id, tgm.id as tgm_id, tg.id as tag_group_id, tg.parent_tag_id as parent_tag_id,
                tg.one_per_topic as one_per_topic
              FROM tags t
              LEFT OUTER JOIN tag_group_memberships tgm ON tgm.tag_id = t.id /*and_name_like*/
              LEFT OUTER JOIN tag_groups tg ON tg.id = tgm.tag_group_id
            )
          SQL
          
          CATEGORY_RESTRICTIONS_SQL = <<~SQL
            category_restrictions AS (
              SELECT t.id as tag_id, ct.id as ct_id, ct.category_id as category_id, NULL AS category_tag_group_id
              FROM tags t
              INNER JOIN category_tags ct ON t.id = ct.tag_id /*and_name_like*/
          
              UNION
          
              SELECT t.id as tag_id, ctg.id as ctg_id, ctg.category_id as category_id, ctg.tag_group_id AS category_tag_group_id
              FROM tags t
              INNER JOIN tag_group_memberships tgm ON tgm.tag_id = t.id /*and_name_like*/

Which is kinda copied in client side here:

https://github.com/discourse/discourse/blob/master/app/assets/javascripts/select-kit/mixins/tags.js.es6#L80

Also this would probably be useful at other places as we for example generate this client side: https://github.com/discourse/discourse/blob/master/app/assets/javascripts/pretty-text/emoji.js.es6#L24

neil · February 13, 2020, 4:25pm

I doubt that we have any tests for emoji tag names. Maybe we can look into supporting this in 2.5?

lucasbasquerotto · February 13, 2020, 5:56pm

I see that the search returns only one topic when using only the tag (without the subway keyword) in the search (it’s not the topic posted in the OP):

https://urbantech-forum.cornelltech.io/search?expanded=true&q=tags%3Anew-york-city-🇺🇸

It also works with other keywords (but only for that topic returned previously):

https://urbantech-forum.cornelltech.io/search?expanded=true&q=tags%3Anew-york-city-🇺🇸%20personal

If I don’t use tags it returns correctly:

https://urbantech-forum.cornelltech.io/search?expanded=true&q=new-york-city-🇺🇸

The topic returned previously, with the tag, is the 1st returned using a keyword with the tag name, but the post is another (but it might just be because of (maybe) a metadata or something like this in the post that is caught by the search, and not the tag itself, but I can’t say for sure).

Topic		Replies	Views
How to create a tag "🎥"? Support tags	5	86	November 21, 2024
Emoji in tag names Feature emoji , tags	20	7733	October 13, 2025
Emoji shortcodes aren't displayed when they're part of the search terms UX emoji , search	0	325	July 7, 2023
:crying_cat_face: Emoji's in title shown as code on initial load of a topic page Bug	4	944	December 22, 2017
Search by tag, not English characters Bug	3	736	August 27, 2018

Search does not seem to work for tags with emoji

Related topics