Search does not seem to work for tags with emoji

arnaud_sahuguet · 13 februari 2020 om 01:36

I have tags that contain emojis, e.g. new-york-city-🇺🇸.
When using the search and advanced search, I don’t get the topics.

Here is an example:
Query:
https://urbantech-forum.cornelltech.io/search?expanded=true&q=tags%3Anew-york-city-🇺🇸subway

And here is a topic that should be returned:
https://urbantech-forum.cornelltech.io/t/subway-accessibility-info-and-elevator-escalator-status-alerts/139

codinghorror · 13 februari 2020 om 02:46

Not sure if this is resolvable, what is your take @neil?

j.jaffeux · 13 februari 2020 om 09:40

AFAIK it’s only because the regex we have here:

github.com/discourse/discourse

lib/search.rb

main


      
          end
          
          advanced_filter(/\Ain:unseen\z/i) do |posts|
            if @guardian.user
              posts.joins(
                "LEFT JOIN post_timings ON
                  post_timings.topic_id = posts.topic_id
                  AND post_timings.post_number = posts.post_number
                  AND post_timings.user_id = #{ActiveRecord::Base.connection.quote(@guardian.user.id)}
                ",
              ).where("post_timings.user_id IS NULL")
            end
          end
          
          advanced_filter(/\Awith:images\z/i) { |posts| posts.where.not(posts: { image_upload_id: nil }) }
          
          advanced_filter(/\Acategor(?:y|ies):(.+)\z/i) do |posts, terms|
            category_ids = []
          
            matches =
              terms

is not allowing emojis. If this was matching on emoji, it would work after.

We might want to consider using something like: GitHub - ticky/ruby-emoji-regex: 💎 A set of Ruby regular expressions for matching Unicode Emoji symbols. · GitHub to have a good emoji regex.

what do you think @sam ?

sam · 13 februari 2020 om 10:22

Oh my, this feels like one big monster to support for a major edge case, do we already have a validating regex on the Tag class?

j.jaffeux · 13 februari 2020 om 10:35

I don’t think we do, might be wrong though @neil probably knows better
AFAIK we just have this clean_tag function:

github.com/discourse/discourse

lib/discourse_tagging.rb

main


      
                return true if guardian.is_admin?
          
                msg =
                  I18n.t(
                    "tags.forbidden.category_does_not_allow_tags",
                    count: unrestricted_tags.size,
                    tags: unrestricted_tags.sort.join(", "),
                    category: category.name,
                  )
                model.errors.add(:base, msg)
                return false
              end
            end
          
            true
          end
          
          def self.validate_one_tag_from_group_per_topic(guardian, model, category, tags = [])
            tags_cant_be_used = filter_tags_violating_one_tag_from_group_per_topic(guardian, category, tags)
          
            return true if tags_cant_be_used.blank?

Which is kinda copied in client side here:

https://github.com/discourse/discourse/blob/master/app/assets/javascripts/select-kit/mixins/tags.js.es6#L80

Also this would probably be useful at other places as we for example generate this client side: https://github.com/discourse/discourse/blob/master/app/assets/javascripts/pretty-text/emoji.js.es6#L24

neil · 13 februari 2020 om 16:25

I doubt that we have any tests for emoji tag names. Maybe we can look into supporting this in 2.5?

lucasbasquerotto · 13 februari 2020 om 17:56

I see that the search returns only one topic when using only the tag (without the subway keyword) in the search (it’s not the topic posted in the OP):

https://urbantech-forum.cornelltech.io/search?expanded=true&q=tags%3Anew-york-city-🇺🇸

It also works with other keywords (but only for that topic returned previously):

https://urbantech-forum.cornelltech.io/search?expanded=true&q=tags%3Anew-york-city-🇺🇸%20personal

If I don’t use tags it returns correctly:

https://urbantech-forum.cornelltech.io/search?expanded=true&q=new-york-city-🇺🇸

The topic returned previously, with the tag, is the 1st returned using a keyword with the tag name, but the post is another (but it might just be because of (maybe) a metadata or something like this in the post that is caught by the search, and not the tag itself, but I can’t say for sure).

Topic		Antwoorden	Weergaven
How to create a tag "🎥"? Support tags	5	229	21 november 2024
Emoji in tag names Feature emoji , tags	20	8124	13 oktober 2025
Emoji shortcodes aren't displayed when they're part of the search terms UX emoji , search	0	361	7 juli 2023
:crying_cat_face: Emoji's in title shown as code on initial load of a topic page Bug	3	999	21 december 2017
Search by tag, not English characters Bug	2	799	27 augustus 2018

Search does not seem to work for tags with emoji

Gerelateerde topics