Work on better emoji support


(Sam Saffron) #21

There is a technical term for this, it is called “imagineering” :blush:

If the real problem you have here is “I don’t like that Emoji added skin tones and I want to be yellow forever, because, reasons”. Well it is your forum and your choice, but I think you are going to have to develop a plugin for it cause its going to be lots of work for us to make this a switch. Work I do not want to invest.


(Rafael dos Santos Silva) #22

It’s based on the:

Hide the skin tone selector with CSS, if that is what you want.


(Tobias Eigen) #23

From the wikipedia talk page about the Fitzpatrick scale:

I find it hard to believe that the Unicode Consortium (apparently on the request of Google and/or Apple) would resort to what is basically institutionalised race baiting. They had the choice to (a) leave it alone, and up to the font creators or (b) decide that no, they should make it the responsability of everyone using emojis to constantly worry about race and skin color. They chose option (b).


(Sam Saffron) #24

It is more going to be like, add a theme component that disables autocomplete when you type t after :man:t … CSS ain’t gonna do the trick.

But what do we do about someone using keyboard on iOS to enter a brown David Bowie :man_singer:t5:

Are we meant to yellowify him? Making this optional is just swimming upstream.


(Rafael dos Santos Silva) #25

@tobiaseigen said:

Hiding it from the UI will give it less prominence, but you can’t completely disable it, like you said it’s swimming upstream. Don’t want skin tone emoji anywhere, you will need to disable emoji.


(Robin Ward) #26

There are many other modifiers that we are going to allow too, like same sex relationships. This is not the place to debate whether they should be part of emoji or not. We’re on board with supporting it all and as it’s been pointed out by other members of the team, we will not be adding the ability to disable it.


(Tobias Eigen) #27

Cool - got it. Appreciate the explanation and that we’re talking about coming into line with an international standard. Go forth. :rocket:


(Joffrey Jaffeux) #28

Making great progress, my work is focused on emoji.rake

Todo:

  • ability to force image for one/more platform on an emoji
  • a way to run a quick integration test from the rake task
  • probably some edge cases to handle

(Joffrey Jaffeux) #29

Ok the script is almost fully functional, few highlights:

➜  Projects rake -f emoji.rake emoji:clean && rake -f emoji.rake emoji:update
Cleaning emoji folder...
Downloading remote emoji list...
Downloading remote emoji images list...
Parsing remote emoji images list...
............................................. [omitted]
[!] Found existing image `sleeping_bed` for platform: apple, might want to remove the patch.
[!] Found existing image `snowboarder` for platform: apple, might want to remove the patch.

Writing emojis to disk...
.............................................. [omitted]
[!] Skipping zero :zero:, undefined platforms: windows

The script will warn you if an emoji hasn’t been created due to images missing on one or more platforms, it will also warn you if you have patched an image of one platform and this platform now has an image available.

A patch is as easy as tweaking a constant, also it uses readable names instead of code points for easier maintenance:

EMOJI_IMAGES_CELLS_INDEX_PATCH = {
  :windows => {
    "zero" => APPLE_CELL_INDEX # this cell index is the index of the col in http://unicode.org/emoji/charts/full-emoji-list.html
  }
}

note: it’s also possible to patch skin toned emojis. (like snowboarder/2/3/4/5/6 for apple)

At the moment the source of truth for the emoji list is https://raw.githubusercontent.com/muan/emojilib/master/emojis.json, some data is outdated, you can perform a patch on the output using a simple constant tweak:

EMOJI_KEYWORDS_PATCH = {
  "blonde_woman" => { "fitzpatrick_scale" => true }
}

The db.json generated (and images) now contain the skin tones variations:

{
  "code": "1f44b-1f3fd",
  "name": "wave:t4"
}

You can read a first draft of the code here, I still have some cleaning/improvements to do, but it’s probably good enough for first comments : gist:f56236a0858471a58c643aa1a3dc3e3d · GitHub

Usage should be as simple as rake -f emoji.rake emoji:clean && rake -f emoji.rake emoji:update

I should soon be able to make a pull request containing this. Thanks to @sam for some of theses ideas.

TODO:

(Rafael dos Santos Silva) #30

Great work @joffreyjaffeux!

Another point for your TODO (probably the last one), use Zopfli instead of pngout to optimize the emojis.


(Joffrey Jaffeux) #31

thx!

Great suggestion, I will use it.


(Joffrey Jaffeux) #32

I have had mixed results with zopflipng on our usecase, so I built a quick benchmarking tool (it tests only 5 images at the moment, I could add more if you want to test on a larger set): benchmark image optim solution · GitHub

➜  Projects ruby test.rb
-- Time benchmark
       user     system      total        real
raw  0.000000   0.000000   0.000000 (  0.001062)
pngout -s0  0.000000   0.000000   0.040000 (  0.054050)
zopflipng 20iter  0.000000   0.010000   2.440000 (  2.449912)
zopflipng max  0.000000   0.000000 579.280000 (579.331887)

-- Size benchmark
Raw: 6737.0 bytes
pngout -s0: 6393.0 bytes (-5.106130325070506 %)
zopflipng 20iter: 6428.0 bytes (-4.586611251298798 %)
zopflipng max: 6409.0 bytes (-4.868635891346297 %)

In the end I feel like zopflipng could still be a small improvement for the 9000+ images (even if the benchmark is saying differently my few tests on very large set shown a very very small improvement using zopflipng), but it would only beat pngout by few kilobytes.

Am I missing something? a flag? a custom binary? @Falco @sam @codinghorror


(Joffrey Jaffeux) #33

In other news I’ve updated the script to now generate app/assets/javascripts/discourse/lib/emoji/groups.js.es6 and warn us for any uncategorized emoji.

Current list (spec 5.0) of uncategorized emojis:

call_me_hand
fist_left
fist_right
raised_back_of_hand
fu
raised_hand_with_fingers_splayed
crossed_fingers
vulcan_salute
selfie
blonde_man
policeman
construction_worker_man
male_detective
mrs_claus
pregnant_woman
prince
man_in_tuxedo
running_man
walking_man
man_dancing
bowing_man
man_facepalming
woman_shrugging
tipping_hand_woman
no_good_woman
raising_hand_woman
pouting_woman
frowning_woman
haircut_woman
massage_woman
golfing_man
rowing_man
swimming_man
surfing_man
basketball_man
weight_lifting_man
biking_man
mountain_biking_man
business_suit_levitating
sleeping_bed

I also added a loop to check for categorized icons which are not created, most of the times it’s related to men only/unisex icons, getting a woman and man version, or a new name:

[!] slight_smile is categorized but doesn’t exist. Remove it or add an alias.
[!] family is categorized but doesn’t exist. Remove it or add an alias.
[!] couple is categorized but doesn’t exist. Remove it or add an alias.
[!] two_men_holding_hands is categorized but doesn’t exist. Remove it or add an alias.
[!] two_women_holding_hands is categorized but doesn’t exist. Remove it or add an alias.
[!] dancers is categorized but doesn’t exist. Remove it or add an alias.
[!] person_with_blond_hair is categorized but doesn’t exist. Remove it or add an alias.
[!] cop is categorized but doesn’t exist. Remove it or add an alias.
[!] construction_worker is categorized but doesn’t exist. Remove it or add an alias.
[!] bow is categorized but doesn’t exist. Remove it or add an alias.
[!] information_desk_person is categorized but doesn’t exist. Remove it or add an alias.
[!] no_good is categorized but doesn’t exist. Remove it or add an alias.
[!] raising_hand is categorized but doesn’t exist. Remove it or add an alias.
[!] person_with_pouting_face is categorized but doesn’t exist. Remove it or add an alias.
[!] person_frowning is categorized but doesn’t exist. Remove it or add an alias.
[!] massage is categorized but doesn’t exist. Remove it or add an alias.
[!] haircut is categorized but doesn’t exist. Remove it or add an alias.
[!] couple_with_heart is categorized but doesn’t exist. Remove it or add an alias.
[!] couplekiss is categorized but doesn’t exist. Remove it or add an alias.
[!] hand is categorized but doesn’t exist. Remove it or add an alias.
[!] anger_right is categorized but doesn’t exist. Remove it or add an alias.
[!] frowning2 is categorized but doesn’t exist. Remove it or add an alias.
[!] hand_splayed is categorized but doesn’t exist. Remove it or add an alias.
[!] head_bandage is categorized but doesn’t exist. Remove it or add an alias.
[!] hugging is categorized but doesn’t exist. Remove it or add an alias.
[!] middle_finger is categorized but doesn’t exist. Remove it or add an alias.
[!] money_mouth is categorized but doesn’t exist. Remove it or add an alias.
[!] nerd is categorized but doesn’t exist. Remove it or add an alias.
[!] punch is categorized but doesn’t exist. Remove it or add an alias.
[!] rolling_eyes is categorized but doesn’t exist. Remove it or add an alias.
[!] skull_crossbones is categorized but doesn’t exist. Remove it or add an alias.
[!] slight_frown is categorized but doesn’t exist. Remove it or add an alias.
[!] spy is categorized but doesn’t exist. Remove it or add an alias.
[!] thumbsup is categorized but doesn’t exist. Remove it or add an alias.
[!] thumbsdown is categorized but doesn’t exist. Remove it or add an alias.
[!] upside_down is categorized but doesn’t exist. Remove it or add an alias.
[!] urn is categorized but doesn’t exist. Remove it or add an alias.
[!] vulcan is categorized but doesn’t exist. Remove it or add an alias.
[!] wind_blowing_face is categorized but doesn’t exist. Remove it or add an alias.
[!] zipper_mouth is categorized but doesn’t exist. Remove it or add an alias.
[!] female_couple_with_heart is categorized but doesn’t exist. Remove it or add an alias.
[!] male_couple_with_heart is categorized but doesn’t exist. Remove it or add an alias.
[!] female_couplekiss is categorized but doesn’t exist. Remove it or add an alias.
[!] male_couplekiss is categorized but doesn’t exist. Remove it or add an alias.
[!] family_man_woman_boys is categorized but doesn’t exist. Remove it or add an alias.
[!] family_man_woman_girls is categorized but doesn’t exist. Remove it or add an alias.
[!] family_women_boy is categorized but doesn’t exist. Remove it or add an alias.
[!] family_women_girl is categorized but doesn’t exist. Remove it or add an alias.
[!] family_women_girl_boy is categorized but doesn’t exist. Remove it or add an alias.
[!] family_women_boys is categorized but doesn’t exist. Remove it or add an alias.
[!] family_women_girls is categorized but doesn’t exist. Remove it or add an alias.
[!] family_men_boy is categorized but doesn’t exist. Remove it or add an alias.
[!] family_men_girl is categorized but doesn’t exist. Remove it or add an alias.
[!] family_men_girl_boy is categorized but doesn’t exist. Remove it or add an alias.
[!] family_men_boys is categorized but doesn’t exist. Remove it or add an alias.
[!] family_men_girls is categorized but doesn’t exist. Remove it or add an alias.
[!] bee is categorized but doesn’t exist. Remove it or add an alias.
[!] feet is categorized but doesn’t exist. Remove it or add an alias.
[!] moon is categorized but doesn’t exist. Remove it or add an alias.
[!] cloud_lightning is categorized but doesn’t exist. Remove it or add an alias.
[!] cloud_rain is categorized but doesn’t exist. Remove it or add an alias.
[!] cloud_snow is categorized but doesn’t exist. Remove it or add an alias.
[!] cloud_tornado is categorized but doesn’t exist. Remove it or add an alias.
[!] lion_face is categorized but doesn’t exist. Remove it or add an alias.
[!] thunder_cloud_rain is categorized but doesn’t exist. Remove it or add an alias.
[!] white_sun_cloud is categorized but doesn’t exist. Remove it or add an alias.
[!] white_sun_rain_cloud is categorized but doesn’t exist. Remove it or add an alias.
[!] white_sun_small_cloud is categorized but doesn’t exist. Remove it or add an alias.
[!] heart_exclamation is categorized but doesn’t exist. Remove it or add an alias.
[!] runner is categorized but doesn’t exist. Remove it or add an alias.
[!] walking is categorized but doesn’t exist. Remove it or add an alias.
[!] rowboat is categorized but doesn’t exist. Remove it or add an alias.
[!] swimmer is categorized but doesn’t exist. Remove it or add an alias.
[!] surfer is categorized but doesn’t exist. Remove it or add an alias.
[!] bicyclist is categorized but doesn’t exist. Remove it or add an alias.
[!] mountain_bicyclist is categorized but doesn’t exist. Remove it or add an alias.
[!] basketball_player is categorized but doesn’t exist. Remove it or add an alias.
[!] golfer is categorized but doesn’t exist. Remove it or add an alias.
[!] hockey is categorized but doesn’t exist. Remove it or add an alias.
[!] skier is categorized but doesn’t exist. Remove it or add an alias.
[!] snowman2 is categorized but doesn’t exist. Remove it or add an alias.
[!] car is categorized but doesn’t exist. Remove it or add an alias.
[!] boat is categorized but doesn’t exist. Remove it or add an alias.
[!] gb is categorized but doesn’t exist. Remove it or add an alias.
[!] airplane_arriving is categorized but doesn’t exist. Remove it or add an alias.
[!] airplane_departure is categorized but doesn’t exist. Remove it or add an alias.
[!] airplane_small is categorized but doesn’t exist. Remove it or add an alias.
[!] beach is categorized but doesn’t exist. Remove it or add an alias.
[!] city_dusk is categorized but doesn’t exist. Remove it or add an alias.
[!] construction_site is categorized but doesn’t exist. Remove it or add an alias.
[!] cruise_ship is categorized but doesn’t exist. Remove it or add an alias.
[!] flag_black is categorized but doesn’t exist. Remove it or add an alias.
[!] flag_white is categorized but doesn’t exist. Remove it or add an alias.
[!] homes is categorized but doesn’t exist. Remove it or add an alias.
[!] house_abandoned is categorized but doesn’t exist. Remove it or add an alias.
[!] island is categorized but doesn’t exist. Remove it or add an alias.
[!] map is categorized but doesn’t exist. Remove it or add an alias.
[!] motorboat is categorized but doesn’t exist. Remove it or add an alias.
[!] park is categorized but doesn’t exist. Remove it or add an alias.
[!] race_car is categorized but doesn’t exist. Remove it or add an alias.
[!] sleeping_accommodation is categorized but doesn’t exist. Remove it or add an alias.
[!] shirt is categorized but doesn’t exist. Remove it or add an alias.
[!] book is categorized but doesn’t exist. Remove it or add an alias.
[!] keycap_star is categorized but doesn’t exist. Remove it or add an alias.
[!] atom is categorized but doesn’t exist. Remove it or add an alias.
[!] bellhop is categorized but doesn’t exist. Remove it or add an alias.
[!] calendar_spiral is categorized but doesn’t exist. Remove it or add an alias.
[!] camera_with_flash is categorized but doesn’t exist. Remove it or add an alias.
[!] card_box is categorized but doesn’t exist. Remove it or add an alias.
[!] clock is categorized but doesn’t exist. Remove it or add an alias.
[!] compression is categorized but doesn’t exist. Remove it or add an alias.
[!] couch is categorized but doesn’t exist. Remove it or add an alias.
[!] cross is categorized but doesn’t exist. Remove it or add an alias.
[!] desktop is categorized but doesn’t exist. Remove it or add an alias.
[!] dividers is categorized but doesn’t exist. Remove it or add an alias.
[!] envelope is categorized but doesn’t exist. Remove it or add an alias.
[!] film_frames is categorized but doesn’t exist. Remove it or add an alias.
[!] fleur-de-lis is categorized but doesn’t exist. Remove it or add an alias.
[!] fork_knife_plate is categorized but doesn’t exist. Remove it or add an alias.
[!] frame_photo is categorized but doesn’t exist. Remove it or add an alias.
[!] hammer_pick is categorized but doesn’t exist. Remove it or add an alias.
[!] helmet_with_cross is categorized but doesn’t exist. Remove it or add an alias.
[!] key2 is categorized but doesn’t exist. Remove it or add an alias.
[!] knife is categorized but doesn’t exist. Remove it or add an alias.
[!] levitate is categorized but doesn’t exist. Remove it or add an alias.
[!] lifter is categorized but doesn’t exist. Remove it or add an alias.
[!] medal is categorized but doesn’t exist. Remove it or add an alias.
[!] microphone2 is categorized but doesn’t exist. Remove it or add an alias.
[!] military_medal is categorized but doesn’t exist. Remove it or add an alias.
[!] mouse_three_button is categorized but doesn’t exist. Remove it or add an alias.
[!] newspaper2 is categorized but doesn’t exist. Remove it or add an alias.
[!] notepad_spiral is categorized but doesn’t exist. Remove it or add an alias.
[!] oil is categorized but doesn’t exist. Remove it or add an alias.
[!] om_symbol is categorized but doesn’t exist. Remove it or add an alias.
[!] peace is categorized but doesn’t exist. Remove it or add an alias.
[!] pen_ballpoint is categorized but doesn’t exist. Remove it or add an alias.
[!] pen_fountain is categorized but doesn’t exist. Remove it or add an alias.
[!] pencil is categorized but doesn’t exist. Remove it or add an alias.
[!] play_pause is categorized but doesn’t exist. Remove it or add an alias.
[!] projector is categorized but doesn’t exist. Remove it or add an alias.
[!] satellite_orbital is categorized but doesn’t exist. Remove it or add an alias.
[!] scales is categorized but doesn’t exist. Remove it or add an alias.
[!] shopping_bags is categorized but doesn’t exist. Remove it or add an alias.
[!] telephone is categorized but doesn’t exist. Remove it or add an alias.
[!] ten is categorized but doesn’t exist. Remove it or add an alias.
[!] thermometer_face is categorized but doesn’t exist. Remove it or add an alias.
[!] timer is categorized but doesn’t exist. Remove it or add an alias.
[!] tools is categorized but doesn’t exist. Remove it or add an alias.
[!] track_next is categorized but doesn’t exist. Remove it or add an alias.
[!] track_previous is categorized but doesn’t exist. Remove it or add an alias.
[!] umbrella2 is categorized but doesn’t exist. Remove it or add an alias.

script: gist:f56236a0858471a58c643aa1a3dc3e3d · GitHub


(Joffrey Jaffeux) #34

@Falco Most of my remaining work (before UI work) is to fix all the uncategorized/new
emojis/different names/aliases… I’m doing my best, but it’s quite an error prone job. I was wondering, will you have a way to track on the instances you are managing an elevation of 404’s on /public/images/emoji/**/*.png ? It would greatly improve my confidence we are not breaking anything, at least for too long.


(Régis Hanol) #35

Yup, we’ll track that once your changes are merged.


(Joffrey Jaffeux) #36

I already have a post on local instance with all the emojis of the current version used, so I should catch most errors :wink:


(Joffrey Jaffeux) #37

Look who is coming to the party soon: :pray:


(Joffrey Jaffeux) #38

Hi everyone,

a quick update on the progress concerning the support of skin tones:


(Rafael dos Santos Silva) #39

Hey @joffreyjaffeux, looks like there is an official emoji order and grouping on the Unicode standard. It even fix some stuff people asked, like thumbs up before thumbs down on autocomplete.

http://www.unicode.org/emoji/charts/emoji-ordering.html


(Joffrey Jaffeux) #40

ok I will make it work :wink: