Arabic Search Normalization: Missing Support for Hamza Variants, Ya/Kaf Forms,and Orthographic Equivalence

Contribute Feature
, , ,
10

I’m not a technical expert, but I’ve been researching this issue because I want to ensure that no search queries are missed on a Persian-Arabic bilingual Discourse forum. Since Discourse uses PostgreSQL, normalization becomes essential: a user might search using Persian characters, while the same word is stored using Arabic ones—or vice versa. Without proper normalization, the search will fail.
Based on what I’ve learned, using Unicode NFKC normalization is a solid starting point—it handles many compatibility cases like ligatures, presentation forms, and Arabic/Persian digits.

However, for Persian and Arabic text, NFKC alone is not sufficient. It does not normalize several critical character variants that are visually and semantically equivalent but differ at the binary level.

Below, I’m outlining the procedures and insights I’ve arrived at through my research and exploration.

:wrench: Overall Design Strategy

  1. Apply Unicode NFKC normalization first to handle ligatures, presentation forms, and digit unification.
  2. Then apply custom character mappings in a defined order (e.g., normalize Hamza variants before Arabic Ya).
  3. Separate normalization policies for storage vs. search:
    • Use a Conservative profile for canonical storage (preserve ZWNJ, avoid semantic shifts).
    • Use a Permissive profile for search (ignore ZWNJ, unify Hamza variants, normalize digits).
  4. All mappings should be configurable via a centralized mapping table in the database or a Ruby hash in the application.

:one: Normalization Profiles

:green_circle: Conservative (for storage)

  • Minimal transformation
  • Apply NFKC
  • Normalize Arabic Kaf/Ya to Persian equivalents
  • Remove diacritics
  • Preserve ZWNJ
  • Store as original_text + normalized_conservative

:blue_circle: Permissive (for search)

  • Aggressive matching
  • Apply all Conservative rules
  • Remove/ignore ZWNJ
  • Normalize Hamza variants to base letters
  • Convert all digits to ASCII
  • Optionally unify Taa Marbuta → Heh
  • Used for query preprocessing

:two: Comprehensive Mapping Table

Source Target Unicode Notes
ك ک U+0643 → U+06A9 Arabic Kaf → Persian Kaf
ي ی U+064A → U+06CC Arabic Ya → Persian Ya
ى ی U+0649 → U+06CC Final Ya variant
أ, إ, ٱ ا Various → U+0627 Hamza forms → Alef
ؤ و U+0624 → U+0648 Hamza Waw
ئ ی U+0626 → U+06CC Hamza Ya
ء — U+0621 Remove or preserve (configurable)
ة ه U+0629 → U+0647 Taa Marbuta → Heh (optional)
ۀ هٔ U+06C0 ↔ U+0647+U+0654 Normalize composed form
ڭ گ U+06AD → U+06AF Regional variants
‌ — U+200C ZWNJ: preserve in Conservative, remove in Permissive
٤, ۴ 4 U+0664, U+06F4 → ASCII Normalize digits
Diacritics — U+064B–U+0652 Remove all harakat
ZWJ — U+200D Remove invisible joiners
Multiple spaces Single space — Normalize spacing

:three: Fast Mapping Snippet (for SQL or Ruby)

ك → ک
ي → ی
ى → ی
أ → ا
إ → ا
ؤ → و
ئ → ی
ة → ه
ۀ → هٔ
ٱ → ا
٤, ۴ → 4
ZWNJ (U+200C) → (removed in permissive)
Harakat (U+064B..U+0652) → removed
ZWJ (U+200D) → removed

:four: Implementation in PostgreSQL

  • Create a text_normalization_map table
  • Use regexp_replace or TRANSLATE chains for performance
  • Optionally implement in PL/Python or PL/v8 for Unicode support
  • Normalize both stored content and incoming queries using the same logic

Indexing Strategy

  • Store normalized_conservative for canonical indexing
  • Normalize queries with normalize_persian_arabic(query, 'permissive')
  • If using permissive search, index must match the same profile
  • Optionally store both versions for cross-compare

:five: Ruby Hash Example (for Discourse)

NORMALIZATION_MAP = {
  "ك" => "ک",
  "ي" => "ی",
  "ى" => "ی",
  "أ" => "ا",
  "إ" => "ا",
  "ٱ" => "ا",
  "ؤ" => "و",
  "ئ" => "ی",
  "ة" => "ه",
  "ۀ" => "هٔ",
  "۴" => "4",
  "٤" => "4",
  "\u200C" => "", # ZWNJ
  "\u200D" => "", # ZWJ
}

:six: Performance and Practical Notes

  1. Apply NFKC in the application layer (e.g., Ruby unicode_normalize(:nfkc))
  2. Use separate indexes for conservative vs. permissive profiles
  3. Avoid forced mappings of semantically sensitive characters (e.g., Hamza, Taa Marbuta) unless explicitly configured
  4. Run A/B tests on real forum data to measure hit rate and false positives
  5. Document each mapping with rationale and examples
  6. Define unit tests in both Ruby and SQL for each mapping

:seven: Final Recommendation

  • Use Unicode NFKC as a base
  • Extend it with a custom mapping layer
  • Maintain dual profiles for storage and search
  • Implement normalization in both the app and database layers
  • Document and test every mapping
  • Build appropriate indexes (GIN + to_tsvector) on normalized columns