parsigo

Persian text processing for Go.

It repairs the silent encoding problems that make Persian search, deduplication, and joins fail: Arabic look-alike letters, mixed digit families, decorative kashida, invisible controls, and the zero-width non-joiner (نیم‌فاصله) that holds می‌رود together.

fmt.Println(parsigo.Normalize("علي كتاب"))
// علی کتاب

fmt.Println(parsigo.Edit("می رود , دنیا ..."))
// می‌رود، دنیا…

fmt.Println(parsigo.EqualFold("علی", "علي"))
// true

Requires Go 1.25+.

go get github.com/masoudkaviani/parsigo@latest

Three functions, three jobs

Function	Use it when	What it changes	What it leaves alone
Normalize	the result will be shown to a person	Arabic ي/ك, kashida, invisibles, broken spacing, misplaced ZWNJ	digits, diacritics, punctuation style, ezafe spelling
Edit	you are publishing the text	everything Normalize does, plus Persian punctuation, guillemets, ellipsis, ZWNJ insertion, Persian digits	quoted Arabic you have shielded
Fold	you are indexing or deduplicating	almost everything, so spelling variants collide onto one key	genuine word differences such as آب vs اب

Fold is lossy. Never display its output.

fmt.Println(parsigo.Normalize("سلـــام"))     // سلام          (kashida removed)
fmt.Println(parsigo.Normalize("سال 1403"))    // سال 1403      (digits kept)
fmt.Println(parsigo.Edit("سال 1403"))         // سال ۱۴۰۳      (digits become Persian)

fmt.Println(parsigo.Fold("می‌رود"))            // میرود
fmt.Println(parsigo.Fold("علي"))               // علی
fmt.Println(parsigo.Fold("أحمد"))              // احمد
fmt.Println(parsigo.Fold("خانۀ"))              // خانه
fmt.Println(parsigo.Fold("سلام، دنیا! خوبی؟")) // سلام دنیا خوبی

EqualFold compares after folding, so the variants a reader treats as the same word compare equal — and real distinctions do not:

fmt.Println(parsigo.EqualFold("علی", "علي"))       // true   (Persian vs Arabic yeh)
fmt.Println(parsigo.EqualFold("می‌رود", "میرود"))  // true   (ZWNJ present or not)
fmt.Println(parsigo.EqualFold("آب", "اب"))         // false  (different words)
fmt.Println(parsigo.Similarity("علي", "علی"))      // 1

Configure a normalizer

Build from a preset, then override only the policies you care about. Every contested choice is an enum, not a boolean.

n := parsigo.New(
    parsigo.WithPreset(parsigo.PresetEditorial),
    parsigo.WithDigits(parsigo.DigitsPersian),
    parsigo.WithZWNJ(parsigo.ZWNJFix),
    parsigo.WithEzafe(parsigo.EzafeKeep),
    parsigo.WithHamza(parsigo.HamzaKeep),
)

fmt.Println(n.Normalize("می رود 123"))
// می‌رود ۱۲۳

A Normalizer is immutable after construction and safe to share across goroutines.

Presets

Preset	For	Behaviour
PresetDefault	user-visible text	Normalize: repair what is objectively broken, leave style alone
PresetEditorial	CMS / publishing	Edit: Persian punctuation, guillemets, ellipsis, ZWNJ insertion, Persian digits
PresetSearch	indexes, dedup	Fold: strip marks, drop ZWNJ, fold hamza/ة/ezafe, ASCII digits, strip punctuation
PresetNLP	tokenizers, models	fold look-alikes, strip marks, fix ZWNJ, ASCII digits, keep punctuation as signal
PresetMinimal	already-clean text	NFC, drop invisibles and bidi controls, normalize line endings

ZWNJ (نیم‌فاصله)

fix := parsigo.New(parsigo.WithZWNJ(parsigo.ZWNJFix))
fmt.Println(fix.Normalize("می رود"))     // می‌رود     (space → ZWNJ)
fmt.Println(fix.Normalize("کتاب ها"))    // کتاب‌ها
fmt.Println(fix.Normalize("سلام دنیا"))  // سلام دنیا  (ordinary words stay apart)

agg := parsigo.New(parsigo.WithZWNJ(parsigo.ZWNJAggressive))
fmt.Println(agg.Normalize("میرود"))      // می‌رود     (solid compound split)
fmt.Println(agg.Normalize("مینا"))       // مینا       (a name, not می + نا)
fmt.Println(agg.Normalize("میدان"))      // میدان      (a square, not می + دان)

ZWNJAggressive is off by default. Splitting a solid word needs a lexicon, and some inputs have no single correct answer: میدانم is both “I know” and “my square”.

Digits, ezafe, hamza

There are three digit families. Unicode will not convert between them; parsigo does, explicitly.

Policy	Example
DigitsKeep	1403 stays 1403
DigitsPersian	1403 → ۱۴۰۳
DigitsASCII	۱۴۰۳ → 1403
DigitsArabic	۱۴۰۳ → ١٤٠٣

The ezafe on words ending in ه has three living spellings — خانۀ, خانهٔ, خانه‌ی — with no accepted authority to choose between them. Display pipelines should EzafeKeep. Search pipelines should EzafeStrip (all three become خانه).

Folding أ/إ to ا and ة to ه is correct for Persian and destructive to quoted Arabic. Keep them for display; fold them for search.

Protected spans

URLs, emails, and code are left untouched so a query string is not rewritten as Persian punctuation:

fmt.Println(parsigo.Edit("ببین https://example.com/a?x=1 و برو"))
// ببین https://example.com/a?x=1 و برو

Add your own shields for templates, product codes, or English spans:

n := parsigo.New(
    parsigo.WithPreset(parsigo.PresetEditorial),
    parsigo.WithProtectedPatterns(regexp.MustCompile(`\{\{[^}]*\}\}`)),
)

fmt.Println(n.Normalize("مبلغ {{user.id, 1}} است"))
// مبلغ {{user.id, 1}} است

Tokenize and split sentences

The default tokenizer repairs ZWNJ first, then keeps compounds as one word. URLs, mentions, hashtags, and numbers stay intact.

fmt.Printf("%q\n", parsigo.Words("می رود به خانه"))
// ["می‌رود" "به" "خانه"]

for _, tok := range parsigo.Tokenize("سلام @ali #تست https://example.com 42") {
    fmt.Printf("%s\t%s\n", tok.Kind, tok.Text)
}
// word     سلام
// mention  @ali
// hashtag  #تست
// url      https://example.com
// number   42

for _, s := range parsigo.Sentences("قیمت 3.14 است. تمام.") {
    fmt.Println(s)
}
// قیمت 3.14 است.
// تمام.

A decimal, a dotted identifier such as main.go, and a title like Dr. are not treated as sentence endings.

Analyze (lint, do not rewrite)

Analyze is the read-only counterpart to Normalize. Offsets are byte offsets into the original string, so an editor can highlight the finding.

rep := parsigo.Analyze("علي")
fmt.Println(rep.NeedsNormalization())
for _, issue := range rep.Issues {
    fmt.Println(issue)
}

true
4: [arabic-yeh] Arabic yeh U+064A should be Persian yeh U+06CC ("ي" -> "ی")

Numbers

fmt.Println(parsigo.ToPersianDigits("2024"))     // ۲۰۲۴
fmt.Println(parsigo.ParseInt("۱٬۲۳۴"))           // 1234 <nil>
fmt.Println(parsigo.ToWords(1234))               // هزار و دویست و سی و چهار
fmt.Println(parsigo.ToOrdinal(30))               // سی‌ام
fmt.Println(parsigo.ToOrdinal(21))               // بیست و یکم
fmt.Println(parsigo.ParseWords("هزار و دویست و سی و چهار"))
// 1234 <nil>
fmt.Println(parsigo.PersianFormat.Int(1234567))  // ۱٬۲۳۴٬۵۶۷
fmt.Println(parsigo.ASCIIFormat.Int(1234567))    // 1,234,567

ParseInt and ParseFloat accept ASCII, Persian, and Arabic-Indic digits, plus Persian decimal (٫) and thousands (٬) separators.

Sort in dictionary order

Byte order misplaces گ, ک, and چ, because those letters were assigned code points far from their alphabetical neighbours. SortStrings uses Persian dictionary order (آ first, then ا … ی).

names := []string{"گل", "الف", "کتاب", "چای"}
parsigo.SortStrings(names)
fmt.Println(names)
// [الف چای کتاب گل]

CollationKey returns a string whose byte order matches that sort, suitable for a database column.

Truncate without corrupting glyphs

مُحَمَّد is eight runes and four letters. Cutting by rune index can leave a dangling diacritic or ZWNJ.

fmt.Println(len([]rune("مُحَمَّد")))            // 8
fmt.Println(parsigo.VisualLen("مُحَمَّد"))     // 4
fmt.Println(parsigo.Truncate("می‌رود", 2, "…")) // می…

Command line

go install github.com/masoudkaviani/parsigo/cmd/parsigo@latest

$ echo علي كتاب | parsigo
علی کتاب

$ echo "می رود , دنیا ..." | parsigo -mode edit
می‌رود، دنیا…

$ echo علی | parsigo -mode fold
علی

$ echo علي | parsigo -mode analyze
bytes=6 runes=3 words=1 sentences=1 script=persian
...
4: [arabic-yeh] Arabic yeh U+064A should be Persian yeh U+06CC ("ي" -> "ی")

$ echo "می رود به خانه" | parsigo -mode words
می‌رود
به
خانه

Useful flags: -preset editorial, -digits fa, -zwnj fix, -ezafe keep, -strip-harakat.

Guarantees

• Idempotent. n.Normalize(n.Normalize(s)) == n.Normalize(s).
• Never reorders text. Logical order is preserved; the renderer handles bidi. Repairing visual-order (legacy DOS/PDF) text is out of scope.
• NFC by default. Mark stripping uses an allow-list, so آ and ۀ are not dismantled.
• Concurrent. A Normalizer and a Tokenizer hold no mutable state after construction.

What this is not

Jalali dates, national-ID validation, and Sheba checks belong in a different library — they share no code with text normalization. Full Unicode grapheme segmentation is also a non-goal.

License

MIT. See LICENSE.