Skip to content

Judeo-Arabic to Arabic-script transliteration helper (representations form) #20

Description

@bsesic

Summary

Add a Judeo-Arabic ↔ Arabic-script transliteration helper that produces an Arabic-script
representations form for Arabic written in Hebrew script. Small, self-contained utility that
fits TRACE's existing Token.representations model and feeds the embedding step of the
cross-lingual path (#17).

Motivation

Much of the target corpus is Judeo-Arabic — Arabic in Hebrew characters. Two concrete payoffs:

  1. For the cross-lingual embedding fill, transliterating Hebrew-script Arabic into Arabic script
    before embedding moves the text closer to what multilingual embedding models (LaBSE etc.)
    have seen and measurably improves matching.
  2. It lets Judeo-Arabic text reuse the Arabic pack's (Arabic language pack (ara) — proclitic tokenization + orthographic normalization #16) normalization/scoring once
    mapped, rather than needing a parallel ruleset.

This is the one place the script-conversion trick legitimately applies — it helps when the
underlying wording is the same text in a different script. (Note for users: it does not
help across a translation, where the wording itself differs.)

Scope

  • A deterministic mapping function Hebrew-script → Arabic-script for Judeo-Arabic, handling the
    conventional correspondences (and the well-known ambiguous letters), exposed as a normalizer
    that writes an arabic_script entry into Token.representations (diplomatic Hebrew-script
    form preserved in raw/text).
  • Configurable/overridable mapping table (conventions vary by period and community), shipped as
    a data file under the relevant pack.
  • Documented as best-effort with known ambiguities flagged, not a lossless transliteration.

Acceptance criteria

  • A transliteration utility producing an arabic_script representation for Judeo-Arabic tokens.
  • Mapping table is data-driven and user-overridable.
  • Tests on a small Judeo-Arabic fixture, including documented ambiguous-letter cases.
  • Integrates as an optional pre-step feeding the cross-lingual embedding adapter (Cross-lingual alignment path: anchor-based mode + embedding-fill delegation hook #17).
  • TDD; suite green on 3.10/3.11/3.12; flake8 clean.

Notes

Roadmap: supporting utility for Stage 6 cross-lingual work; serves the Judeo-Arabic portion
of the medieval-editions corpus.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions