You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a Judeo-Arabic ↔ Arabic-script transliteration helper that produces an Arabic-script representations form for Arabic written in Hebrew script. Small, self-contained utility that
fits TRACE's existing Token.representations model and feeds the embedding step of the
cross-lingual path (#17).
Motivation
Much of the target corpus is Judeo-Arabic — Arabic in Hebrew characters. Two concrete payoffs:
For the cross-lingual embedding fill, transliterating Hebrew-script Arabic into Arabic script
before embedding moves the text closer to what multilingual embedding models (LaBSE etc.)
have seen and measurably improves matching.
This is the one place the script-conversion trick legitimately applies — it helps when the
underlying wording is the same text in a different script. (Note for users: it does not
help across a translation, where the wording itself differs.)
Scope
A deterministic mapping function Hebrew-script → Arabic-script for Judeo-Arabic, handling the
conventional correspondences (and the well-known ambiguous letters), exposed as a normalizer
that writes an arabic_script entry into Token.representations (diplomatic Hebrew-script
form preserved in raw/text).
Configurable/overridable mapping table (conventions vary by period and community), shipped as
a data file under the relevant pack.
Documented as best-effort with known ambiguities flagged, not a lossless transliteration.
Acceptance criteria
A transliteration utility producing an arabic_script representation for Judeo-Arabic tokens.
Mapping table is data-driven and user-overridable.
Tests on a small Judeo-Arabic fixture, including documented ambiguous-letter cases.
Summary
Add a Judeo-Arabic ↔ Arabic-script transliteration helper that produces an Arabic-script
representationsform for Arabic written in Hebrew script. Small, self-contained utility thatfits TRACE's existing
Token.representationsmodel and feeds the embedding step of thecross-lingual path (#17).
Motivation
Much of the target corpus is Judeo-Arabic — Arabic in Hebrew characters. Two concrete payoffs:
before embedding moves the text closer to what multilingual embedding models (LaBSE etc.)
have seen and measurably improves matching.
mapped, rather than needing a parallel ruleset.
This is the one place the script-conversion trick legitimately applies — it helps when the
underlying wording is the same text in a different script. (Note for users: it does not
help across a translation, where the wording itself differs.)
Scope
conventional correspondences (and the well-known ambiguous letters), exposed as a normalizer
that writes an
arabic_scriptentry intoToken.representations(diplomatic Hebrew-scriptform preserved in
raw/text).a data file under the relevant pack.
Acceptance criteria
arabic_scriptrepresentation for Judeo-Arabic tokens.flake8clean.Notes
but the mapping utility itself can be built and tested independently.
Roadmap: supporting utility for Stage 6 cross-lingual work; serves the Judeo-Arabic portion
of the medieval-editions corpus.