fix(indexer): give lang the same tag as content_<lang> when a document brings its own language - #3450
Merged
Merged
Conversation
…t brings its own language A page declaring <html lang="zh-CN"> was indexed with lang=zh_CN while its text went to content_zh-cn, and a Chinese page without the attribute got lang=zh-cn. The same language was stored two ways, and lang:zh-cn found only the pages whose language had been detected. FessXpathTransformer stores SystemHelper#normalizeLang of the attribute, which answers with the spelling of supported.languages, zh_CN. LanguageHelper#updateDocument resolves it to zh-cn for the content copy but left lang untouched. createScript, used by the label, click and favorite count updates, then copied the text to content_zh_CN, which no analyzer template matches. When a document already carries a lang string that resolves to a supported language, updateDocument now writes the resolved tag back. This also covers crawler.document.html.default.lang and crawler.document.file.default.lang configured as zh_CN, and pt_BR, en_IE and ckb_IQ. SystemHelper#normalizeLang is unchanged: its request-side values go through query.language.mapping and are never compared with the stored lang.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while verifying 15.9.0. A page that declares
<html lang="zh-CN">is indexed withlang=zh_CN, while its text is copied tocontent_zh-cnand a Chinese page without the attribute getslang=zh-cn. The same language ends up in the index written two ways, andlang:zh-cnfinds only the pages whose language was detected.Cause
FessXpathTransformerstoresSystemHelper#normalizeHtmlLangof thelangattribute (FessXpathTransformer.java:509-512), andSystemHelper#normalizeLanganswers with the entry ofsupported.languagesas it is spelled there,zh_CN(SystemHelper.java:586-599). Since #3394,LanguageHelper#updateDocumentresolves that value throughgetSupportedLanguage, which returns the normalized tagzh-cn, and copies the text tocontent_zh-cn, but leaveslangas it found it. A detected language, on the other hand, is written tolangalready normalized.The stored value is used again later.
LanguageHelper#createScript, which the label update job and the click and favorite count updates use, copies the text tocontent_+lang. For a document withlang=zh_CNthat iscontent_zh_CN, which no analyzer template infess_indices/fess/doc.jsonmatches (the Chinese one matches*_zh-cn).Change
When a document already carries a
langstring that resolves to a supported language,updateDocumentwrites the resolved tag back. That covers thelangattribute of HTML, acrawler.document.html.default.langorcrawler.document.file.default.langconfigured aszh_CN, and a data store that setslang; it applies tozh_TW,pt_BR,en_IEandckb_IQthe same way. A value that is already normalized, such asja, is unchanged.SystemHelper#normalizeLangstays as it is. It also resolves thelangrequest parameter and the browser locale, which pass throughquery.language.mapping(it maps bothzh_CNandzh-cntozh-cn) to choose thecontent_<lang>fields to boost; they are never compared with the storedlang.Compatibility
Documents already in the index keep
lang=zh_CNuntil they are crawled again. Alangsent as a list through the admin documents API is left as the client sent it.This is independent of the kana folding change to language detection in the same class; either can be merged first.
Verification
LanguageHelperTest#test_updateDocument_normalizesExistingLangfailed before the change (expected: <zh-cn> but was: <zh_CN>) and passes after it.LanguageHelperTest: 32 tests, 0 failures.mvn test: 7,488 tests, 0 failures, 0 errors.