Grammar download failure reported as "Language not found" and silently degrades chunking; support tree-sitter-* wheels

[pszemraj]

Follow-on from #94. Catching the exception fixed the crash, but semble now silently falls back to line chunking and the caller never finds out.

Clean repro. Box behind a TLS-intercepting proxy with no route to github.com:

$ python -c "from tree_sitter_language_pack import download; download(['python'])"
DownloadError: Download error: Failed to fetch manifest from
https://github.com/kreuzberg-dev/tree-sitter-language-pack/releases/download/v1.6.2/parsers.json:
io: invalid peer certificate: UnknownIssuer

$ pip install tree-sitter-python tree-sitter-cpp tree-sitter-c
Successfully installed tree-sitter-c-0.24.2 tree-sitter-cpp-0.23.4 tree-sitter-python-0.25.0

Same box, same proxy, a minute apart. pip is allowed, the pack's downloader isn't. Every environment I run semble in looks like this: package mirror, no general egress. Staging the cache with download() on a machine that has egress and copying it over works, but that's a second machine, a build step, and an ABI nothing pins. There's no in-process path.

To reproduce: install semble, rm -rf ~/.cache/tree-sitter-language-pack, run a search from a host with no egress to github.com¹

$ semble search "complex exponential moving average" -k 5 . 2>&1 >/dev/null
Language python not found, falling back to line chunking
Language cpp not found, falling back to line chunking
Language c not found, falling back to line chunking
Language cuda not found, falling back to line chunking
$ echo $?
0

That message points the wrong way: get_parser("python") raises LanguageNotFoundError, so chunking/core.py:37 logs "not found", but the message on the exception is Download error: Failed to fetch manifest from .... Python isn't missing, the network is. Nothing in the output says so unless you print the exception yourself; logging str(e) there would cover it I think (with a path forward)

Cost is smaller than I claimed last time. the Ranking barely moves: six queries over a 43-file repo with and without parsers, same top-hit file 6/6, scores within 0.002 BUT What changes is where chunks get cut:

line chunking: 329 chunks,  30 start at a definition   (9.1%)
AST chunking : 378 chunks,  81 start at a definition  (21.4%)

E.g. one query returns lines 28-53, which starts mid-body and runs into the next class declaration; w parsers it returns 42-52. So it's the chunks handed to the agent that degrade, not the ranking

Suggested fix

The tree-sitter org ships every grammar as its own pre-compiled wheel: https://github.com/tree-sitter/py-tree-sitter#install-languages. No toolchain, no runtime network, and it pins in a lockfile. Adding some sort of second step or fallback if the main payload is unable to be downloaded of yo, we found Python and Haskell or something and we're gonna use those and then fall back for plain text for everything else would be fine

Happy to take a pass at a PR if you'd take it. Also fine if you'd rather solve the offline case another way - I mostly just want it solved.

semble                    0.5.1
tree-sitter-language-pack 1.6.2
tree-sitter               0.25.2
python                    3.12
ubuntu                    24.04

Footnotes

1. Literally how I ran into this was from Claud.Ai where you can whitelist package managers and so on, but random GitHub release URLs and other stuff does not work for downloads ↩

[pszemraj]

Correcting myself: tree_sitter_language_pack.download([...]) does exist and the cache dir is overridable, so preloading is possible. I was wrong above.

It doesn't help here though. download() fails the same way get_parser() does - the endpoint it needs is the one that's blocked - so it only works from a machine that already has egress (Or if you were clever and preemptively downloaded and stashed it, somehow knowing you wouldn't be able to in the future)

[Pringled]

Hey @pszemraj, we are planning on looking for an alternative way to ship the grammars with Semble, but this is quite involved to do in a proper way for all systems. For the time being I suggest you download the grammars yourself separately in an environment where you are allowed to download them, and then package them yourself with Semble, that's probably the quickest fix.

[pszemraj]

I gave it a shot! a bit jank, but it works. everything (except other pypi deps) in the .whl: https://github.com/pszemraj/semble-offline/releases

I made a little Colab example:

I forked and based off of v0.5.1 - all you need is this:

# URL to the GitHub releases wheel
URL="https://github.com/pszemraj/semble-offline/releases/download/v0.5.1%2Boffline.1/semble-0.5.1+offline.1-py3-none-manylinux_2_34_x86_64.whl"
pip install -q $URL
# make sure it exists
semble search --help

[Pringled]

@pszemraj nice, that's good to hear! We will do something like this in a future version as well after some more testing and optimization (we want to keep the wheel as small as possible for example). I can keep you posted in this issue on any updates if you want 👍

[pszemraj]

yeah please do! and fwiw the whole bundle is only ~60 MB - pretty cheap for never having to hit the network at runtime. I get it's not the main path for most people though

[pszemraj]

Following up here for anyone curious or reading this: I've added a macOS wheel in the latest release, also ~60 MB:

pip install "semble[mcp] @ https://github.com/pszemraj/semble-offline/releases/download/v0.5.1%2Boffline.2/semble-0.5.1%2Boffline.2-py3-none-macosx_11_0_arm64.whl"