Skip to content

fix: skip index build when a source produces no text chunks - #320

Open
pcbeingused333 wants to merge 1 commit into
pyfenn:mainfrom
pcbeingused333:fix/index-empty-chunks
Open

fix: skip index build when a source produces no text chunks#320
pcbeingused333 wants to merge 1 commit into
pyfenn:mainfrom
pcbeingused333:fix/index-empty-chunks

Conversation

@pcbeingused333

Copy link
Copy Markdown
Contributor

Problem

Retriever.index() appends chunks and then always builds the backend:

for doc in docs:
    for chunk in chunk_text(doc, mode=self.chunk_mode):
        self.chunks.append(chunk)

if self.use_faiss:
    self._build_faiss()

A source that yields no text — an empty file, a scanned/image-only PDF (_read_pdf returns None, load_documents filters it out), or a blank string — leaves self.chunks empty. _build_faiss() then calls _embed([]) and reads vectors.shape[1]:

>>> RAG(use_faiss=True).add_source("scanned.pdf")
IndexError: tuple index out of range          # openai-compatible embedder
# or
ValueError: need at least one array to concatenate   # local embedder

Querying before a successful index also raises from _get_faiss() rather than returning nothing.

Fix

  • index() logs a warning and returns early when self.chunks is empty after chunking.
  • _query_faiss() returns [] when self._faiss_index is None.

No change when there is real text to index. index([]) on the BM25 path already worked and still does.

Tests

Added TestIndexWithoutChunks: FAISS index([]) and index([" \n\n "]) don't build, BM25 index([]) doesn't raise, and query() before any index returns []. The FAISS cases fail on main. nox -t fenn is green (922 passed, 7 skipped); ruff check / ruff format clean.

`Retriever.index()` appends chunks and then always calls `_build_faiss()`
(for `use_faiss=True`). A source that yields no text -- an empty file, a
scanned/image-only PDF, a blank string -- leaves `self.chunks` empty, and
`_build_faiss()` then embeds `[]` and reads `vectors.shape[1]`, raising
`IndexError: tuple index out of range` (or `ValueError: need at least one
array to concatenate` for the local embedder).

`index()` now logs a warning and returns early when there is nothing to
index, and `_query_faiss()` returns `[]` when the FAISS index was never
built instead of raising from `_get_faiss()`.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GtYgwYNfJ3wHrw9xrooVoB
@pcbeingused333

Copy link
Copy Markdown
Contributor Author

The pre-commit check is red on the ty hook, but that's pre-existing on main — commit 0d1862e (current main HEAD) fails the same job, on fenn/experimental/vision/normalize.py and fenn/logging.py, neither of which this PR touches. ruff check, ruff format and typos pass, and all four test (3.11–3.14) jobs are green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant