feat: introduce PageIndex SDK with Collection-based local/cloud API#272
Open
KylinMountain wants to merge 128 commits into
Open
feat: introduce PageIndex SDK with Collection-based local/cloud API#272KylinMountain wants to merge 128 commits into
KylinMountain wants to merge 128 commits into
Conversation
The cloud backend previously polled tree_resp["retrieval_ready"] as the ready signal. Empirically this flag is not a reliable indicator — docs can reach status=="completed" without retrieval_ready flipping, causing col.add() to wait until the 10 min timeout before giving up on otherwise-successful uploads. The cloud API's canonical ready signal is status=="completed"; switch the poll to check that instead.
* feat:compatible with Pageindex SDK * corner cases fixed * fix: mock behavior of old SDK * fix: close streaming response and warn on empty api_key - LegacyCloudAPI: close response in `finally` for both _stream_chat_response variants so abandoned iterators no longer leak the TCP connection. - PageIndexClient: emit a warning instead of silently falling back to local when api_key is the empty string, surfacing typical env-var-unset misconfig. - FakeResponse: add close()/closed to match the real requests.Response API. - Add unit coverage for stream close (both paths) and the empty-api_key warning. - Add scripts/e2e_legacy_sdk.py to smoke-test the legacy SDK contract end-to-end against api.pageindex.ai. * chore: mark legacy SDK methods with @deprecated and docstring pointers - Decorate the 12 PageIndexClient cloud-SDK compat methods with @typing_extensions.deprecated(..., category=PendingDeprecationWarning): - IDE/type-checkers render them with a strikethrough hint - runtime warnings stay silent by default (no spam for existing callers), surfaceable via `python -W default::PendingDeprecationWarning` - Add a one-line docstring on each pointing to the Collection-based equivalent. - Promote typing-extensions to a direct dependency (was transitive via litellm). --------- Co-authored-by: XinyanZhou <xinyanzhou@XinyanZhoudeMacBook-Pro.local> Co-authored-by: saccharin98 <xinyanzhou938@gmail.com> Co-authored-by: mountain <kose2livs@gmail.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Collaborator
Author
Code reviewFound 1 issue:
Lines 1 to 7 in 595895c Lines 22 to 33 in 595895c 🤖 Generated with Claude Code - If this code review was useful, please react with 👍. Otherwise, react with 👎. |
pageindex/config.py imports `from pydantic import BaseModel` in production code, but pyproject.toml only pulled pydantic in transitively via litellm. A future litellm release could drop or re-pin pydantic and break installs. Pin to `>=2.5.0,<3.0.0` to match the v2-style BaseModel usage already in the codebase, and to stay compatible with litellm's own pydantic constraint.
Filename is always built as `.png` regardless of the source ext, so the variable was dead. Flagged by github-code-quality.
- get_agent_tools branches on doc_ids:
- scoped (doc_ids=[...]): drops list_documents and hard-enforces a
whitelist on the remaining tools; system prompt switches to
SCOPED_SYSTEM_PROMPT (no list_documents instruction); doc list +
summaries are prepended to the user message via wrap_with_doc_context.
- open (doc_ids=None): unchanged 4-tool agent loop.
- list_documents now exposes doc_description (sqlite + cloud).
- Collection.query emits UserWarning when doc_ids is None and the
collection holds >1 documents; PAGEINDEX_EXPERIMENTAL_MULTIDOC=1
silences it. Single-doc collections skip the warning; empty
collections raise ValueError.
- Agents SDK tracing upload disabled by default (avoids SSL timeouts);
PAGEINDEX_AGENTS_TRACING=1 re-enables it.
- README: new SDK Usage section covering local/cloud quick start,
streaming, multi-doc as experimental, and runnable examples.
- Collection.query and Backend.query/query_stream accept doc_ids as str, list[str] or None. Single str is normalized to [str] inside each backend; bare [] is rejected with ValueError at both layers. - wrap_with_doc_context wraps the scoped doc list in <docs>...</docs> and SCOPED_SYSTEM_PROMPT instructs the agent to treat that block as data, not instructions (defense against prompt injection via auto-generated doc_description). - _require_cloud_api now distinguishes api_key="" from api_key=None; the former gives a targeted error pointing at the empty-string vs fall-back-to-local situation when legacy SDK methods are called. - Legacy PageIndexClient.list_documents docstring spells out the return-shape difference vs collection.list_documents() to flag a silent migration footgun (paginated dict with id/name keys vs plain list[dict] with doc_id/doc_name keys). - Remove dead CloudBackend.get_agent_tools stub (not on the Backend protocol; only ever returned an empty AgentTools()) and the SYSTEM_PROMPT alias (OPEN_/SCOPED_SYSTEM_PROMPT are the explicit names now). - README quick start and streaming example now pass doc_ids; new multi-document section shows both str and list forms. - examples/demo_query_modes.py exercises all five query-mode cases (single-doc, multi-doc with/without env var, scoped single, scoped multi) for manual verification.
KylinMountain
changed the title
BukeLy
previously approved these changes
May 15, 2026
scripts/e2e_legacy_sdk.py becomes examples/demo_legacy_sdk.py to sit alongside the other runnable demos (local/cloud/query-modes), and the README's Runnable examples list now points at it. Docstring command updated to the new path; the legacy script docstring also calls out that it exercises the 0.2.x compatibility methods. The scripts/ directory had no other entries and is removed.
…e global
llm_completion / llm_acompletion (pageindex/utils.py and
pageindex/index/utils.py) set `litellm.drop_params = True` on the litellm
module. litellm is a process-wide singleton, so this leaked into every other
library sharing it (e.g. a host app like OpenKB that exposes its own litellm
config) and could not be turned off.
Replace the hardcoded `temperature=0` + global `drop_params` with a single
PageIndex-owned per-call kwargs mechanism (config._LLM_PARAMS):
- defaults preserve behavior: {"temperature": 0, "drop_params": True};
- passed per call via **get_llm_params(), never writing litellm's globals, so
nothing leaks into other litellm users in the same process;
- externally configurable: pageindex.set_llm_params(drop_params=False,
temperature=1, num_retries=5, ...) or the PAGEINDEX_DROP_PARAMS env shortcut;
- model/messages are reserved (PageIndex supplies them) and rejected.
| self._init_local(model, retrieve_model, storage_path, storage, index_config) | ||
|
|
||
|
|
||
| class CloudClient(PageIndexClient): |
list_to_tree() deletes the 'nodes' key from leaf nodes entirely via
clean_node(). Direct access via structure['nodes'] raises KeyError on
these nodes. Using structure.get('nodes') returns None (falsy) safely,
consistent with how 'nodes' is accessed elsewhere in the codebase.
Fixes #330
…ayer Fixes the cloud/local contract mismatches from the PR #272 review (verified against the official API docs — the cloud API has no folder/collection endpoints publicly, GET /docs supports limit<=100 with offset): - query_stream: emit a terminal answer_done event with the full answer (same contract as the local backend); raise CloudAPIError instead of disguising HTTP errors as answer events; move the initial connect inside try so a connection failure can no longer strand the consumer awaiting a sentinel that never arrives - _request: rewind file objects before retrying so a transient 5xx/429 during upload no longer re-sends an empty multipart body; carry the HTTP status on CloudAPIError (status_code) and keep the last status in the max-retries error - list_documents: paginate with limit/offset instead of a hard-coded limit=100, so >100-doc collections are no longer silently truncated (whole-collection queries rely on this list) - folders: treat only 403/404 as "folders unavailable" (warned via warnings.warn instead of an invisible logger.warning, matched on status_code instead of a "403" substring); transient errors now propagate instead of being permanently cached as folder_id=None - error taxonomy parity: cloud doc endpoints map HTTP 404 to DocumentNotFoundError; local get_document raises DocumentNotFoundError instead of returning {}; local delete_document raises on missing doc_id instead of silently deleting nothing - cloud get_document warns that include_text is not supported instead of silently ignoring it Adds regression tests for each fix (11 new tests). Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- local delete_collection: validate the collection name before rmtree. An unvalidated name like "../.." escaped files_dir and deleted arbitrary directories (path traversal). - legacy page_index(): restore the node_id/summary/text/description enhancements. IndexConfig now carries booleans (pydantic coerces the legacy 'yes'/'no' strings at the boundary), but page_index_main still compared `opt.if_add_node_id == 'yes'` — always False — so every enhancement was silently skipped for legacy-API callers. Conditions now branch on the booleans, matching pageindex/index/page_index.py. - LegacyCloudAPI._request: bound every request with a timeout (30s, 120s read timeout for streamed responses) so a dead connection can't hang legacy submit/poll/chat callers forever. The legacy contract tests pinned the missing timeout; updated to pin its presence instead. Adds regression tests: path-traversal rejection, 'yes'/'no' -> bool coercion, and timeout assertions. Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- AgentRunner.run: offload to a worker-thread event loop when called from inside a running loop (Jupyter, FastAPI handlers) — mirrors pipeline._run_async; Runner.run_sync raised RuntimeError there. - SQLiteStorage: create connections with check_same_thread=False so close() can actually close connections created by worker threads. Each thread still gets its own connection via threading.local; with the default True those closes raised ProgrammingError (silently swallowed) and leaked every worker connection. - CloudBackend.query: non-streaming chat completions now use a 300s timeout and a single attempt. The default 30s ReadTimeout fired before generation finished and the retry loop re-billed the full server-side retrieval + generation up to three times. _request gains retries/timeout overrides; the exhausted-retry path also no longer sleeps before raising. - MarkdownParser: content before the first heading (abstract/preamble) becomes a node instead of being silently dropped and unretrievable; a file with no headings at all yields a single document node instead of zero nodes (which pushed an empty page list into the pipeline). - LegacyCloudAPI.is_retrieval_ready: API failures (revoked key, network down) now propagate as PageIndexAPIError instead of reading as "not ready", which turned polling loops into infinite loops. Adds regression tests for each fix. Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
…on shims The new SDK copied the legacy indexing pipeline into pageindex/index/ instead of moving it, leaving two divergent copies of page_index.py / page_index_md.py / utils.py. They had already drifted (the legacy copy still compared IndexConfig booleans against 'yes' — a separate fix), and every pipeline change had to be applied twice. Make pageindex/index/ the single source of truth (same pattern as the LegacyCloudAPI shim for the 0.2.x cloud SDK): - pageindex/index/utils.py absorbs the 27 legacy-only helpers/classes (get_page_tokens, convert_page_to_int, ConfigLoader, PDF text helpers, ...) so it's the sole utils module. Reconciled the diverged funcs: kept the modern versions, backported the #331 get_leaf_nodes .get() fix, and restored remove_fields' max_len parameter (superset). - index/page_index*.py now import `from .utils import *`; index/legacy_utils.py (a re-export of the old top-level utils) deleted. - Top-level page_index.py / page_index_md.py / utils.py become thin re-export shims that emit PendingDeprecationWarning. The md_to_tree shim coerces legacy 'yes'/'no' string flags to bool (the canonical version is boolean-typed). - ConfigLoader no longer reads the deleted config.yaml; it builds defaults from IndexConfig (was an unconditional FileNotFoundError). - __init__.py and retrieve.py import from pageindex.index.* directly so `import pageindex` does not trip the shims. Adds tests/test_legacy_shims.py pinning the contract: clean top-level import doesn't warn, legacy submodule imports warn, symbols still resolve, shim and canonical share one implementation, the #331 fix and ConfigLoader-without-yaml both hold, and the md_to_tree coercion works. Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- legacy call_llm: open AsyncOpenAI via `async with` so the client (and its HTTP connection pool) is closed instead of leaked. - LocalBackend.add_document: fail fast with CollectionNotFoundError when the collection doesn't exist, before the expensive parse + LLM index (previously the missing FK only tripped at save time, after paying for the LLM work). Also raise builtin FileNotFoundError for a missing path instead of FileTypeError (which now means only "unsupported extension"). - Collection.query(doc_ids=None): the empty-collection guard now always runs — previously it was skipped once PAGEINDEX_EXPERIMENTAL_MULTIDOC was set. A single list_documents call serves both the guard and the multi-doc warning (no separate call just to decide whether to warn). - CloudBackend.query_stream: on early consumer break / GeneratorExit, signal the background SSE thread to stop and force-close the response, so it no longer drains the whole stream in the background. Adds regression tests for each (client closed, fail-fast on unknown collection, FileNotFoundError, empty-check under the multidoc env flag, single list call, early-break thread stop). Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
- SQLite dedup race: add UNIQUE(collection_name, file_hash) and switch save_document to a plain INSERT. add_document now catches the IntegrityError from a concurrent add of the same content, cleans up its managed files, and returns the winning doc_id — instead of two doc_ids for one file (each having paid for its own LLM indexing). - PDF image paths: store the absolute path to each extracted image instead of a path relative to the indexing process's cwd. The  references broke as soon as a query ran from a different directory. - LegacyCloudAPI: URL-encode doc_id / retrieval_id path segments (added _enc()), matching CloudBackend. An id containing '/', '?', '#' or a space previously hit the wrong endpoint or produced a malformed URL. Adds regression tests: UNIQUE enforcement, the add-race resolving to the winner, absolute image paths, and encoded legacy URLs. Claude-Session: https://claude.ai/code/session_01Kx5DgKbhK1N8autqXH8SmS
File-first ordering meant a failed DB delete left a listed, 'completed' document whose files were gone — and its surviving file_hash made re-adding the same file dedup to the broken doc. DB-first (the order delete_collection already uses) degrades the same failure to harmless orphan files.
Both legacy entry points deliberately finish with format_structure — whitelisting public fields, fixing key order, pruning empty 'nodes'. build_index skipped it, so the SDK stored raw trees: empty nodes lists on every markdown leaf, internal keys leaking into get_document output, and a different shape from the same document indexed via page_index().
Parsers used path.stem, so the same file was named 'report' in SDK local mode but 'report.pdf' in the legacy path and cloud mode; docs differing only by extension became indistinguishable to the selection agent.
Member
Code reviewFound 3 issues:
PageIndex/pageindex/backend/local.py Lines 134 to 140 in d1824e4
PageIndex/pageindex/cloud_api.py Lines 43 to 48 in d1824e4
Lines 150 to 156 in d1824e4 🤖 Generated with Claude Code - If this code review was useful, please react with 👍. Otherwise, react with 👎. |
…tching 0.2.x 0.2.8 had no try/except around its requests calls: ConnectionError/Timeout reached the caller as-is, and is_retrieval_ready — which only catches PageIndexAPIError — deliberately let them escape its polling loop. Wrapping every RequestException into PageIndexAPIError (introduced alongside the timeout in 595895c, unmentioned in its message) turned a network outage into a silent not-ready that polls until timeout. Keep the timeout; drop the wrapping, including the two per-stream re-wraps.
…hing legacy Both legacy entry points nest if_add_doc_description inside the if_add_node_summary branch: create_clean_structure_for_description keeps only titles and summaries, so without summaries the description LLM call runs on a bare title listing. The SDK pipeline (c7fe93b) hoisted the check to top level with no stated rationale, so summary=False + description=True paid for an extra LLM call and produced a low-quality description the legacy path deliberately refuses to generate.
The server's GET /docs/ returns createdAt DESC (id ASC tiebreak) and the cloud backend preserves that order, so list_documents()[0] meant the newest document on cloud but the oldest on local (ORDER BY created_at ascending). The same order also feeds _get_all_doc_ids' doc priority for multi-doc chat. rowid DESC tiebreaks same-second inserts by insertion order.
0.2.8 stored self.api_key on the client and _headers() re-read it per request, so reading client.api_key and reassigning it after construction both worked. The refactor handed the key to the internal backends and never set it on PageIndexClient: reads raised AttributeError and reassignment silently kept requests on the old key. Same shape as the BASE_URL snapshot fixed in dea211b — apply the same callable-indirection pattern to api_key on both CloudBackend and LegacyCloudAPI (which also stops CloudBackend snapshotting its headers dict at construction).
main's ConfigLoader never validated the YAML's own keys — config.yaml defined the legal key set and unknown entries merged harmlessly into the SimpleNamespace. Wiring the YAML into IndexConfig (f995e64) inherited c7fe93b's extra=forbid, inverting that: any deprecated or user-added key now raised an uncaught ValidationError from the CLI and ConfigLoader. Filter YAML data to known fields with a warning; explicit keyword overrides keep strict validation.
…lder deletion
The server exposes only POST /folder and GET /folders (verified against the
full route table and its git history — a DELETE /folder route never
existed; the 0.2.x SDK accordingly shipped create_folder/list_folders but
no delete). The DELETE /folder/{id}/ call could only 404, surfacing as a
misleading 'Not Found' CloudAPIError while leaving the stale folder id in
the cache. Raise a clear not-supported error pointing at the dashboard;
keep the idempotent no-op for missing collections and folderless plans.
…d create_collection
The server rejects a duplicate folder name with HTTP 400 ('A folder named
"x" already exists in this location'); the local backend raises
CollectionAlreadyExistsError for the same condition (4e6a135 wired the new
exception into sqlite only). except CollectionAlreadyExistsError written
against local mode never fired on cloud, where the 400 surfaced as a bare
CloudAPIError — the same parity gap _doc_request already closes for 404 →
DocumentNotFoundError.
…tion
The compute API now exposes folder deletion (cascade: documents and
subfolders; 404 folder_not_found; 409 when descendants are still
queued/processing), so replace the not-supported error with the real call.
The endpoint's own 404 ('Folder not found.') is treated as idempotent
success and drops the cached folder id; a bare route-miss 404 from a server
without the endpoint still raises so the delete can't silently no-op
against old deployments. 409/5xx propagate with the cache intact.
… tree endpoint only returns 'result'
Folder names are only unique per parent, and an unscoped GET /folders/ returns every nesting level — a nested folder sharing the name could be matched first and silently receive documents. Scope all name resolution with parent_folder_id=root, matching where collections are created.
- _require_document: always verify doc existence via _get_metadata, even when folders are unavailable — previously a non-existent doc_id silently passed through to the chat API on non-Max plans - get_document: forward pageNum from the server metadata as page_count, matching the local backend's DocumentDetail contract - _normalize_nodes: preserve prefix_summary as a separate field instead of collapsing it into summary, matching the server's leaf/non-leaf distinction
| @runtime_checkable | ||
| class Backend(Protocol): | ||
| # Collection management | ||
| def create_collection(self, name: str) -> None: ... |
| class Backend(Protocol): | ||
| # Collection management | ||
| def create_collection(self, name: str) -> None: ... | ||
| def get_or_create_collection(self, name: str) -> None: ... |
| # Collection management | ||
| def create_collection(self, name: str) -> None: ... | ||
| def get_or_create_collection(self, name: str) -> None: ... | ||
| def list_collections(self) -> list[str]: ... |
| def create_collection(self, name: str) -> None: ... | ||
| def get_or_create_collection(self, name: str) -> None: ... | ||
| def list_collections(self) -> list[str]: ... | ||
| def delete_collection(self, name: str) -> None: ... |
|
|
||
| # Document management. Contract: a doc_id not belonging to `collection` | ||
| # must behave exactly like a missing one (DocumentNotFoundError). | ||
| def add_document(self, collection: str, file_path: str) -> str: ... |
| # Document management. Contract: a doc_id not belonging to `collection` | ||
| # must behave exactly like a missing one (DocumentNotFoundError). | ||
| def add_document(self, collection: str, file_path: str) -> str: ... | ||
| def get_document(self, collection: str, doc_id: str, include_text: bool = False) -> DocumentDetail: ... |
| # must behave exactly like a missing one (DocumentNotFoundError). | ||
| def add_document(self, collection: str, file_path: str) -> str: ... | ||
| def get_document(self, collection: str, doc_id: str, include_text: bool = False) -> DocumentDetail: ... | ||
| def get_document_structure(self, collection: str, doc_id: str) -> list: ... |
- Add `status` column to the documents table (NOT NULL, no default for new databases; ALTER with DEFAULT 'completed' for migration). save_document requires an explicit status — no silent fallback. local.py passes "completed" at save time instead of hardcoding it at read time, aligning with the server's FilePageIndex schema. - Derive doc_type from the document filename extension in the cloud backend instead of hardcoding "pdf" — the cloud API accepts DOCX, PPTX, TXT, MD and other formats. - Fix incorrect comment claiming the cloud /docs/ endpoint caps limit at 100 (the server accepts up to 10000; 100 was a client-side restriction in the 0.2.x SDK).
The server's chat/completions agent identifies documents by doc_name (api.py:3108-3112), not doc_id. The local agent tools used UUID-based doc_id, forcing the LLM to copy opaque strings and making same-name documents indistinguishable. - Agent tools now take doc_name (doc_id accepted as fallback) - _resolve: name→id within scope, ambiguity returns candidates - _sanitize_doc_name: mirrors server sanitize_filename (NFKC, whitespace collapse, 180-byte truncation with md5 suffix) - _dedupe_doc_name: a.pdf → a_1.pdf on collision, matching the server's _find_unique_key suffix scheme - _defang_delimiters: also collapses whitespace to block newline injection in the <docs> prompt block - System prompts and wrap_with_doc_context updated to name-first
- Remove stream_metadata from chat/completions payloads — the server's ChatCompletionRequest model has no such field; it was silently ignored. - Local list_documents tie-breaker: rowid DESC → doc_id ASC, matching the server's "createdAt" DESC, "id" ASC ordering. - Cloud get_document: skip page_count when pageNum is 0 (server returns 0 for NULL, meaning "not yet computed", not "zero pages").
The 0.2.x surface is the mainstream cloud API for now — Collection is an additive layer, and chat_completions has capabilities query() does not cover (temperature, citations, message history). PEP 702 deprecated + PendingDeprecationWarning told users these methods are going away (IDE strikethrough, pytest warning spam) when there is no removal plan. Docstrings keep neutral Collection API cross-references.
Same adjudication as the client methods (42d57b6): pageindex.utils is the documented 0.2.8 location, and the top-level modules are permanent compatibility surface — "will be removed in a future release" was not true. The shims stay; only the import-time warnings go.
`import pageindex` eagerly pulled the whole indexing stack — litellm (which fetches a remote model cost map at import) and PyPDF2 — costing ~3.3s plus a network attempt for cloud-only 0.2.x users who never touch local indexing. Legacy pre-SDK names now resolve via PEP 562 __getattr__; a TYPE_CHECKING block keeps real signatures for IDEs. Import drops to ~0.3s with zero network and no litellm/PyPDF2 in sys.modules until a legacy indexing name is actually used.
index/utils.py mixes LLM callers with plain string/dict helpers, so any consumer of parse_pages/create_node_mapping (retrieve, page_index_md, the cloud backend's tree normalization) paid litellm's multi-second import and network fetch. The heavy imports now happen at first real use; sys.modules caches them after that.
Once litellm/PyPDF2 moved inside the functions that use them, the legacy modules became cheap to import — the PEP 562 __getattr__ table in __init__ was guarding a door that no longer needs guarding. Restore plain eager imports (~0.3s total, still zero heavy deps at import); keep only the small submodule __getattr__ for pageindex.utils / pageindex.page_index_md attribute access.
11ba63d copied the server's "id" ASC rule, but server ids are time-ordered cuids so that rule means insertion order there — local uuid4 doc_ids sort randomly. rowid ASC reproduces the actual effect.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turn PageIndex into a proper Python SDK. The repo currently exposes a tree-generation CLI (
run_pageindex.py) and a legacy cloud HTTP wrapper (pageindex_sdk0.2.x). This PR introduces a unifiedPageIndexClientwith a Collection-based API that works in both local self-hosted mode (your LLM key) and cloud-managed mode (PageIndex API key), while keeping the legacy SDK callable on the same client for backward compatibility.What's in the SDK
Public surface (
from pageindex import PageIndexClient):PageIndexClient(api_key=...)— auto-detects cloud vs localclient.collection(name)→CollectionCollection.add/list_documents/get_document/get_document_structure/get_page_content/delete_document/querycol.query(question, doc_ids=..., stream=...)—doc_idsacceptsstr | list[str] | NoneQueryEventTwo execution paths behind the same API:
/chat/completions/,/doc/...endpointsLegacy 0.2.x compatibility on the same
PageIndexClient: all 12 methods (submit_document,get_ocr,get_tree,chat_completions, document & folder management) preserved as@deprecatedwrappers, same signatures and return shapes.Highlights
pageindex_sdk0.2.x methods onPageIndexClient(submit_document,chat_completions, etc.), with stream-close fixes and@deprecatedmigration markersadd_documentpollsstatus == "completed"instead of the unreliableretrieval_readyflagdoc_idsacceptsstr | list[str] | None; empty list rejected at both Collection and Backend layersdoc_idsis provided): dropslist_documentsfrom the agent's tool set, hard-enforces whitelist on doc_id args, and injects doc summaries into the user message inside<docs>with system-prompt guidance treating it as data, not instructions (prompt-injection mitigation)Collection.list_documents()now exposesdoc_descriptionso the agent can route to the right docdoc_ids=Noneover a multi-doc collection emits aUserWarning(cross-doc retrieval is experimental;PAGEINDEX_EXPERIMENTAL_MULTIDOC=1to silence). Single-doc skipped, empty collection raisesValueErrorPageIndexClient.list_documentsdocstring spells out the return-shape difference vsCollection.list_documents(); clearer error when legacy methods are called afterapi_key=""PAGEINDEX_AGENTS_TRACING=1re-enables)examples/local_demo.py,examples/cloud_demo.py,examples/demo_query_modes.py(5 query-mode cases)dist/to gitignoreTest plan
pytest tests/— 101 passed, 2 skippedexamples/demo_legacy_sdk.py— all 7 legacy methods green againstapi.pageindex.aiexamples/demo_query_modes.py— all 5 Collection.query modes (single/multi/scoped × 2, env-var silencing) green0.2.x behavior notes
Compatibility regressions found in review are fixed (
dea211b,0f593c6):page_index()/ConfigLoaderreadconfig.yamlagain (explicit args > YAML >IndexConfigfield defaults, soif_add_doc_descriptionstays off by default on the legacy path),ConfigLoader(default_path=...)is honored (missing file raisesFileNotFoundError),pageindex.utils.*/pageindex.page_index_md.*attribute access works again (lazily bound — plainimport pageindexstays warning-free), andclient.BASE_URLreassignment after construction routes requests again.Three 0.2.x behaviors are intentionally not preserved:
get_page_content(..., "3,8")on a Markdown doc returns the nodes at lines 3 and 8, mirroring the PDF path — 0.2.x returned everything in the [3, 8] envelope. Use"3-8"to request a range."1-1500") return an actionable error instead of expanding — split into chunks ("1-1000","1001-1500"). The spec is LLM/caller-reachable, so unbounded expansion was a DoS vector (a"1-2000000000"spec previously allocated billions of ints).from pageindex import *is narrowed to__all__. The old star surface was an accident of chained star-imports (it even leakedjson/os/yaml). Explicit forms are unaffected:from pageindex import extract_json,pageindex.extract_json, andfrom pageindex.utils import ...all keep working.https://claude.ai/code/session_014B4HZkjdSiZXDmJtH5Jexn