Skip to content

feat: incremental + parallel indexing for large monorepos - #7

Merged
NeelPrime merged 2 commits into
mainfrom
feature/incremental-parallel-indexing
Jul 7, 2026
Merged

feat: incremental + parallel indexing for large monorepos#7
NeelPrime merged 2 commits into
mainfrom
feature/incremental-parallel-indexing

Conversation

@NeelPrime

Copy link
Copy Markdown
Owner

Summary

Implements incremental indexing (#5) and parallel indexing (#6) for large monorepos.

Changes

Incremental Indexing (default behavior)

  • Compares file content hash before parsing — unchanged files are skipped
  • Deleted files are automatically removed from the database
  • Second run on same codebase completes in milliseconds
$ arbiterx map          # First run: indexes 300 files (3s)
$ arbiterx map          # Second run: skips 300 unchanged files (0.05s)
$ arbiterx map --force  # Force full re-index

Parallel Indexing (opt-in)

  • Uses ProcessPoolExecutor for multi-core parsing
  • Each worker gets its own TreeSitterParser (no shared state)
  • Single writer merges results into SQLite (no locking issues)
  • Only activates for 100+ files (overhead threshold)
$ arbiterx map --workers 8    # Use 8 cores
$ arbiterx map --workers 0    # Auto-detect CPU cores

CLI Flags

  • --workers / -w — Number of parallel workers (default: 1)
  • --force / -f — Re-index all files ignoring cache

Testing

  • 109 tests passing
  • Lint clean
  • Tested with multi-project monorepo (7 languages, 22 files)
  • Verified incremental skip + force re-index + parallel parsing

Closes #5, Closes #6

NeelPrime added 2 commits July 7, 2026 16:49
Implements #5 and #6:

Incremental indexing:
- Compares file content hash before parsing
- Unchanged files are skipped automatically
- Deleted files are removed from the database
- Second run on same codebase completes instantly

Parallel indexing:
- arbiterx map --workers 8 (or --workers 0 for auto-detect)
- Uses ProcessPoolExecutor for multi-core parsing
- Each worker gets its own TreeSitterParser instance
- Single writer merges results into SQLite (no concurrency issues)
- Only activates for 100+ files (overhead threshold)

CLI changes:
- Added --workers / -w flag (default: 1, sequential)
- Added --force / -f flag (re-index all, ignore cache)
- Output table shows files_total, files_indexed, files_skipped

Closes #5, Closes #6
@NeelPrime
NeelPrime merged commit 32352ca into main Jul 7, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Parallel indexing for large monorepos (1M+ files) [Feature]: Incremental map — skip unchanged files, only re-index what changed

1 participant