Skip to content

perf(remote): rescan granularity is the project folder, not the file #216

Description

@devsuitup

refreshHost() reduces each changed path to its top folder via topFolderOf()
(remote-hosts.js:84-87) and builds toScan from folders
(remote-index.js:92,114-119). The worker then re-enumerates every .jsonl in
that folder (workers/scan-projects.js:20-25), and readSessionFile() reads
each one whole — fs.readFileSync then split('\n') then JSON.parse per line
(read-session-file.js:167-169,193), with no offset and no incremental path.

Consequence: a 22 MB transcript that gains one line causes that whole file, plus
every other transcript in the same project folder, to be re-read and re-parsed.
For a host with live sessions writing continuously, this is paid every cycle.

It runs in a worker, so the UI does not freeze — the cost is CPU and memory, not
responsiveness.

Scope: make the scan unit the file, or make the parse incremental from the last
known offset. Either changes the cost from proportional-to-folder to
proportional-to-change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions