terminal-screenreader-multiplexer

A proof-of-concept terminal multiplexer built to make scrollback content accessible to screen readers on Windows.

The problem

Windows consoles (conhost.exe / Windows Terminal) don't expose their buffer through UI Automation in a way that lets a screen reader navigate scrollback line-by-line the way it can navigate a text editor or a web page. Querying the console's own window via UIA from another process can even deadlock. That makes console-heavy workflows — build logs, REPLs, long-running tools — hard to review non-visually once output has scrolled past.

The approach

This project runs a small, invisible bridge window alongside the terminal. While in "copy mode":

• The bridge window takes keyboard focus and answers UI Automation queries with an AccessKit tree that mirrors the visible terminal lines, cursor position, and a live status region.
• A screen reader navigating that tree (e.g. moving its cursor to a specific line and character) routes the app's own cursor to match, scrolling the view as needed.
• Arrow keys move the cursor; landing on a line containing an error or warning keyword (English or German) plays a distinct tone.
• New background output plays a debounced "activity" tone.
• Bookmarks can be set/toggled and cycled through.
• On leaving copy mode, keyboard focus returns to the console.

Commands use a tmux-style one-shot prefix: press F1 to arm the prefix, then a command key. Without the prefix armed, those keys are left for the shell.

Key	Action
Up / Down	Move cursor one line
Esc / F2	Exit copy mode
F1	Arm the command prefix
F1 then m	Toggle bookmark on the cursor line
F1 then n	Jump to next bookmark
F1 then p	Jump to previous bookmark
Esc / F1 while prefix armed	Cancel the prefix

Platform support

This crate targets Windows only. The AccessKit/UI Automation bridge in src/platform/win32.rs is not one backend among several — it is the whole product, so there is no portable subset worth shipping. Building for any other target stops at a compile_error! in src/lib.rs rather than failing with a wall of unresolved imports.

Building and running

cargo build
cargo run

cargo run starts a demo: it fabricates scrolling sample output to exercise activity tones and error/warning detection, and drops you into copy mode using the keys above.

Options are parsed with clap. Run cargo run -- --help for the full list.

Every option can also be set from the environment, which matters here: the binary is often started from a shortcut or a screen reader script where passing arguments is awkward. An explicit flag beats the environment variable.

cargo run -- --activity-debounce-ms 500   # activity-tone debounce, default 1000
TSM_ACTIVITY_DEBOUNCE_MS=500 cargo run    # same thing, from the environment

Running a real shell

--pty swaps the demo for PowerShell in a pseudo console:

cargo run -- --pty
cargo run -- --pty --shell cmd.exe

This is milestone 1 of the terminal backend and is deliberately transparent: shell output is written straight through without being parsed, and there is no copy mode over it yet. Every key is forwarded to the shell, so there is no local quit binding — type exit.

Values outside 100–10000 ms are rejected with an error message instead of being silently clamped, and the check runs before the terminal is put into raw mode.

Testing

cargo test

Integration tests cover the copy-mode state machine (tests/copy_mode.rs), line classification (tests/scan.rs) and the grapheme-index/display-column conversion (tests/width.rs).

Status

This is an early-stage proof of concept, not a production terminal multiplexer — there's no real pane/session management, and the "terminal" content in the demo is simulated rather than a real shell.