Feat/feasibility check

[voorhs]

What this adds

A pre-flight compute feasibility advisor. Before any download or training, it estimates VRAM, RAM, disk, and wall-time for a search space on the current machine, and either reports or blocks when the budget will not hold.

Two entry points:

• CLI — autointent-advisor inspect <preset|config.yaml> and autointent-advisor recommend, both accepting a real --dataset or --n-samples/--n-classes/--avg-tokens placeholders. Non-zero exit when nothing is feasible, so it works as a CI gate.
• Python — autointent.advisor, a deliberately narrow 15-name surface marked experimental.

Docs: docs/source/advisor.rst. The rename from _advisor to advisor also means the package now publishes an auto-generated API reference page.

Validation

Validated on real constrained hardware (RTX 3060 Laptop, 6 GB VRAM / 16 GB RAM) in Darinochka/AutoIntent-experiments#40, which closes issue #39 there: 4/4 fitted presets matched their predicted verdict — both OVER predictions actually OOM'd, both feasible predictions actually fit, reduce-to-fit produced a runnable pipeline, and the strict gate aborted before allocating any VRAM.

That validation also surfaced a bug now fixed here (below), and two accuracy limits that are documented rather than fixed: VRAM is close but not a guaranteed upper bound (one preset at 1.22× prediction), and wall-time estimates are indicative only, with measured error running in both directions. Both are stated plainly in docs/source/advisor.rst; the "pessimistic upper bound" language has been removed from the code, since the measurements do not support it.

Reviewer notes

Pipeline.fit(preflight=...) defaults to "off". resolve_model() calls HfApi().model_info() per distinct model name with no cache-first short-circuit, so a default-on gate would add N Hub round-trips to every fit() and one WARNING per model when offline — on the library's hottest path, for estimates that are explicitly heuristic. The advisor import is lazy and a test asserts import autointent does not import autointent.advisor. Flipping the default to "warn" later is purely additive.

psutil becomes a new core dependency. There is no cross-platform stdlib RAM query, and no psutil-absent fallback exists — _hardware.py imports it unguarded. It is small and ubiquitous, but it is a genuine addition to the base install, so flagging it explicitly rather than burying it.

Bug fix: reduce_to_fit pruned by VRAM regardless of the binding constraint. Finding.metric holds short names ("vram", "ram", "disk", "time") but _pick_module_to_drop tested membership against ["vram_gb", "time_hours", ...]. Disjoint sets, so the lookup never matched and every prune silently fell back to VRAM — correct by accident on VRAM-bound machines, wrong everywhere else. Every existing test used a profile with hardcoded ram_gb/free_disk_gb, which is why a green suite missed it. Found in experiments#40, finding 3.

Behaviour change: validate_modules now runs before the preflight gate. Previously preflight priced the unfiltered search space, so it charged for modules fit() was about to discard — mlknn on multiclass datasets, and dnnc (plus its ~6.4 GB reranker) on multilabel. On a small GPU that could flip the verdict to OVER and raise PreflightError on a fit that would have succeeded.

⚠️ One consequence worth a decision: with incompatible_search_space="raise" and preflight="strict", an invalid-and-infeasible config now raises ValueError where it previously raised PreflightError. The precedence is deliberate — a config that cannot run should not be priced — but fit()'s Raises: section does not yet mention ValueError. Happy to add that line if you want it.

SearchSpacePreset keeps its original ordering. An earlier revision of this branch reordered the public Literal into a cost ranking and then derived the ranking from get_args() declaration order, making a public type alias's element order load-bearing. That is now an explicit PRESET_COST_ORDER with a test that fails if a preset is added to one and not the other.

PreflightError is importable only from autointent.advisor. There is deliberately no re-export from autointent._pipeline: autointent/__init__.py imports ._pipeline, so an eager advisor import there would load the advisor on every import autointent and defeat the lazy-import guarantee above.

The calibration harness is not in this diff. scripts/calibrate_advisor.py and friends were validation instruments; they belong beside the results they produced, not in the library — and one of them was imported by a test from scripts/, which is not a package, breaking mypy. See the companion PR against the experiments repo.

Verification

ruff check clean · mypy src/autointent tests clean (323 files) · docs mypy clean · 143 advisor + preflight tests passing · make docs succeeds · no JSON-schema diff · import autointent does not import autointent.advisor · advisor.__all__ is exactly 15 names.

Please apply the full-ci label before merging — the full matrix and the docs build have never run on this branch, and the advisor has platform-specific code (psutil, torch.mps, shutil.disk_usage, Windows drive-letter parsing).

[voorhs]

в целом по методике и алгоритам ок за исключением мелочей которые прокомментил (посмотрел не прямо все но пока это стоит исправить)

есть два пожелания:

• наверное стоит добавить какой-то обоснованности всем используемым формулам (ссылки на внешние ресурсы, бенчмарки, статьи в которых исследуется такое) - вообще с этого стоило начать выполнение этой задачи)
• очень неудобно ревьюить когда в одном бульоне приватные утилиты и публичные функции, мне кажется стоит руками самому как-то разнести все это на подфайлы и подпапки, потому что иишке это ок, а человечески очень тяжело когда файл на 800 строк и в нем центральный публичный метод с главным алгоритмом спрятан где-то посередине или в конце

[voorhs]

норм

[voorhs]

Отвечаю на вопрос про кеширование эмбеддингов (он так и остался без ответа) — да, учитывается.

run_preflight принимает embedding_cache_probe: Callable[[str], bool]: коллбэк (embedder_model_name) -> bool, который отвечает, лежат ли уже в кеше эмбеддинги этой модели для текущего датасета. Для таких эмбеддеров _apply_embedding_cache (advisor/_estimates/_resource.py) обнуляет и forward-время, и disk_embedding_cache_gb — по той же логике, что и cached_locally для весов с HF Hub. В отчёте это видно отдельной строкой: +N GB embedding cache в disk-разбивке.

По умолчанию коллбэка нет, и advisor считает пессимистично — как будто кеш холодный и каждый эмбеддер платит за прогон один раз. Это осознанно: советчик должен оценивать сверху, а не снизу.

Отдельный от кеша момент — сам размер кеша на диске считается как n_samples x hidden_size x 4 (fp32) на файл, _embedding_cache_disk_gb.